AR multi-person collaborative interaction projection system and method for public space

Abstract

This invention discloses an AR multi-person collaborative interactive projection system and method for public spaces, belonging to the field of AR multi-person collaborative interactive projection technology. It includes a multi-source spatial calibration module configured to extract static physical feature point clouds in the public space based on real-time environmental images and depth data collected by multiple AR devices, and to perform global registration of the feature point clouds using a weighted nearest neighbor iterative algorithm to generate a unified world coordinate system. This invention utilizes the multi-source spatial calibration module to perform global registration of static physical feature point clouds collected by multiple AR devices using a weighted nearest neighbor iterative algorithm to generate a unified world coordinate system, and detects the relative pose deviation between each AR device. When the deviation exceeds a preset threshold, a dynamic recalibration process is triggered, effectively solving the problem of inconsistent coordinate systems among multiple AR devices in public spaces, and significantly improving the spatial consistency and registration accuracy of multi-person collaboration.

Description

Technical Field

[0001] This invention relates to the field of AR multi-person collaborative interactive projection technology, and more specifically, to an AR multi-person collaborative interactive projection system and method for public spaces. Background Technology

[0002] With the continuous maturation of augmented reality (AR) technology, multi-user collaborative interactive systems for public spaces have gradually become a research and application hotspot. In typical scenarios such as digital displays of museum artifacts, collaborative design of urban planning models, emergency escape drills, and interactive games in science museums, multiple users wear AR glasses or use mobile AR devices to jointly observe and manipulate virtual content superimposed on the real physical environment. These systems typically rely on technologies such as real-time visual localization and mapping, multi-device relative pose estimation, and centralized data synchronization to attempt to establish a unified spatial reference system and enable shared operations on the same virtual object by multiple users. To obtain a usable collaborative experience in open and dynamic public environments, existing solutions often require periodic pose calibration between devices and simple locking mechanisms or timestamp sorting of user operation commands to maintain basic interaction consistency.

[0003] However, existing technologies still have the following significant drawbacks in practical deployments in public spaces: First, multiple AR devices typically construct their own independent spatial coordinate systems, lacking a high-precision, dynamically correctable global unified world coordinate system. Furthermore, dynamic interference in public spaces, such as pedestrian movement and lighting changes, easily leads to drift and deviation in the relative poses between devices. The registration error accumulates over time, significantly reducing the spatial consistency of multi-user collaboration. Second, when multiple users simultaneously issue concurrent operation commands to the same virtual object (such as virtual artifacts, fire extinguishers, or puzzle pieces), existing systems lack a priority arbitration mechanism that comprehensively considers the order of operation time and the spatial distance between the operators. Simple timestamp arbitration easily overlooks the actual reachability of the operator and the target, while simple distance-based judgments may be unfair due to network or perception delays, resulting in unclear operation results or poor user experience. Third, the existing rendering strategy does not adaptively adjust to the real-time field of view and screen resolution of AR devices, causing virtual content to be easily ignored by users in narrow fields of view. At the same time, a large amount of content outside the field of view still occupies rendering resources, reducing system efficiency and immersion. Fourth, users will experience significant physical fatigue when performing gestures such as raising their hands and extending their arms for a long time. The existing system does not actively detect the duration of hand-raising operations during the interaction process and automatically switch to auxiliary interaction channels such as eye tracking or voice, lacking the ability to intelligently alleviate the physiological burden on users. Fifth, because the physical environment of public spaces (such as lighting and shadows, temporary exhibits) changes slowly, the existing coordinate system lacks a reconstruction mechanism based on stable feature point screening and periodic aging compensation. After long-term operation, it is difficult to maintain sub-centimeter-level high-precision registration, which cannot meet the stringent requirements of professional collaborative scenarios for spatial accuracy.

[0004] Based on this, the present invention designs an AR multi-person collaborative interactive projection system and method for public spaces to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide an AR multi-person collaborative interactive projection system and method for public spaces, in order to solve the problems mentioned in the background art.

[0006] An AR multi-person collaborative interactive projection system for public spaces includes: a multi-source spatial calibration module, configured to extract static physical feature point clouds in the public space based on real-time environmental images and depth data collected by multiple AR devices, and to perform global registration of the feature point clouds using a weighted nearest neighbor iterative algorithm to generate a unified world coordinate system. The multi-source spatial calibration module is also used to detect the relative pose deviation between AR devices. When the deviation exceeds a preset spatial consistency threshold, a dynamic recalibration process is triggered. The collaborative perception synchronization unit, connected to the multi-source spatial calibration module, is used to broadcast the pose data of the same virtual interactive object in the unified world coordinate system and receive user operation instructions uploaded by various AR devices. The collaborative perception synchronization unit includes a conflict resolution submodule, which is configured to: when receiving concurrent operation instructions from different AR devices that act on the same virtual object, calculate the comprehensive priority of each operation instruction based on the order of operation timestamps and the spatial distance between the operation initiation location and the virtual object, wherein the instruction initiated earlier and the initiation location is closer to the virtual object, obtains a higher priority; arbitrate according to the comprehensive priority and output a unique legal operation result, and at the same time return an operation conflict prompt to the AR device that initiated the illegal operation instruction; An adaptive interactive projection engine, connected to the collaborative perception synchronization unit, includes: The field-of-view adapting renderer is used to obtain the real-time field-of-view angle parameters and screen resolution of each AR device, and dynamically adjust the rendering range and level of detail of the virtual content so that the virtual content is rendered at the highest resolution only within the current field of view of the device. When the device's field of view is lower than the preset wide field of view threshold, a guide mark pointing to the adjacent virtual content is generated at the edge of the screen. The posture compensation interactor is used to collect real-time six-DOF posture data of the user's head and hands, and map the hand ray or gaze point coordinates to the interactive area of ​​the virtual object according to the unified world coordinate system. The posture compensation interactor also includes a fatigue prediction submodule, which is configured to automatically activate the eye-tracking or voice-assisted interaction channel when it detects that the user's continuous hand-raising operation exceeds a preset duration threshold.

[0007] Preferably, the multi-source space calibration module further includes: The feature point cloud dynamic filter is configured to remove instantaneous feature points generated by moving objects or dynamic lighting when extracting the static physical feature point cloud, and only retain corner points, edges and texture features that are stable within a preset time window. The coordinate system aging compensator, connected to the feature point cloud dynamic filter, is used to record the time length of the unified world coordinate system since its generation. When the time length exceeds the preset reconstruction cycle threshold, the multi-source spatial calibration module is automatically triggered to re-execute global registration to generate an updated unified world coordinate system, thereby eliminating the cumulative registration error caused by the slow changes in the public space physical environment.

[0008] Preferably, the attitude compensation interactor further includes: The collaborative gesture whitelist submodule stores a preset set of multi-user collaborative operation gestures. Each gesture in the set is associated with an operation intent tag and the maximum number of users allowed to execute the gesture simultaneously. The posture compensation interactor is configured to: when it is recognized that the user's current gesture belongs to the set of multi-user collaborative operation gestures stored in the collaborative gesture whitelist submodule, send the operation intent tag and user identifier corresponding to the gesture to the collaborative perception synchronization unit, and when the number of concurrent users executing the same operation intent tag exceeds the maximum number of users, issue a waiting prompt to the excess users. The seamless interaction modality switcher, connected to the fatigue prediction submodule, is configured to automatically switch the current user's interaction input source from hand ray to eye-tracking input or voice command input when the fatigue prediction submodule detects that the user's continuous hand-raising operation exceeds a preset duration threshold, while maintaining the continuity of the interaction operation before and after the switch, so that the user does not need to repeat the partially completed interaction action.

[0009] Preferably, the dynamic recalibration process includes: Each AR device is instructed to collect a local map of the public space at the current moment, and upload the local map to the multi-source spatial calibration module. The multi-source spatial calibration module calculates the rigid body transformation matrix between the local map and the unified world coordinate system, and sends the rigid body transformation matrix to the corresponding AR device to correct its pose, thereby eliminating spatial misalignment caused by device drift or user movement.

[0010] Preferably, the specific method by which the conflict resolution submodule calculates the comprehensive priority based on the order of operation timestamps and the spatial distance between the operation initiation location and the virtual object is as follows: Each operation instruction is assigned a comprehensive priority, which is inversely proportional to the recentity of the operation timestamp, i.e., the earlier the operation instruction is initiated, the higher the priority. It is also inversely proportional to the spatial distance from the operation initiation location to the virtual object, i.e., the closer the distance, the higher the priority. When the difference between the timestamps of two operation instructions is less than a preset time ambiguity threshold, the one with the closer spatial distance is given priority.

[0011] An AR multi-person collaborative interactive projection method for public spaces includes the following steps: S1. Spatial initialization step: Through the multi-source spatial calibration module, based on the real-time environmental images and depth data collected by multiple AR devices, the static physical feature point cloud in the public space is extracted, and the weighted nearest neighbor iterative algorithm is used to perform global registration of the feature point cloud to generate a unified world coordinate system. S2. Collaborative synchronization step: The pose data of the virtual interactive object is broadcast in the unified world coordinate system through the collaborative perception and synchronization unit, and user operation instructions uploaded by each AR device are received. When concurrent operation instructions from different AR devices that act on the same virtual object are received, the comprehensive priority of each operation instruction is calculated by the conflict resolution submodule based on the order of operation timestamps and the spatial distance between the operation initiation location and the virtual object. The submodule arbitrates according to the priority and outputs a unique legal operation result, and returns an operation conflict prompt to the AR device that initiated the illegal operation instruction. S3. Adaptive Interaction Step: The following sub-steps are executed through the adaptive interactive projection engine: S3.1 Field of view adaptation rendering: The field of view adaptation renderer obtains the real-time field of view angle parameters and screen resolution of each AR device, dynamically adjusts the rendering range and level of detail of the virtual content, so that the virtual content is rendered at the highest resolution only within the current field of view angle of the device, and when the field of view angle of the device is lower than the preset wide field of view threshold, a guide mark pointing to the adjacent virtual content is generated at the edge of the screen. S3.2 Posture Compensation Interaction: The posture compensation interactor collects real-time six-degree-of-freedom posture data of the user's head and hands, maps the hand ray or gaze point coordinates to the interactive area of ​​the virtual object according to the unified world coordinate system, and detects whether the user's continuous hand-raising operation exceeds a preset time threshold through the fatigue prediction submodule. If it exceeds the threshold, the eye-tracking or voice-assisted interaction channel is automatically activated.

[0012] Preferably, step S1 further includes a dynamic recalibration sub-step: S1.1 When the multi-source spatial calibration module detects that the relative pose deviation between each AR device exceeds the preset spatial consistency threshold, it instructs each AR device to collect the local map of the public space at the current moment and upload it to the multi-source spatial calibration module. S1.2 Calculate the rigid body transformation matrix between the local map and the unified world coordinate system through the multi-source spatial calibration module, and send the rigid body transformation matrix to the corresponding AR device to correct its pose and eliminate spatial misalignment caused by device drift or user movement.

[0013] Preferably, step S1 further includes a periodic aging compensation sub-step: S1.3. The time length of the unified world coordinate system since its generation is recorded by the coordinate system aging compensator. When the time length exceeds the preset reconstruction cycle threshold, the multi-source spatial calibration module is automatically triggered to re-execute global registration and generate an updated unified world coordinate system. In the process of extracting the static physical feature point cloud through the multi-source spatial calibration module, the feature point cloud dynamic filter removes instantaneous feature points generated by moving objects or dynamic lighting, and only retains corner points, edges and texture features that are stable within a preset time window.

[0014] Preferably, step S3.2 further includes a collaborative gesture management and modal switching sub-step: S3.2.1 The collaborative gesture whitelist submodule stores a preset set of multi-user collaborative operation gestures. Each gesture in the set is associated with an operation intent tag and the maximum number of users allowed to execute the gesture simultaneously. When the posture compensation interactor recognizes that the user's current gesture belongs to the set of multi-user collaborative operation gestures stored in the collaborative gesture whitelist submodule, it sends the operation intent tag and user identifier corresponding to the gesture to the collaborative perception synchronization unit. When the number of concurrent users executing the same operation intent tag exceeds the maximum number of users, it sends a waiting prompt to the excess users. S3.2.2 When the fatigue prediction submodule detects that the user's continuous hand-raising operation exceeds the preset duration threshold, the interaction modality seamless switcher automatically switches the current user's interaction input source from hand ray to eye-tracking input or voice command input, while maintaining the continuity of the interaction operation before and after the switch, so that the user does not need to repeat the partially completed interaction action.

[0015] Compared with the prior art, the advantages of this invention are: 1. This invention uses a multi-source spatial calibration module to perform global registration of static physical feature point clouds collected by multiple AR devices using a weighted nearest neighbor iterative algorithm to generate a unified world coordinate system, and detects the relative pose deviation between each AR device. When the deviation exceeds a preset threshold, a dynamic recalibration process is triggered, which effectively solves the problem of inconsistent coordinate systems of multiple AR devices in a public space and significantly improves the spatial consistency and registration accuracy of multi-person collaboration.

[0016] 2. This invention uses a conflict resolution submodule in the collaborative sensing synchronization unit to calculate the comprehensive priority of each operation instruction based on the order of operation timestamps and the spatial distance between the operation initiation location and the virtual object. It then arbitrates according to priority and outputs a unique legal operation result, while returning conflict prompts to illegal operation instructions. This effectively solves the problem of instruction conflict when multiple users operate on the same virtual object concurrently.

[0017] 3. This invention obtains the real-time field of view parameters and screen resolution of each AR device through the field of view adaptation renderer in the adaptive interactive projection engine, dynamically adjusts the rendering range and level of detail of the virtual content, and generates a guide mark pointing to the adjacent virtual content at the edge of the screen when the field of view is lower than the wide field of view threshold, which significantly optimizes the perception capability of virtual content under limited field of view in public spaces.

[0018] 4. This invention collects six-degree-of-freedom posture data of the user's head and hands through a posture compensation interactive device, and maps the hand ray or gaze point coordinates to the interactive area of ​​the virtual object according to a unified world coordinate system. At the same time, the fatigue prediction submodule automatically activates the eye-tracking or voice-assisted interaction channel when it detects that the user's continuous hand-raising operation exceeds a preset time threshold, which effectively alleviates the problem of limb fatigue caused by long-term AR interaction and improves the comfort and accessibility of the interaction.

[0019] 5. This invention uses a feature point cloud dynamic filter to remove instantaneous feature points generated by moving objects or dynamic lighting, retaining only stable corner points, edges, and texture features. Combined with a coordinate system aging compensator, it automatically triggers global registration to update the world coordinate system when the reconstruction cycle threshold is exceeded. This effectively eliminates the cumulative registration error caused by dynamic environmental changes or long-term operation in public spaces, and enhances the robustness and long-term stability of the system. Attached Figure Description

[0020] Figure 1 This is a system diagram of the AR multi-person collaborative interactive projection system for public spaces proposed in this invention. Figure 2 This is a flowchart of the AR multi-person collaborative interactive projection method for public spaces proposed in this invention. Detailed Implementation

[0021] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1

[0023] Scenario Description: In an ancient artifact exhibition area of ​​a museum, three visitors (users A, B, and C) each wear AR glasses and jointly view and operate a virtual 3D bronze artifact model.

[0024] Implementation process: Space initialization and calibration: After the three AR devices are activated, they collect real-time environmental images and depth data of the exhibition area. The multi-source spatial calibration module extracts static physical feature point clouds of the exhibition area, background wall, etc. (approximately 300 feature points per square meter, and points with position fluctuations of less than 1cm within 1 second are considered static).

[0025] The server executes a weighted nearest neighbor iterative algorithm for global registration, generating a unified world coordinate system with the origin set as the physical center of the booth, and the registration error controlled within 0.5cm.

[0026] When user A moves and causes the relative pose deviation of their device to exceed 3cm, the system triggers dynamic recalibration, instructing their device to collect the current local map, calculate the rigid body transformation matrix and issue a correction, so that the virtual model and the physical display stand are realigned in user A's view.

[0027] Collaborative interaction and conflict resolution: The server broadcasts the pose data of the bronze artifact model to all users in a unified world coordinate system.

[0028] User A and User B simultaneously (time difference < 50ms) issued the "rotate" command, which was applied to the bronze artifacts respectively. The conflict resolution submodule calculates the overall priority: User A's command timestamp is earlier (20ms earlier), and the hand ray is 0.3 meters away from the virtual object.

[0029] User B's command timestamp was slightly later, and the hand ray distance was 0.5 meters.

[0030] If the time difference is less than the preset 100ms ambiguity threshold, the system prioritizes the one with the closer spatial distance and determines that user A's operation is a legal operation. The bronze artifact rotates according to user A's gesture.

[0031] User B's AR view immediately displays a red, semi-transparent message that reads "Operation conflict, please try again later," which disappears automatically after 1.5 seconds.

[0032] Adaptive rendering and pose compensation: Field of view adaptation: When user C turns to look at the other side of the exhibition hall, the field of view of their AR device changes from 70° to 30°. The field of view adaptation renderer detects that the field of view is lower than the wide field of view threshold (preset 45°) and generates a flashing blue arrow guide on the right edge of the screen, pointing to the location of the bronze artifact; at the same time, the number of rendered polygons of the bronze artifact model is reduced from 500,000 to 100,000, retaining only the basic outline, saving graphics card resources.

[0033] Fatigue prediction: User A continuously holds their hand in front of them and performs fine rotation for 7 seconds (exceeding the preset duration threshold of 5 seconds). The fatigue prediction submodule is activated, and an automatic pop-up window asks "Switch to voice control?" After User A confirms, hand ray interaction is disabled, and voice commands (such as "turn 10 degrees to the left" or "zoom in") take over the interaction, and User A does not need to reposition the current rotation angle.

[0034] This embodiment fully demonstrates the advantages of AR multi-user collaborative interactive projection systems for public spaces in static museum exhibition scenarios. Through the dynamic recalibration capability of the multi-source spatial calibration module, spatial consistency is ensured during long-term, mobile viewing. The time-distance dual-factor priority arbitration mechanism of the conflict resolution submodule effectively solves the problem of multiple users simultaneously operating the same cultural relic. Combined with field-of-view adaptation rendering and fatigue prediction functions, the physiological burden on users is significantly reduced while ensuring a highly immersive experience.

[0035] Example 2

[0036] Scene description: In a city planning exhibition hall, four planners work together around a large physical sand table, using AR headsets to edit a virtual city model (including roads, buildings, and pipelines) superimposed on the sand table.

[0037] Implementation process: Collaborative gesture whitelist management: The system has a pre-defined whitelist of collaborative gestures: the gesture "pinch + drag" is associated with the "move building" intent, with a maximum of 2 concurrent users; the gesture "draw a circle with your index finger" is associated with the "rotate view" intent, with a maximum of 4 concurrent users.

[0038] Three planners simultaneously used pinch-and-drag gestures to move three different virtual building models. After recognizing the gestures, the gesture compensation interactor sent the intent label and user ID to the collaborative perception synchronization unit. Since the maximum concurrent users for moving buildings is two, the system issued a voice and text prompt to the third planner, "The current number of users is full, please wait," until one of the planners completed the operation.

[0039] Periodic aging compensation: The system has been running continuously for more than the reconstruction cycle threshold (preset 10 minutes). Due to the slow change in shadows on the sand table caused by the movement of sunlight, minor registration errors have accumulated. The coordinate system aging compensator automatically triggers the multi-source spatial calibration module to re-perform global registration.

[0040] During the re-registration process, the feature point cloud dynamic filter eliminated moving light spot feature points caused by changes in the angle of sunlight, retaining only the corner points and textures of the sand table edges and fixed landmarks. An updated world coordinate system was generated, reducing the error from 2cm to 0.8cm and eliminating the sense of slow drift in the model.

[0041] Off-field content guidance: Planner D focused on the northwest corner of the model, where her AR device's field of view was only 40°. The field-of-view renderer generated two semi-transparent guide bars at the edge of the screen: one pointing to the unrendered outline of the "commercial area" to the east and the other to the unrendered highlighted "subway station" to the south. As Planner D turned her head, these areas were dynamically loaded and rendered at the highest resolution, achieving "what you see is what you get" on-demand rendering.

[0042] Seamless modal switching: Planner B continuously raised his hand to operate the virtual pipeline connection for more than 10 seconds, and the system automatically activated eye tracking as auxiliary input. When his gaze lingered on a certain building for more than 0.5 seconds, the system automatically selected the building and popped up a voice prompt, "Selected, please issue an instruction." Planner B lowered his arm and switched to voice commands, saying "Increase the floor area ratio." The system maintained the continuity of interaction and directly adjusted the parameters of the selected building without requiring a re-operation.

[0043] This embodiment verifies the engineering practicality of the system in complex, long-term professional collaborative work scenarios. The collaborative gesture whitelist sub-module effectively avoids operational chaos when multiple people are editing by precisely controlling the number of concurrent users. The coordinate system aging compensator combined with the feature point cloud dynamic filter solves the problem of long-term accumulated error caused by environmental changes (such as light drift), ensuring the spatial accuracy of professional planning work. The out-of-view content guidance and seamless mode switching functions significantly improve information acquisition efficiency and user operation comfort in large-scale, high-information-density sand table operations.

[0044] Example 3

[0045] Scenario Description: In a public corridor of an office building, five employees and a safety commander conduct a fire escape drill using AR equipment. Virtual flames, smoke, and evacuation indicator lights are overlaid on the real environment.

[0046] Implementation process: Dynamic recalibration for mobile environments: After the exercise began, the employees' rapid running caused the AR device's pose estimation to drift. The multi-source spatial calibration module detected that the pose deviation of the two employees' devices relative to the world coordinate system exceeded 5cm (the spatial consistency threshold), and immediately triggered dynamic recalibration.

[0047] The instruction was issued: the drifting device was required to collect and upload local maps of the current corridor corners, fire hydrant boxes, etc. The server calculated the transformation matrix and issued it, correcting the pose within 0.3 seconds. The evacuation arrows in the employees' eyes always accurately covered the ground guide lines, without any offset or ghosting.

[0048] High-concurrency conflict resolution: A virtual fire extinguisher object is located at the virtual fire site. Three employees reach for the object almost simultaneously. The conflict resolution submodule receives three operation commands: Employee A: Earliest timestamp (t=0ms), distance 0.4 meters.

[0049] Employee B: The timestamp is 15ms late, and the distance is 0.45 meters.

[0050] Employee C: Timestamp 18ms later, distance 0.35 meters (closer distance but slightly later time).

[0051] Since the time difference of the three instructions was all within the 100ms ambiguity threshold, the system ultimately determined that employee C, who was closest in spatial distance, obtained the virtual fire extinguisher. Employees A and B received a notification that "the fire extinguisher has been used by someone else" and saw that the fire extinguisher model in employee C's hand was activated with a dry ice spray effect.

[0052] Fatigue prediction and accessible interaction: An employee with limited mobility needs to keep their hand raised to activate the "Emergency Help" button. After raising their hand for 8 seconds, the fatigue prediction submodule automatically activates the voice-assisted interaction channel. The system prompts: "You can now call for help via voice." The employee simply says "I need help," and the system sends their precise location and help signal to the commander's interface, without requiring them to maintain the raised hand posture.

[0053] This embodiment focuses on verifying the reliability and barrier-free interaction capabilities of the present invention in dynamic, high-pressure emergency scenarios. The rapid response of the dynamic recalibration process ensures the accurate alignment of virtual guidance information with the physical environment during rapid user movement, which is crucial for the accuracy of safe evacuation routes. In scenarios with highly concurrent and competing resources (such as virtual fire extinguishers), the conflict resolution submodule provides clear, fair, and predictable arbitration results, avoiding delays or confusion caused by operational conflicts. The automatic activation of fatigue prediction and auxiliary interaction channels provides barrier-free operation access for people with special needs.

[0054] Example 4

[0055] Scene description: In front of a large interactive projection wall in a children's science museum, multiple children use lightweight AR glasses (or mobile AR) to participate in a "virtual jigsaw puzzle" game.

[0056] Implementation process: Lightweight spatial calibration and feature selection: Because children are active and energetic, there is a lot of moving interference in the environment (other running children, waving arms). The feature point cloud dynamic filter of the multi-source spatial calibration module sets the time window to 2 seconds, retaining only the wall texture and corner features that are stable within this window, and removing the instantaneous feature points of moving people in real time to ensure that the world coordinate system is not frequently disturbed.

[0057] Conflict resolution tips and techniques for children: Two children simultaneously reach out and tap the same virtual puzzle piece on the screen. The conflict resolution submodule calculates the priority: Child A: Click timestamp t=0ms, the click coordinates mapped by the hand ray are 0.1 meters away from the center of the puzzle piece.

[0058] Child B: Click timestamp t=80ms, click distance 0.12 meters.

[0059] If the system determines the time difference is less than 100ms, it prioritizes the closer spatial distance, and child A gains control of the puzzle piece. A cartoon panda appears on the edge of child B's AR interface, accompanied by a regretful emoticon bubble: "Oops, someone else got it first. Let's try another one!" Simultaneously, the puzzle piece is grayed out for 5 seconds to indicate it's locked.

[0060] Field of view adaptation and ultra-wide field of view guidance: Child C's AR glasses have a field of view of only 50° (the wide field of view threshold is preset to 70°). The field-of-view adaptor renderer detected that its field of view did not cover the right 1 / 3 of the puzzle wall, so it generated a string of "twinkling stars" guide icons on the right edge of the screen to attract child C to turn their head to the right. When child C turned their head, the right puzzle area was immediately rendered at the highest resolution (1080p texture for each puzzle piece), while the resolution of the left edge area was dynamically reduced to balance performance.

[0061] Collaborative gesture whitelist and concurrency control: The game rules allow a maximum of two children to simultaneously drag different puzzle pieces to complete the puzzle. The whitelist defaults to associating the "grab" gesture with the "move puzzle" intent, with a maximum of 2 concurrent users.

[0062] When a third child attempts to grab the third puzzle piece, the gesture compensation interactor recognizes the gesture and checks the whitelist. Finding that the limit has been reached, it displays a "hand blocked" animation on the child's puzzle piece and announces "Waiting, please wait." Once one child completes the puzzle and releases the piece, the waiting child's gesture immediately takes effect, gaining control and achieving smooth queuing collaboration.

[0063] This embodiment focuses on the performance of the invention in lightweight public entertainment and education scenarios for the general public, especially children. The feature point cloud dynamic filter effectively filters out unavoidable dynamic interferences in public spaces (such as crowds and body movements), providing a stable underlying spatial coordinate system for game interaction. The cartoon-style conflict prompts and visual guidance design demonstrate that the conflict resolution and field of view adaptation modules can be customized to provide a user-friendly experience according to specific application scenarios. The collaborative gesture whitelist mechanism realizes a simple and intuitive queuing collaboration logic, meeting the high-concurrency and low-threshold interaction needs in public facilities.

[0064] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An AR multi-person collaborative interactive projection system for public spaces, characterized in that: include: The multi-source spatial calibration module is configured to extract static physical feature point clouds in a common space based on real-time environmental images and depth data collected by multiple AR devices, and to perform global registration of the feature point clouds using a weighted nearest neighbor iterative algorithm to generate a unified world coordinate system. The multi-source spatial calibration module is also used to detect the relative pose deviation between AR devices. When the deviation exceeds a preset spatial consistency threshold, a dynamic recalibration process is triggered. The collaborative perception synchronization unit, connected to the multi-source spatial calibration module, is used to broadcast the pose data of the same virtual interactive object in the unified world coordinate system and receive user operation instructions uploaded by various AR devices. The collaborative perception synchronization unit includes a conflict resolution submodule, which is configured to: when receiving concurrent operation instructions from different AR devices that act on the same virtual object, calculate the comprehensive priority of each operation instruction based on the order of operation timestamps and the spatial distance between the operation initiation location and the virtual object, wherein the instruction initiated earlier and the initiation location is closer to the virtual object, obtains a higher priority; arbitrate according to the comprehensive priority and output a unique legal operation result, and at the same time return an operation conflict prompt to the AR device that initiated the illegal operation instruction; An adaptive interactive projection engine, connected to the collaborative perception synchronization unit, includes: The field-of-view adapting renderer is used to obtain the real-time field-of-view angle parameters and screen resolution of each AR device, and dynamically adjust the rendering range and level of detail of the virtual content so that the virtual content is rendered at the highest resolution only within the current field of view of the device. When the device's field of view is lower than the preset wide field of view threshold, a guide mark pointing to the adjacent virtual content is generated at the edge of the screen. The posture compensation interactor is used to collect real-time six-DOF posture data of the user's head and hands, and map the hand ray or gaze point coordinates to the interactive area of ​​the virtual object according to the unified world coordinate system. The posture compensation interactor also includes a fatigue prediction submodule, which is configured to automatically activate the eye-tracking or voice-assisted interaction channel when it detects that the user's continuous hand-raising operation exceeds a preset duration threshold.

2. The AR multi-person collaborative interactive projection system for public spaces according to claim 1, characterized in that, The multi-source space calibration module also includes: The feature point cloud dynamic filter is configured to remove instantaneous feature points generated by moving objects or dynamic lighting when extracting the static physical feature point cloud, and only retain corner points, edges and texture features that are stable within a preset time window. The coordinate system aging compensator, connected to the feature point cloud dynamic filter, is used to record the time length of the unified world coordinate system since its generation. When the time length exceeds the preset reconstruction cycle threshold, the multi-source spatial calibration module is automatically triggered to re-execute global registration to generate an updated unified world coordinate system, thereby eliminating the cumulative registration error caused by the slow changes in the public space physical environment.

3. The AR multi-person collaborative interactive projection system for public spaces according to claim 1, characterized in that, The attitude compensation interactor also includes: The collaborative gesture whitelist submodule stores a preset set of multi-user collaborative operation gestures. Each gesture in the set is associated with an operation intent tag and the maximum number of users allowed to execute the gesture simultaneously. The posture compensation interactor is configured to: when it is recognized that the user's current gesture belongs to the set of multi-user collaborative operation gestures stored in the collaborative gesture whitelist submodule, send the operation intent tag and user identifier corresponding to the gesture to the collaborative perception synchronization unit, and when the number of concurrent users executing the same operation intent tag exceeds the maximum number of users, issue a waiting prompt to the excess users. The seamless interaction modality switcher, connected to the fatigue prediction submodule, is configured to automatically switch the current user's interaction input source from hand ray to eye-tracking input or voice command input when the fatigue prediction submodule detects that the user's continuous hand-raising operation exceeds a preset duration threshold, while maintaining the continuity of the interaction operation before and after the switch, so that the user does not need to repeat the partially completed interaction action.

4. The AR multi-person collaborative interactive projection system for public spaces according to claim 1, characterized in that, The dynamic recalibration process includes: Each AR device is instructed to collect a local map of the public space at the current moment, and upload the local map to the multi-source spatial calibration module. The multi-source spatial calibration module calculates the rigid body transformation matrix between the local map and the unified world coordinate system, and sends the rigid body transformation matrix to the corresponding AR device to correct its pose, thereby eliminating spatial misalignment caused by device drift or user movement.

5. The AR multi-person collaborative interactive projection system for public spaces according to claim 1, characterized in that, The specific method by which the conflict resolution submodule calculates the overall priority based on the order of operation timestamps and the spatial distance between the operation initiation location and the virtual object is as follows: Each operation instruction is assigned a comprehensive priority, which is inversely proportional to the recentity of the operation timestamp, i.e., the earlier the operation instruction is initiated, the higher the priority. It is also inversely proportional to the spatial distance from the operation initiation location to the virtual object, i.e., the closer the distance, the higher the priority. When the difference between the timestamps of two operation instructions is less than a preset time ambiguity threshold, the one with the closer spatial distance is given priority.

6. An AR multi-person collaborative interactive projection method for public spaces, using any of the AR multi-person collaborative interactive projection systems for public spaces as described in claims 1-5, characterized in that, Includes the following steps: S1. Spatial initialization step: Through the multi-source spatial calibration module, based on the real-time environmental images and depth data collected by multiple AR devices, the static physical feature point cloud in the public space is extracted, and the weighted nearest neighbor iterative algorithm is used to perform global registration of the feature point cloud to generate a unified world coordinate system. S2. Collaborative synchronization step: The pose data of the virtual interactive object is broadcast in the unified world coordinate system through the collaborative perception and synchronization unit, and user operation instructions uploaded by each AR device are received. When concurrent operation instructions from different AR devices that act on the same virtual object are received, the comprehensive priority of each operation instruction is calculated by the conflict resolution submodule based on the order of operation timestamps and the spatial distance between the operation initiation location and the virtual object. The submodule arbitrates according to the priority and outputs a unique legal operation result, and returns an operation conflict prompt to the AR device that initiated the illegal operation instruction. S3. Adaptive Interaction Step: The following sub-steps are executed through the adaptive interactive projection engine: S3.1 Field of view adaptation rendering: The field of view adaptation renderer obtains the real-time field of view angle parameters and screen resolution of each AR device, dynamically adjusts the rendering range and level of detail of the virtual content, so that the virtual content is rendered at the highest resolution only within the current field of view angle of the device, and when the field of view angle of the device is lower than the preset wide field of view threshold, a guide mark pointing to the adjacent virtual content is generated at the edge of the screen. S3.2 Posture Compensation Interaction: The posture compensation interactor collects real-time six-degree-of-freedom posture data of the user's head and hands, maps the hand ray or gaze point coordinates to the interactive area of ​​the virtual object according to the unified world coordinate system, and detects whether the user's continuous hand-raising operation exceeds a preset time threshold through the fatigue prediction submodule. If it exceeds the threshold, the eye-tracking or voice-assisted interaction channel is automatically activated.

7. The AR multi-person collaborative interactive projection method for public spaces according to claim 6, characterized in that, Step S1 further includes a dynamic recalibration sub-step: S1.1 When the multi-source spatial calibration module detects that the relative pose deviation between each AR device exceeds the preset spatial consistency threshold, it instructs each AR device to collect the local map of the public space at the current moment and upload it to the multi-source spatial calibration module. S1.2 Calculate the rigid body transformation matrix between the local map and the unified world coordinate system through the multi-source spatial calibration module, and send the rigid body transformation matrix to the corresponding AR device to correct its pose and eliminate spatial misalignment caused by device drift or user movement.

8. The AR multi-person collaborative interactive projection method for public spaces according to claim 6, characterized in that, Step S1 further includes a periodic aging compensation sub-step: S1.

3. The time length of the unified world coordinate system since its generation is recorded by the coordinate system aging compensator. When the time length exceeds the preset reconstruction cycle threshold, the multi-source spatial calibration module is automatically triggered to re-execute global registration and generate an updated unified world coordinate system. In the process of extracting the static physical feature point cloud through the multi-source spatial calibration module, the feature point cloud dynamic filter removes instantaneous feature points generated by moving objects or dynamic lighting, and only retains corner points, edges and texture features that are stable within a preset time window.

9. The AR multi-person collaborative interactive projection method for public spaces according to claim 6, characterized in that, Step S3.2 also includes a collaborative gesture management and modality switching sub-step: S3.2.1 Store a preset set of multi-user collaborative operation gestures through the collaborative gesture whitelist sub-module. Each gesture in the set is associated with an operation intent tag and the maximum number of users allowed to execute the gesture simultaneously. When the posture compensation interactor recognizes that the user's current gesture belongs to the set of multi-person collaborative operation gestures stored in the collaborative gesture whitelist sub-module, it sends the operation intent tag and user identifier corresponding to the gesture to the collaborative perception synchronization unit, and when the number of concurrent users executing the same operation intent tag exceeds the maximum number of users, it sends a waiting prompt to the excess users. S3.2.2 When the fatigue prediction submodule detects that the user's continuous hand-raising operation exceeds the preset duration threshold, it automatically switches the current user's interaction input source from hand ray to eye-tracking input or voice command input through the interaction modality seamless switcher, while maintaining the continuity of the interaction operation before and after the switch, so that the user does not need to repeat the partially completed interaction action.

Images