A multi-person collaborative interaction method and system in a virtual-real fusion scene

Abstract

The application relates to the technical field of virtual reality and augmented reality, and discloses a multi-person cooperative interaction method and system in a virtual and real fusion scene, which acquires three-dimensional geometric structure and semantic information of a physical space, loads a graph structure of a virtual scene and a current plot state vector, establishes an elastic mapping relationship between the physical space and the virtual scene, and generates a space anchor point coordinate system; according to a current cooperative task demand vector and real-time pose distribution of each user in the physical space, a differentiated role is dynamically allocated to each user; under the constraint of the elastic mapping relationship, multi-person cooperative operation is carried out based on the space anchor point coordinate system and natural gesture interaction; elastic mapping parameters, allocation states and multi-user interaction operation sequences are collected in real time, a multi-agent cooperative state space is constructed, a cooperative task completion degree and interaction fluency are taken as reward targets, an optimization strategy is output, and a task allocation scheme is dynamically corrected; and the application supports multi-person deep cooperative interaction.

Description

Technical Field

[0001] This invention relates to the field of virtual reality and augmented reality technology, specifically to a multi-person collaborative interaction method and system in a virtual-real fusion scenario. Background Technology

[0002] With the rapid development of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies, multi-person collaborative interaction in virtual-real fusion scenarios has become a research hotspot in the field of human-computer interaction. Existing virtual-real fusion systems generally adopt rigid spatial mapping strategies, that is, a fixed mapping that corresponds one-to-one between spatial positions in the virtual scene and physical positions. When physical space is limited, users cannot freely explore large-scale scenes in the virtual world. When multiple users are in physical spaces of different sizes and layouts, the forced uniform spatial mapping severely restricts the natural activities of some users. The current interaction methods in multi-person virtual-real fusion systems mainly rely on controllers, buttons, or simple voice commands, resulting in a single interaction modality that is difficult to support team collaboration tasks that require fine coordination. Summary of the Invention

[0003] The purpose of this invention is to solve the above problems by designing a multi-person collaborative interaction method and system for virtual-real fusion scenarios.

[0004] The first aspect of this invention provides a method for multi-person collaborative interaction in a virtual-real fusion scenario, the method comprising the following steps: Obtain the three-dimensional geometric structure and semantic information of the physical space, and load the graph structure of the virtual scene and the current plot state vector; Based on the plot state vector, the semantic partitions of the physical space and the semantic regions of the virtual scene are dynamically and optimally paired to establish an elastic mapping relationship between the physical space and the virtual scene, and a corresponding spatial anchor point coordinate system is generated based on the user's real-time pose. Based on the current collaborative task requirements and the real-time pose distribution of each user in the physical space, dynamically assign differentiated roles to each user. Under the constraints of the elastic mapping relationship, multi-person collaborative operation is carried out based on the spatial anchor point coordinate system and natural gesture interaction; Real-time acquisition of elastic mapping parameters, allocation status, and multi-user interaction operation sequences is used to construct a multi-agent collaborative state space. With the completion rate of collaborative tasks and the smoothness of interaction as reward objectives, the MAPPO algorithm is used to output optimization strategies and dynamically correct task allocation schemes.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of obtaining the three-dimensional geometric structure and semantic information of the physical space, and loading the graph structure of the virtual scene and the current plot state vector, includes: Acquire color and depth images of the physical space, determine camera pose, and optimize global trajectory using bundle adjustment; Based on the optimized camera pose, the depth image is back-projected into a 3D point cloud, and a voxel fusion algorithm with a truncated signed distance function is used to generate a dense voxel mesh. The traveling cube algorithm is applied to extract the triangular mesh surface model from the dense voxel mesh, and the region growing segmentation algorithm is used to divide the surface into physical space regions with semantic labels. Load the virtual scene description file, extract all spatial nodes and their connectivity relationships in the virtual scene, and construct the directed acyclic graph structure of the virtual scene; Obtain the current plot progress status, read the list of unlocked scene nodes, the list of untriggered plot events, and the active narrative branch identifiers, and combine them into the current plot status vector.

[0006] Optionally, in a second implementation of the first aspect of the present invention, based on the plot state vector, the semantic partitions of the physical space and the semantic regions of the virtual scene are dynamically and optimally paired to establish an elastic mapping relationship between the physical space and the virtual scene, and a corresponding spatial anchor point coordinate system is generated based on the user's real-time pose, including: Extract the geometric features of each semantic region in the physical space and the functional semantic features of each node in the virtual scene, and combine them with the narrative priority parameters of each virtual region in the current plot state vector to construct the cost matrix between the physical region and the virtual region. With the goal of minimizing the total matching cost, the Hungarian algorithm is used to solve for the optimal pairing scheme between physical and virtual regions; Establish an elastic mapping lookup table based on the optimal pairing scheme, record the correspondence between each physical region identifier and its mapped virtual region identifier, and use it as the elastic mapping relationship at the current moment. Obtain the real-time three-dimensional position coordinates and head pose of each user in the physical space, and convert the physical coordinates of each user into the corresponding virtual space coordinates according to the elastic mapping lookup table; Each user's independent local spatial anchor point coordinate system is established with the user's physical location as the origin, and all local coordinate systems are uniformly transformed to the same world coordinate system to form a shared spatial anchor point system.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of dynamically assigning differentiated roles to each user based on the current collaborative task's demand vector and the real-time pose distribution of each user in the physical space includes: Extract the current collaborative task identifier from the current story state vector, and obtain the minimum requirement level of each ability dimension required to complete the collaborative task through the corresponding task description file to form a requirement vector. Three character templates are preset: Explorer, Guardian and Puzzle Solver. Each template is pre-configured with corresponding baseline values ​​and operational efficiency coefficients for each ability dimension. Calculate the matching score between each user and each role template, and select the role template with the highest matching score as the initial candidate role for that user; Determine whether the combination of all initial candidate roles for users meets all capability dimensions in the demand vector. If there is a capability gap, perform a role upgrade operation on the user with the second highest matching degree. With the goal of maximizing the sum of the matching degree of all users' ability vectors and demand vectors, and with the uniqueness constraint of role allocation as a condition, the optimal role allocation matrix is ​​determined, and a corresponding subset of collaborative tasks is generated for each user.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the step of performing multi-person collaborative operation based on the spatial anchor point coordinate system and natural gesture interaction under the constraint of the elastic mapping relationship includes: Hand region detection and 3D key point regression are performed on hand images collected by each user's device. The key point coordinates are transformed to the spatial anchor point coordinate system, and the key point sequence of multiple consecutive frames is temporally encoded to identify the current dynamic gesture category. After detecting a grasping gesture, determine whether the virtual object near the hand is a graspable object. If so, establish a temporary binding relationship between the hand and the object and control the object to follow the hand's movement. Upon detecting a release gesture, the binding relationship is removed, the virtual object is fixed to the current spatial anchor point, the object status is updated to be pickable, and a status change message is broadcast to all users. When it is detected that the hands of at least two users are simultaneously within the preset radius of the same spatial anchor point, the role type of each relevant user is read and the preset collaborative operation rule library is queried to determine the type of collaborative operation mode triggered. If multiple users simultaneously initiate interactions with the same object and the conditions for collaborative operation mode are not met, it is determined to be an interaction conflict event. Based on the role priority table, the user with the highest priority is selected as the initiator, and the requests of other users are rejected.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of performing hand region detection and three-dimensional keypoint regression on the hand images collected by each user's device, transforming the keypoint coordinates to a spatial anchor point coordinate system, and performing temporal encoding on a series of consecutive keypoint sequences to identify the current dynamic gesture category includes: The acquired hand images are filtered by confidence and suppressed by non-maximum to output the hand bounding box. The image block is cropped with the hand bounding box as the region of interest and the two-dimensional pixel coordinates and depth values ​​of each hand key point are determined. The three-dimensional coordinates of the key points in the camera coordinate system are obtained by inverse projection of the camera intrinsic parameters. The three-dimensional coordinates of key points are transformed to the spatial anchor point coordinate system through rigid body transformation. The coordinates of multiple consecutive frames are integrated to obtain a fixed-length buffer queue. The displacement vector and velocity vector of each key point between adjacent frames are calculated and concatenated into a temporal feature vector. The temporal feature vector is input into a temporal convolutional network for temporal dimension feature extraction and global pooling. The network is then passed through a fully connected classification network to output the probability distribution of each predefined gesture category, and the category with the highest probability is selected as the recognition result.

[0010] Optionally, in the sixth implementation of the first aspect of the present invention, the real-time acquisition of elastic mapping parameters, allocation states, and multi-user interaction operation sequences to construct a multi-agent collaborative state space, using collaborative task completion and interaction fluency as reward objectives, employing the MAPPO algorithm to output optimization strategies, and dynamically correcting the task allocation scheme, includes: Real-time acquisition of elastic mapping parameters, allocation status, and multi-user interaction operation sequences, which are then normalized and combined into joint observation features of multiple agents; The local observation features of each user are input into the corresponding Actor policy network to output the action probability distribution, and the global state value is output through the global Critic evaluation network using the joint observations and joint actions of all users as input. The global reward value is calculated by weighting the ratio of the number of sub-tasks completed by each user to the total number of sub-tasks, as well as the average response latency and error rate of interactive operations. The generalized advantage estimation algorithm is used to calculate the advantage value of each action and update the Actor-Critic network parameters. The multi-agent collaborative strategy after training convergence is deployed to each user terminal, and the recommended action for the next moment is output in real time. Continuously monitor the completion progress of each user's subtasks, and use a greedy algorithm to search for and reassign subtasks that have not been completed within the time limit to other users with the lowest current load. Dynamically update the collaborative task subsets of each user.

[0011] A second aspect of the present invention provides a multi-person collaborative interaction system in a virtual-real fusion scenario, the system comprising: The physical space perception module is used to acquire the three-dimensional geometric structure and semantic information of the physical space, and load the graph structure of the virtual scene and the current plot state vector. The elastic mapping module is used to dynamically and optimally pair the semantic partitions of the physical space with the semantic regions of the virtual scene according to the plot state vector, establish an elastic mapping relationship between the physical space and the virtual scene, and generate the corresponding spatial anchor point coordinate system based on the user's real-time pose. The role management module is used to dynamically assign differentiated roles to each user based on the current collaborative task's requirement vector and the real-time pose distribution of each user in the physical space. An interaction recognition module is used to perform multi-person collaborative operations based on the spatial anchor point coordinate system and natural gesture interaction under the constraints of the elastic mapping relationship. The collaborative optimization module is used to collect elastic mapping parameters, allocation status, and multi-user interaction operation sequences in real time, construct a multi-agent collaborative state space, use the completion rate of collaborative tasks and the smoothness of interaction as reward objectives, output optimization strategies using the MAPPO algorithm, and dynamically correct the task allocation scheme.

[0012] A third aspect of the present invention provides a multi-person collaborative interaction device in a virtual-real fusion scenario, the multi-person collaborative interaction device in a virtual-real fusion scenario including a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the multi-person collaborative interaction device in a virtual-real fusion scenario to execute the various steps of the multi-person collaborative interaction method in a virtual-real fusion scenario as described in any of the preceding claims.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the multi-person collaborative interaction method in a virtual-real fusion scenario as described in any of the preceding claims.

[0014] The technical solution provided by this invention involves acquiring the three-dimensional geometric structure and semantic information of the physical space, and loading the graph structure of the virtual scene and the current plot state vector; based on the plot state vector, dynamically and optimally pairing the semantic partitions of the physical space with the semantic regions of the virtual scene to establish an elastic mapping relationship between the physical space and the virtual scene, and generating a corresponding spatial anchor point coordinate system based on the user's real-time pose; dynamically assigning differentiated roles to each user based on the current collaborative task's requirement vector and the real-time pose distribution of each user in the physical space; under the constraints of the elastic mapping relationship, performing multi-user collaborative operations based on the spatial anchor point coordinate system and natural gesture interaction; and real-time acquisition of elastic mapping parameters, allocation status, and multi-user interaction operation sequences to construct a multi-intelligent... The system employs a collaborative state space, with task completion and interaction fluency as reward objectives. It uses the MAPPO algorithm to output optimization strategies and dynamically adjusts task allocation schemes. This invention supports deep multi-user collaborative interaction, allowing users to experience narrative exploration of a vast virtual world within a small physical space. The mapping relationship dynamically adjusts with the plot, significantly enhancing the flexibility and immersion of narrative expression. Combined with natural gesture-based interaction supported by spatial anchors, it achieves intuitive and efficient deep collaboration among multiple users, significantly improving the replay value of content and the personalized experience for users. It solves the technical problem of monotonous narrative caused by homogenized characters, enabling the system to adaptively optimize interaction strategies based on user behavior and team composition, continuously improving collaborative efficiency and user experience. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0016] Figure 1 A flowchart of a multi-person collaborative interaction method in a virtual-real fusion scenario provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a multi-person collaborative interaction system in a virtual-real fusion scenario provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a multi-person collaborative interaction device in a virtual-real fusion scenario provided by an embodiment of the present invention. Detailed Implementation

[0017] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 A flowchart of a multi-person collaborative interaction method in a virtual-real fusion scenario provided by an embodiment of the present invention is shown. The method specifically includes the following steps: Step 101: Obtain the three-dimensional geometric structure and semantic information of the physical space, and load the graph structure of the virtual scene and the current plot state vector; In this embodiment, color image sequences and depth image sequences in physical space are simultaneously acquired using a multi-view RGB-D camera, and timestamp alignment is performed on each frame. ORB feature points are extracted from the acquired color image sequences. Through feature point matching and epipolar geometry constraints, the pose transformation matrix of the camera between adjacent frames is estimated, and bundle adjustment is used to jointly optimize the global camera trajectory to obtain the precise pose of the camera at each time step. Based on the optimized camera pose, the pixels in each frame of depth image are back-projected to the world coordinate system to generate a sparse 3D point cloud in physical space. Voxel filtering is then applied to the sparse point cloud to remove outliers and noise points. A voxel fusion algorithm with a truncated signed distance function is used to fuse the depth data of multiple frames into a dense 3D voxel mesh. Each voxel in this mesh stores its signed distance value to the nearest physical surface. The traveling cube algorithm is applied to the dense voxel mesh to extract isosurfaces, generating a triangular mesh surface model in physical space. At the same time, the normal direction and curvature value of each mesh vertex are calculated. The triangular mesh surface model is segmented into regions through growth: the vertex with the smallest curvature is selected as the seed point, and the merging criterion is that the angle between the normal directions is less than a preset threshold. The physical surface is divided into multiple planar and curved regions, and each region is labeled with a semantic category such as ground, wall, tabletop, or obstacle, resulting in a 3D physical space mesh map with semantic labels. The geometric features of each semantic region are calculated, including the region boundary length, region area, region bounding box size, region center coordinates, and region principal direction vector. A preset virtual scene description file is loaded from the server, which contains all the features of the virtual scene. The spatial node information of virtual rooms and virtual passages, as well as the connectivity and directional constraints between nodes, are used to construct a directed acyclic graph structure for the virtual scene. The current plot progress status file is loaded, from which the list of currently unlocked virtual scene nodes, the list of plot events that have not yet been triggered, the trigger condition queue for each plot event, and the identifier of the currently active narrative branch are read. The above information is combined into the current plot status vector. Each virtual scene node is associated with its functional semantic label and spatial attribute parameters, where the spatial attribute parameters include the node's design area, shape outline, and maximum number of people it can accommodate in the virtual world.

[0019] Step 102: Based on the plot state vector, dynamically and optimally pair the semantic partitions of the physical space with the semantic regions of the virtual scene, establish an elastic mapping relationship between the physical space and the virtual scene, and generate the corresponding spatial anchor point coordinate system based on the user's real-time pose. In this embodiment, all semantic regions are extracted from the 3D mesh graph of the physical space to construct a set of physical regions. A region feature descriptor is generated for each physical region, which includes semantic category encoding, region area range, and region shape category. The directed acyclic graph structure of the virtual scene is loaded, and all virtual scene nodes in the graph are extracted to construct a set of virtual regions. A virtual region feature descriptor is generated for each virtual region, which includes functional semantic category encoding, virtual design area range, and virtual space usage type. Priority parameters are extracted from the current plot state vector. These priority parameters include the activation state of each virtual scene node in the current plot stage, the number of events to be triggered, and the narrative importance level. Iterate through each pair of physical and virtual regions, calculating the matching cost for each pair: calculate the semantic distance between the semantic category of the physical region and the functional semantic category of the virtual region; calculate the area difference ratio between the area of ​​the physical region and the designed area of ​​the virtual region; determine the narrative urgency level of the virtual region based on the current plot state vector; if the virtual region is a node that must be visited in the current plot, the matching cost is reduced; organize the matching costs of each region pair into a cost matrix, and use the Hungarian algorithm to solve the pairing problem one by one between physical and virtual regions with the goal of minimizing the total matching cost, to obtain the initial region pairing scheme; perform spatial continuity verification on the initial region pairing scheme: check whether adjacent regions in the physical space still maintain an adjacent relationship in the virtual space after pairing; if not, perform local swap adjustments on the pairing scheme until the spatial topology consistency meets the preset threshold; Based on the above pairing scheme, an elastic mapping lookup table is established. Each record in the lookup table stores a physical region identifier and its corresponding mapped virtual region identifier. The elastic mapping lookup table represents the elastic mapping relationship between the physical space and the virtual space at the current moment. It checks whether there are any idle physical regions that have not been paired with any virtual region. If so, the idle physical region is marked as a transition buffer for users to transition between different areas of the virtual scene. The real-time 3D position coordinates and head pose quaternions of each user in the physical space are obtained. Based on the elastic mapping lookup table, the 3D position coordinates of each user are converted into corresponding virtual space coordinates. Using the current position of each user in the physical space as the origin, independent local spatial anchor point coordinate systems are established for each user. The origin coordinates and axial directions of all local spatial anchor point coordinate systems are uniformly transformed to the same world coordinate system, forming a shared spatial anchor point system. Each user synchronizes the coordinate transformation parameters of this shared spatial anchor point system through their respective AR / MR devices.

[0020] Step 103: Based on the current collaborative task's requirement vector and the real-time pose distribution of each user in the physical space, dynamically assign differentiated roles to each user; In this embodiment, the currently active collaborative task identifier is extracted from the current storyline state vector. Based on the task identifier, the corresponding task description file is loaded from the task library. The task description file defines all capability dimensions required to complete the collaborative task and the minimum requirement level for each capability dimension, forming the requirement vector of the current collaborative task. The minimum requirement level of each capability dimension in the requirement vector is increased by a preset step to generate an enhanced requirement vector, which is used for redundant matching in subsequent role allocation. The total number of currently online users and the real-time three-dimensional coordinates of each user in the physical space are obtained. The spatial distance matrix between each pair of users is calculated, and users are spatially clustered based on the spatial distance matrix, dividing users in the same physical area into the same spatial group. Three preset role templates are used: Explorer template, Guardian template, and Puzzle solver template. Each role template is pre-configured with corresponding capability baseline values ​​and operation efficiency coefficients for each capability dimension. For each user, perform initial role matching calculation: sequentially compare the user's spatial group number and real-time pose information with the applicable scenario conditions of each role template, and exclude role templates that are not suitable for the current spatial group; for the retained role templates, calculate the matching score between the user's historical operation data performance statistics in each capability dimension and the capability baseline value of each role template, and select the role template with the highest matching score as the user's initial candidate role; check whether the combination of all users' initial candidate roles can cover all capability dimensions in the requirement vector: if there are uncovered capability dimensions, identify the capability gap of the current team, and perform a role upgrade operation on the user with the second highest matching degree in the corresponding capability dimension, changing their role to the target role that can fill the gap; perform global optimization of the role allocation scheme: with the goal of maximizing the sum of the matching degree of all users' capability vectors and requirement vectors in each dimension, and with the constraints that each user can only be assigned one role and each role can be assigned at least one user, use integer programming to solve for the optimal role allocation matrix; Based on the optimal role allocation matrix, a role configuration file is generated for each user. This configuration file records the role type identifier assigned to the user, the corresponding ability vector parameters, and the whitelist of operation permissions exclusive to that role. Based on each user's role type and ability vector, a corresponding collaborative task subset is generated for each user. This collaborative task subset contains a list of sub-tasks that the user needs to complete in the current collaborative task and the execution order constraints of each sub-task.

[0021] Step 104: Under the constraints of the elastic mapping relationship, perform multi-person collaborative operation based on the spatial anchor point coordinate system and natural gesture interaction; In this embodiment, hand region detection is performed on the RGB images transmitted from each user's AR / MR device. A target detection network is used to crop out the region of interest (ROI) containing the hand from the image. Hand keypoint regression is performed on the cropped ROI to locate the two-dimensional pixel coordinates and depth values ​​of each hand joint. The two-dimensional coordinates combined with the depth values ​​are then back-projected to the three-dimensional keypoint coordinates in the camera coordinate system. The three-dimensional keypoint coordinates of each user's hand are transformed from their respective camera coordinate system to a shared spatial anchor point coordinate system to ensure that the hand positions of all users are in the same spatial reference system. The three-dimensional keypoint sequences of each user's hand in multiple consecutive frames are temporally encoded to extract the motion trajectory vectors and motion velocity features of each keypoint, forming a dynamic gesture feature sequence. The dynamic gesture feature sequence is input into a classifier to identify the current gesture category, which includes at least grasping gestures, releasing gestures, pointing gestures, pushing gestures, rotating gestures, and confirming gestures. Based on the current gesture category and the hand's 3D position in the shared spatial anchor coordinate system, combined with the current elastic mapping relationship, the user's operational intent and the target virtual object are determined. Upon detecting a grasping gesture, it is determined whether the virtual object near the user's hand is a graspable object. If so, a temporary binding relationship is established between the user's hand and the virtual object, and the virtual object is controlled to move synchronously with the hand. Upon detecting a release gesture, the user's hand is determined to be currently bound to a virtual object and the spatial anchor position of the hand. The binding relationship is released, and the virtual object is fixed to the current spatial anchor position. At the same time, the virtual object's state is updated to a pickable state, and the state change message is broadcast to all users. The positions of each user's hand in the shared spatial anchor coordinate system are continuously monitored. When at least two users' hands are detected to be simultaneously within a preset radius of the same spatial anchor point, it is determined to be a multi-user spatiotemporal overlap event. After determining a multi-user spatiotemporal overlap event, the role type of each relevant user is read, and the collaborative operation rule base is queried based on the role type combination to determine the type of collaborative operation mode that should be triggered by the current spatiotemporal overlap event. Based on the collaborative operation mode type, the current gesture categories of each user are combined and matched. If the combination match is successful, the corresponding collaborative interaction operation is executed, including multiple users working together to lift a virtual object, multiple users pressing different mechanisms simultaneously, or multiple users passing virtual items in sequence. When multiple users are detected simultaneously initiating interaction requests to the same virtual object or the same spatial anchor point without meeting the triggering conditions of any collaborative operation mode, it is determined to be an interaction conflict event. After determining the interaction conflict event, the role type of each relevant user is read, and a preset conflict arbitration priority table is queried according to the role type. The user with the highest priority is selected as the legitimate initiator of the interaction request, the interaction requests of other users are rejected, and a visual prompt is sent to the rejected user.

[0022] Step 105: Collect elastic mapping parameters, allocation status, and multi-user interaction operation sequences in real time to construct a multi-agent collaborative state space. With the completion rate of collaborative tasks and the smoothness of interaction as reward objectives, the MAPPO algorithm is used to output optimization strategies and dynamically correct the task allocation scheme.

[0023] In this embodiment, data at the current moment is sampled at fixed time intervals. The data includes: the version number and mapping pairing record of the elastic mapping lookup table in step 102, the pairing matching weight parameters of the current physical region and virtual region, the values ​​of each element in the optimal role allocation matrix in step 103, the role type and ability vector of each user currently assigned, and the time sequence log of each user's interaction operation recorded in step 104. The operation sequence of each user within a preset backtracking time window is extracted from the interaction operation time sequence log. The operation sequence includes the gesture category, operation object identifier, and operation result status at each time point. The combination of the elastic mapping parameters, role allocation vector, and operation sequence of all users at the current moment is used as the input feature. The input feature is normalized to map the value range of each dimension to a preset interval to obtain the normalized joint observation feature. Each user's local observation features are input into their corresponding Actor policy network. This network outputs the probability distribution of each user's available actions in the current state. Available actions include moving to an adjacent spatial anchor point, performing a certain type of gesture operation, releasing a character-specific skill, or switching the current interaction target. During the training phase, a global Critic evaluation network is invoked. This network takes the joint observation features and joint action features of all users as input and outputs the value assessment of the current global state. During environmental interaction, a preset batch size of experience samples is collected. Each experience sample contains each user's local observations, actions performed by each user, the global reward value from the environmental feedback, and each user's local observations at the next moment. The global reward value is calculated as follows: the task completion component is calculated based on the ratio of the number of sub-tasks completed by all users at the current moment to the total number of sub-tasks; the interaction fluency component is calculated based on the average response latency and error rate of all user interaction operations; and the task completion component and the interaction fluency component are weighted and summed to obtain the comprehensive reward value. Based on the collected experience samples, the generalized advantage estimation algorithm is used to calculate the advantage value for each action. This advantage value is the actual cumulative reward of the action minus the baseline value evaluated by the Critic network. The calculated advantage value is used to update the parameters of the Actor policy network: for each experience sample, the ratio of the probability of the action under the current policy to the probability of the action under the old policy is calculated. This ratio is multiplied by the corresponding advantage value as the objective function for policy optimization, and a pruning operation is used to limit the range of variation of this ratio. The mean squared error between the global reward value in the collected experience samples and the current value evaluation value output by the Critic network is used to update the parameters of the Critic evaluation network. This process is repeated until the Actor... The multi-agent cooperative strategy is obtained when the parameter changes of the policy network and the Critic evaluation network are lower than the preset convergence threshold. The trained multi-agent cooperative strategy is then deployed to each user terminal for inference. Each user terminal only runs its local Actor policy network and outputs the recommended action for the next moment in real time based on its own local observations. During the operation, the progress of each user's subtasks is continuously monitored. When a user's completion speed is detected to be lower than the average level, a greedy algorithm is used to search for the set of subtasks to be adjusted based on the role allocation matrix and execution status of all users. Subtasks that have timed out and have not been completed are reassigned to other users with the lowest current load, and the corresponding cooperative task subset is updated. After each collaborative task is completed in a phase, the actual performance data of each role in that phase is statistically analyzed. The actual performance data is compared with the preset performance benchmark. If the deviation exceeds the preset threshold, a role configuration adjustment signal is output. In the next role allocation, the initial matching weight of low-performance roles is reduced. The current version number, current matching weight parameters, role allocation matrix, and subtask allocation records of each user in the elastic mapping lookup table in step 102 are saved as state snapshots to the cache for use in the experience replay pool of the next round of MAPPO training.

[0024] Please see Figure 2 A schematic diagram of the structure of a multi-person collaborative interaction system in a virtual-real fusion scenario provided by an embodiment of the present invention. The system includes: The physical space perception module is used to acquire the three-dimensional geometric structure and semantic information of the physical space, and load the graph structure of the virtual scene and the current plot state vector. The elastic mapping module is used to dynamically and optimally pair the semantic partitions of the physical space with the semantic regions of the virtual scene according to the plot state vector, establish an elastic mapping relationship between the physical space and the virtual scene, and generate the corresponding spatial anchor point coordinate system based on the user's real-time pose. The role management module is used to dynamically assign differentiated roles to each user based on the current collaborative task's requirement vector and the real-time pose distribution of each user in the physical space. An interaction recognition module is used to perform multi-person collaborative operations based on the spatial anchor point coordinate system and natural gesture interaction under the constraints of the elastic mapping relationship. The collaborative optimization module is used to collect elastic mapping parameters, allocation status, and multi-user interaction operation sequences in real time, construct a multi-agent collaborative state space, use the completion rate of collaborative tasks and the smoothness of interaction as reward objectives, output optimization strategies using the MAPPO algorithm, and dynamically correct the task allocation scheme.

[0025] Figure 3 This is a schematic diagram of the structure of a multi-user collaborative interaction device in a virtual-real fusion scenario provided by an embodiment of the present invention. The multi-user collaborative interaction device 300 in this virtual-real fusion scenario can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the multi-user collaborative interaction device 300 in the virtual-real fusion scenario. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute a series of instruction operations in the storage media 330 on the multi-user collaborative interaction device 300 in the virtual-real fusion scenario to implement the method provided in the above embodiment.

[0026] The multi-user collaborative interaction device 300 in a virtual-real fusion scenario may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The multi-person collaborative interaction device structure shown in the virtual-real fusion scenario does not constitute a limitation on the computer device provided by the present invention. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0027] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the various steps of the multi-person collaborative interaction method in the virtual-real fusion scenario provided in the above embodiments.

[0028] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described equipment or apparatus / unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0029] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0030] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for multi-person collaborative interaction in a virtual-real fusion scenario, characterized in that, The method includes the following steps: Obtain the three-dimensional geometric structure and semantic information of the physical space, and load the graph structure of the virtual scene and the current plot state vector; Based on the plot state vector, the semantic partitions of the physical space and the semantic regions of the virtual scene are dynamically and optimally paired to establish an elastic mapping relationship between the physical space and the virtual scene, and a corresponding spatial anchor point coordinate system is generated based on the user's real-time pose. Based on the current collaborative task requirements and the real-time pose distribution of each user in the physical space, dynamically assign differentiated roles to each user. Under the constraints of the elastic mapping relationship, multi-person collaborative operation is carried out based on the spatial anchor point coordinate system and natural gesture interaction; Real-time acquisition of elastic mapping parameters, allocation status, and multi-user interaction operation sequences is used to construct a multi-agent collaborative state space. With the completion rate of collaborative tasks and the smoothness of interaction as reward objectives, the MAPPO algorithm is used to output optimization strategies and dynamically correct task allocation schemes.

2. The multi-person collaborative interaction method in a virtual-real fusion scenario as described in claim 1, characterized in that, The process of acquiring the three-dimensional geometric structure and semantic information of the physical space, and loading the graph structure of the virtual scene and the current plot state vector, includes: Acquire color and depth images of the physical space, determine camera pose, and optimize global trajectory using bundle adjustment; Based on the optimized camera pose, the depth image is back-projected into a 3D point cloud, and a voxel fusion algorithm with a truncated signed distance function is used to generate a dense voxel mesh. The traveling cube algorithm is applied to extract the triangular mesh surface model from the dense voxel mesh, and the region growing segmentation algorithm is used to divide the surface into physical space regions with semantic labels. Load the virtual scene description file, extract all spatial nodes and their connectivity relationships in the virtual scene, and construct the directed acyclic graph structure of the virtual scene; Obtain the current plot progress status, read the list of unlocked scene nodes, the list of untriggered plot events, and the active narrative branch identifiers, and combine them into the current plot status vector.

3. The multi-person collaborative interaction method in a virtual-real fusion scenario as described in claim 1, characterized in that, Based on the aforementioned plot state vector, the semantic partitions of the physical space and the semantic regions of the virtual scene are dynamically and optimally paired to establish an elastic mapping relationship between the physical space and the virtual scene. A corresponding spatial anchor point coordinate system is generated based on the user's real-time pose, including: Extract the geometric features of each semantic region in the physical space and the functional semantic features of each node in the virtual scene, and combine them with the narrative priority parameters of each virtual region in the current plot state vector to construct the cost matrix between the physical region and the virtual region. With the goal of minimizing the total matching cost, the Hungarian algorithm is used to solve for the optimal pairing scheme between physical and virtual regions; Establish an elastic mapping lookup table based on the optimal pairing scheme, record the correspondence between each physical region identifier and its mapped virtual region identifier, and use it as the elastic mapping relationship at the current moment. Obtain the real-time three-dimensional position coordinates and head pose of each user in the physical space, and convert the physical coordinates of each user into the corresponding virtual space coordinates according to the elastic mapping lookup table; Each user's independent local spatial anchor point coordinate system is established with the user's physical location as the origin, and all local coordinate systems are uniformly transformed to the same world coordinate system to form a shared spatial anchor point system.

4. The multi-person collaborative interaction method in a virtual-real fusion scenario as described in claim 1, characterized in that, The process of dynamically assigning differentiated roles to each user based on the current collaborative task's demand vector and the real-time pose distribution of each user in physical space includes: Extract the current collaborative task identifier from the current story state vector, and obtain the minimum requirement level of each ability dimension required to complete the collaborative task through the corresponding task description file to form a requirement vector. Three character templates are preset: Explorer, Guardian and Puzzle Solver. Each template is pre-configured with corresponding baseline values ​​and operational efficiency coefficients for each ability dimension. Calculate the matching score between each user and each role template, and select the role template with the highest matching score as the initial candidate role for that user; Determine whether the combination of all initial candidate roles for users meets all capability dimensions in the demand vector. If there is a capability gap, perform a role upgrade operation on the user with the second highest matching degree. With the goal of maximizing the sum of the matching degree of all users' ability vectors and demand vectors, and with the uniqueness constraint of role allocation as a condition, the optimal role allocation matrix is ​​determined, and a corresponding subset of collaborative tasks is generated for each user.

5. A multi-person collaborative interaction method in a virtual-real fusion scenario as described in claim 1, characterized in that, The method of multi-person collaborative operation based on the spatial anchor point coordinate system and natural gesture interaction under the constraint of elastic mapping relationship includes: Hand region detection and 3D key point regression are performed on hand images collected by each user's device. The key point coordinates are transformed to the spatial anchor point coordinate system, and the key point sequence of multiple consecutive frames is temporally encoded to identify the current dynamic gesture category. After detecting a grasping gesture, determine whether the virtual object near the hand is a graspable object. If so, establish a temporary binding relationship between the hand and the object and control the object to follow the hand's movement. Upon detecting a release gesture, the binding relationship is removed, the virtual object is fixed to the current spatial anchor point, the object status is updated to be pickable, and a status change message is broadcast to all users. When it is detected that the hands of at least two users are simultaneously within the preset radius of the same spatial anchor point, the role type of each relevant user is read and the preset collaborative operation rule library is queried to determine the type of collaborative operation mode triggered. If multiple users simultaneously initiate interactions with the same object and the conditions for collaborative operation mode are not met, it is determined to be an interaction conflict event. Based on the role priority table, the user with the highest priority is selected as the initiator, and the requests of other users are rejected.

6. A multi-person collaborative interaction method in a virtual-real fusion scenario as described in claim 5, characterized in that, The process of performing hand region detection and 3D keypoint regression on hand images collected from each user's device, transforming the keypoint coordinates to a spatial anchor point coordinate system, and performing temporal encoding on a series of consecutive keypoint sequences to identify the current dynamic gesture category includes: The acquired hand images are filtered by confidence and suppressed by non-maximum to output the hand bounding box. The image block is cropped with the hand bounding box as the region of interest and the two-dimensional pixel coordinates and depth values ​​of each hand key point are determined. The three-dimensional coordinates of the key points in the camera coordinate system are obtained by inverse projection of the camera intrinsic parameters. The three-dimensional coordinates of key points are transformed to the spatial anchor point coordinate system through rigid body transformation. The coordinates of multiple consecutive frames are integrated to obtain a fixed-length buffer queue. The displacement vector and velocity vector of each key point between adjacent frames are calculated and concatenated into a temporal feature vector. The temporal feature vector is input into a temporal convolutional network for temporal dimension feature extraction and global pooling. The network is then passed through a fully connected classification network to output the probability distribution of each predefined gesture category, and the category with the highest probability is selected as the recognition result.

7. A multi-person collaborative interaction method in a virtual-real fusion scenario as described in claim 1, characterized in that, The system collects real-time elastic mapping parameters, allocation states, and multi-user interaction sequences to construct a multi-agent collaborative state space. Using collaborative task completion and interaction fluency as reward objectives, it employs the MAPPO algorithm to output optimization strategies and dynamically corrects the task allocation scheme, including: Real-time acquisition of elastic mapping parameters, allocation status, and multi-user interaction operation sequences, which are then normalized and combined into joint observation features of multiple agents; The local observation features of each user are input into the corresponding Actor policy network to output the action probability distribution, and the global state value is output through the global Critic evaluation network using the joint observations and joint actions of all users as input. The global reward value is calculated by weighting the ratio of the number of sub-tasks completed by each user to the total number of sub-tasks, as well as the average response latency and error rate of interactive operations. The generalized advantage estimation algorithm is used to calculate the advantage value of each action and update the Actor-Critic network parameters. The multi-agent collaborative strategy after training convergence is deployed to each user terminal, and the recommended action for the next moment is output in real time. Continuously monitor the completion progress of each user's subtasks, and use a greedy algorithm to search for and reassign subtasks that have not been completed within the time limit to other users with the lowest current load. Dynamically update the collaborative task subsets of each user.

8. A multi-person collaborative interaction system in a virtual-real fusion scenario, characterized in that, The system includes: The physical space perception module is used to acquire the three-dimensional geometric structure and semantic information of the physical space, and load the graph structure of the virtual scene and the current plot state vector. The elastic mapping module is used to dynamically and optimally pair the semantic partitions of the physical space with the semantic regions of the virtual scene according to the plot state vector, establish an elastic mapping relationship between the physical space and the virtual scene, and generate the corresponding spatial anchor point coordinate system based on the user's real-time pose. The role management module is used to dynamically assign differentiated roles to each user based on the current collaborative task's requirement vector and the real-time pose distribution of each user in the physical space. An interaction recognition module is used to perform multi-person collaborative operations based on the spatial anchor point coordinate system and natural gesture interaction under the constraints of the elastic mapping relationship. The collaborative optimization module is used to collect elastic mapping parameters, allocation status, and multi-user interaction operation sequences in real time, construct a multi-agent collaborative state space, use the completion rate of collaborative tasks and the smoothness of interaction as reward objectives, output optimization strategies using the MAPPO algorithm, and dynamically correct the task allocation scheme.

9. A multi-person collaborative interaction device for a virtual-real fusion scenario, characterized in that, The multi-person collaborative interaction device in the virtual-real fusion scenario includes a memory and at least one processor. The memory stores instructions. The at least one processor calls the instructions in the memory to cause the multi-person collaborative interaction device in the virtual-real fusion scenario to execute each step of the multi-person collaborative interaction method in the virtual-real fusion scenario as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the various steps of the multi-person collaborative interaction method in the virtual-real fusion scenario as described in any one of claims 1-7.

Images