Multi-person action collaborative recognition method and system based on virtual motion capture technology

Abstract

The present invention relates to the field of moving point recognition technology, and discloses a method and system for collaborative recognition of multi-person actions based on virtual moving point motion capture technology, including: deploying multiple cameras to capture the operator's action information from different perspectives and perform three-dimensional reconstruction; in three-dimensional space, perform moving point capture on the reconstructed model; when an operation occurs, use game theory to perform collaborative action recognition on the operation without occlusion; for the operation with occlusion, judge the feasible domain of the moving point, predict according to the judgment result, and select the optimal output as the recognition result based on the prediction result. Accurate recognition of collaborative operation behavior of multiple people is achieved. The accuracy of action recognition is optimized; the efficiency and robustness of collaborative recognition in complex action scenes of multiple people are significantly improved, and at the same time, it has strong real-time and adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of moving point recognition, and in particular to a method and system for collaborative recognition of multi-person actions based on virtual moving point motion capture technology. Background Art

[0002] With the development of intelligent interactive technology, motion capture technology has been widely used in virtual reality, film and television animation, sports training, medical rehabilitation and other fields. Traditional motion capture systems mainly rely on marker tracking, inertial sensors and depth camera technology to achieve accurate capture of human movements. However, these technologies have some limitations in practical applications. For example, the marker system requires wearing professional equipment and is easily affected by occlusion and ambient light; inertial sensors are limited by installation position and accuracy, making it difficult to achieve efficient recognition in multi-person interactive scenes; and depth cameras face the challenges of reduced accuracy and data loss when dealing with complex multi-person scenes or long-distance capture. These problems limit the further application of motion capture technology in complex scenes such as multi-person collaborative operations.

[0003] In recent years, motion capture technology based on 3D reconstruction and virtual moving points has gradually become a research hotspot. By capturing the operator's motion information through multi-view cameras and combining it with 3D modeling technology, the details of the motion can be restored more comprehensively, and virtual moving points can be constructed to represent the motion characteristics of the human body. At the same time, with the development of artificial intelligence and deep learning algorithms, the extraction of motion representation vectors with the help of neural network models and accurate recognition through analysis of motion trajectories, skeletal postures and other features have become the forefront of motion capture technology. However, in multi-person collaborative scenarios, due to problems such as occlusion, interference, and increased motion complexity, the robustness of existing technologies in motion completion and collaborative recognition is still insufficient and needs to be further optimized. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing motion capture technology has problems such as inaccurate recognition results, inability to exclude recognition of abnormal parts, and inability to predict occlusion.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a multi-person action collaborative recognition method based on virtual moving point motion capture technology, comprising:

[0007] By deploying multiple cameras, the operator's action information is captured from different perspectives and three-dimensional reconstruction is performed;

[0008] In three-dimensional space, the reconstructed model is captured in motion;

[0009] When an operation occurs, game theory is used to identify collaborative actions for operations without occlusion. For operations with occlusion, the feasible domain of the moving point is judged, predictions are made based on the judgment results, and the optimal output is selected as the recognition result based on the prediction results.

[0010] As a preferred solution of the multi-person action collaborative recognition method based on virtual moving point motion capture technology described in the present invention, wherein: the three-dimensional reconstruction includes constructing a three-dimensional model of the part where the operation occurs based on the information collected by the camera;

[0011] and constructing a virtual moving point according to the three-dimensional model;

[0012] The virtual moving points include identifying the points at the joints and limb ends of the operation-occurring part in the three-dimensional model; marking the virtual moving points with a connection relationship as the same control branch according to the three-dimensional model, and determining the fixed distance between every two interconnected virtual moving points in the control branch, as well as the rotatable angle of the connection part.

[0013] As a preferred solution of the multi-person action collaborative recognition method based on virtual moving point motion capture technology described in the present invention, wherein: the moving point capture includes tracking the motion trajectory of the virtual moving point and updating the position information in the three-dimensional space in real time;

[0014] During the update process, the number of virtual moving points in each control branch before and after the update is analyzed. If the number decreases, it is preliminarily judged that there is occlusion.

[0015] As a preferred solution of the multi-person action collaborative recognition method based on virtual moving point motion capture technology described in the present invention, wherein: the moving point capture also includes analyzing the number of virtual moving points in each occluded part. When the occluded part is only one virtual moving point, the occluded part is completed by the positions of other virtual moving points connected to each other and the angle of the connected part, and the current operation behavior is regarded as an operation behavior without occlusion; when the occluded part is multiple virtual moving points, the current operation behavior is regarded as an operation behavior with occlusion.

[0016] As a preferred solution of the multi-person action collaborative recognition method based on virtual moving point motion capture technology described in the present invention, wherein: the game theory includes, according to the selected operation purpose, matching the node flow of the operation process;

[0017] Assuming that the number of initially matched node processes is n, each control branch is regarded as a player, and according to the motion characteristics of the virtual moving point and the action contribution of each player to the success of the collaborative operation, the roles of different players in the collaborative operation are identified;

[0018] The action contribution includes, in each node process path, analyzing the current node and the next node respectively, and the degree of consistency with the current collaborative operation behavior;

[0019] The local motion features are extracted through the convolutional neural network, and the time series characteristics of the virtual moving points under the collaborative operation are captured by the recurrent neural network, and the output is the feature vector ;

[0020] The fully connected layer reduces the dimension of the feature vector to , generating a compact action representation vector ;

[0021] in, represents the d-dimensional real number space, represents k-dimensional real number space;

[0022] Network output: Each node The action representation vector is ; The feature vector of the current collaborative operation behavior is ;

[0023] The degree of coincidence includes, for each action representation vector, determining its characteristic subspace through principal component analysis, and the basis matrix of the characteristic subspace is ;

[0024] Will Projected onto the feature subspace of the node process superior:

[0025] ;

[0026] in, for exist Projection in subspace;

[0027] Calculate projection error To quantify whether the feature vector belongs to the node feature subspace:

[0028] ;

[0029] Analyzing the projection vector The direction and target vector The degree of fit :

[0030] ;

[0031] in, express and Directional consistency;

[0032] Combining projection error and direction consistency, define the action consistency score :

[0033] ;

[0034] in, is the maximum allowable value of the projection error; , is the weight parameter;

[0035] If the score is less than the matching threshold, all players are considered collaborators and identified normally;

[0036] If the score is not less than the matching threshold, it is determined that there is an interferer between the collaborators, and selective identification is performed to obtain only the identification results of the collaborators;

[0037] After the recognition is completed, the node where the recognition result is located is determined, and its position between the current node and the next node. If the recognition result is the next node, the node process is updated, and the original next node is used as the current node to continue the recognition;

[0038] In the n node processes, the paths that do not conform to the selected node sequence are eliminated through the selection of nodes by operation behavior, and nm node processes that conform to the current operation process are obtained; m represents the number of eliminated paths, m<n;

[0039] The node process includes defining each link in the operation process and using it as a node in the process, connecting the nodes in series to obtain a node process after the connection; if the order of the nodes in the process is replaceable, then connecting different node orders in series according to the replaceable positions to generate all the node processes;

[0040] The node flow corresponding to each operation purpose is constructed, and after the operation purpose is selected, the node flow after the series connection is automatically matched.

[0041] As a preferred solution of the multi-person action collaborative recognition method based on virtual moving point motion capture technology described in the present invention, wherein: the selective recognition includes randomly removing a control branch from all control branches, re-recognizing the degree of fit of the current collaborative operation behavior, and if the score is less than the fit threshold, the removed control branch is judged to be an interferer; if the score is not less than the fit threshold, one more control branch is removed until the score is less than the fit threshold;

[0042] In the visual interface, the node corresponding to the interferer is marked, and only participates in the identification process, but does not issue any operation instructions to the control end.

[0043] As a preferred solution of the multi-person action collaborative recognition method based on virtual moving point motion capture technology described in the present invention, wherein: the determination of the feasible domain of the moving point includes, according to the three-dimensional model of each control branch, determining the number of virtual moving points in the blocked part, and obtaining virtual nodes directly connected to the blocked virtual moving points as reference nodes;

[0044] Assume reference node A, determine the position A1 of an obscured virtual moving point according to the angle between reference node A and the connection part and the fixed distance from the adjacent node; determine the feasible domain a2 of the next obscured virtual moving point position according to the rotatable angle of the connection part of A1 and the fixed distance from the adjacent node; randomly select a point in a2 as the position A2 of the obscured virtual moving point, and determine the next virtual moving point, until the positions of all obscured virtual moving points are determined, and a position combination of obscured virtual moving points is obtained;

[0045] The operation of randomly selecting virtual moving point positions in the feasible domain is continuously repeated until no new virtual moving point position combinations are generated;

[0046] The optimal output includes dynamically analyzing all position combinations, combining any one of the position combinations with other unobstructed virtual moving points as a collaborative state, analyzing the action contribution, finding the combination with the highest degree of consistency and outputting it as the optimal result, and color-coding the virtual moving point recognition results of the obstructed part in the visualization interface.

[0047] A multi-person action collaborative recognition system based on virtual moving point motion capture technology using any method as described in the present invention, wherein:

[0048] The acquisition unit deploys multiple cameras to capture the operator's motion information from different perspectives and perform three-dimensional reconstruction;

[0049] A capture unit captures the moving points of the reconstructed model in three-dimensional space;

[0050] The recognition unit, when an operation occurs, uses game theory to perform collaborative action recognition for operation behaviors without occlusion; for operation behaviors with occlusion, it judges the feasible domain of the moving point, makes predictions based on the judgment results, and selects the optimal output as the recognition result based on the prediction results.

[0051] A computer device comprises: a memory and a processor; the memory stores a computer program, wherein: the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0052] A computer-readable storage medium stores a computer program, wherein: when the computer program is executed by a processor, the steps of any one of the methods of the present invention are implemented.

[0053] Beneficial effects of the present invention: The method for collaborative recognition of multi-person actions based on virtual moving point motion capture technology provided by the present invention realizes accurate recognition of multi-person collaborative operation behaviors by constructing a virtual moving point model, combining multi-view 3D reconstruction and game theory analysis. In view of the occlusion problem, the present invention optimizes the accuracy of action recognition through moving point feasible domain judgment and dynamic completion; by calculating the degree of fit between the action representation vector and the feature subspace, as well as interferer detection and selective recognition, the efficiency and robustness of collaborative recognition in multi-person complex action scenes are significantly improved, while having strong real-time and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0055] Figure 1 This is an overall flow chart of a multi-person action collaborative recognition method based on virtual moving point motion capture technology provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0057] Example 1, reference Figure 1 , is an embodiment of the present invention, and provides a multi-person action collaborative recognition method based on virtual moving point motion capture technology, comprising:

[0058] S1: Deploy multiple cameras to capture the operator’s motion information from different perspectives and perform 3D reconstruction.

[0059] Furthermore, the three-dimensional reconstruction includes, based on the information collected by the camera, constructing a three-dimensional model of the part where the operation occurs. And based on the three-dimensional model, constructing a virtual moving point. The virtual moving point includes identifying the points at the joints and limb ends of the part where the operation occurs in the three-dimensional model; based on the three-dimensional model, marking the virtual moving points with a connection relationship as the same control branch, and determining the fixed distance between each two interconnected virtual moving points in the control branch, as well as the rotatable angle of the connection part.

[0060] By generating virtual moving points where the operation takes place through 3D reconstruction, the operator's key joints and limb end positions are accurately modeled, providing a clear representation of the motion structure. Compared with 2D plane data, 3D models can more intuitively and accurately restore action details, which helps with recognition and analysis in complex action scenes.

[0061] It is also worth mentioning that by marking virtual moving points with connection relationships as the same control branch, the topological structure of the human body or object is constructed, and the movement association between the various parts is clarified. It is convenient to describe the collaborative relationship between different parts (such as substructures such as arms and legs). It supports complex multi-branch collaborative motion analysis. By defining parameters such as fixed distances and rotatable angles, it provides necessary geometric and physical constraints for the movement of virtual moving points. It prevents unreasonable movements from occurring during the capture and prediction of virtual moving points (such as bone dislocation or joints that do not conform to the physiological structure). It provides a scientific basis for subsequent action completion and recognition, ensuring that the prediction results conform to the movement laws of the human body or operating equipment.

[0062] S2: In three-dimensional space, the reconstructed model is captured in motion.

[0063] The moving point capture includes tracking the motion trajectory of the virtual moving point and updating the position information in the three-dimensional space in real time. During the updating process, the number of virtual moving points in each control branch before and after the update is analyzed. If the number decreases, it is preliminarily determined that there is occlusion.

[0064] Furthermore, the number of virtual moving points in each occluded part is analyzed. When the occluded part consists of only one virtual moving point, the occluded part is completed through the positions of other virtual moving points connected to each other and the angle of the connected part, and the current operation behavior is regarded as an operation behavior without occlusion; when the occluded part consists of multiple virtual moving points, the current operation behavior is regarded as an operation behavior with occlusion.

[0065] What needs to be said is that by tracking the motion trajectory of the virtual moving point and updating its position information in three-dimensional space in real time, the captured action information is ensured to be continuous and dynamic. Real-time updating of the moving point position helps to obtain accurate motion trajectories and supports subsequent recognition and analysis. This solves the problem that static models cannot reflect dynamic actions and lays the foundation for dynamic behavior analysis.

[0066] By analyzing the changes in the number of virtual moving points in the control branch, it is automatically determined whether there is occlusion. When the occlusion only involves a single virtual moving point, it is completed based on kinematic constraints through the relationship with the adjacent connected moving points and angles. The position of the occluded moving point is quickly inferred using the position information and connection angles of the adjacent points. The single-point occlusion action is completed to ensure the integrity of the motion capture and avoid misidentification caused by small occlusions.

[0067] When occlusion involves multiple virtual moving points, it is marked as complex occlusion and treated as a special scenario. Simple occlusion and complex occlusion are processed separately to optimize computational efficiency. In complex occlusion scenarios, processing space is reserved for subsequent further prediction and recognition.

[0068] S3: When an operation occurs, game theory is used to identify collaborative actions for operations without occlusion. For operations with occlusion, the feasible domain of the moving point is judged, predictions are made based on the judgment results, and the optimal output is selected as the recognition result based on the prediction results.

[0069] Furthermore, the game theory includes matching the node flow of the operation process according to the selected operation purpose. Assuming that the initially matched node flow is n, each control branch is regarded as a player, and according to the motion characteristics of the virtual moving point, the roles of different players in the collaborative operation are identified through the action contribution of each player to the success of the collaborative operation.

[0070] It should be noted that the action contribution includes analyzing the current node and the next node in each node process path, and the degree of consistency with the current collaborative operation behavior. The local motion features are extracted through the convolutional neural network, and the time series characteristics of the virtual moving points under the collaborative operation are captured by the recurrent neural network, and the output is the feature vector ; The fully connected layer reduces the dimension of the feature vector to , generating a compact action representation vector ; Network output: Each node The action representation vector is ; The feature vector of the current collaborative operation behavior is .in, represents the d-dimensional real number space, represents k-dimensional real number space.

[0071] According to the purpose of the operation, match the node process associated with it, and analyze whether the current collaborative operation behavior meets the requirements of the node process. Filter out the path that best matches the current operation behavior from multiple possible paths. Ensure the flexible adaptability of the node process through dynamic path matching. Provide dynamic evaluation of multiple paths for complex multi-person operation scenarios to improve the adaptability of recognition.

[0072] With the help of deep learning technology, high-quality motion features are extracted from the motion trajectory of the virtual moving point to form a feature vector as the core input for motion recognition. Convolutional neural network (CNN) is used to extract local motion features (such as position changes and speed trends); recurrent neural network (RNN) is used to capture time series characteristics (such as the continuity and rhythm of the action).

[0073] It is important to know that the operation process is divided into multiple nodes, each of which represents an independent link in the operation (such as picking, moving, and placing). There may be multiple combinations of the connection order between different nodes. Some operation nodes can be completed in different orders without affecting the final operation purpose. For example: in an assembly task, you can assemble part A first and then part B, or you can assemble part B first and then part A. Different roles can complete different subtasks at the same time, forming parallel paths. There may be more than one way to divide the work between roles. For example: in a two-person lifting task, A dominates the direction of movement and B dominates the speed; role responsibilities can also be exchanged.

[0074] The degree of coincidence includes, for each action representation vector, determining its characteristic subspace through principal component analysis, and the basis matrix of the characteristic subspace is .Will Projected onto the feature subspace of the node process superior:

[0075] ;

[0076] in, for exist Projection in subspace. Calculate projection error To quantify whether the feature vector belongs to the node feature subspace:

[0077] ;

[0078] Analyze the projection vector The direction and target vector The degree of fit :

[0079] ;

[0080] in, express and The directional consistency of When , it means that the feature vector direction is consistent and the operation behavior fully meets the node requirements. Combining the projection error and direction consistency, the action consistency score is defined :

[0081] ;

[0082] in, is the maximum allowable value of the projection error; , is the weight parameter.

[0083] If the score is less than the consistency threshold, all players are considered collaborators and are identified normally. If the score is not less than the consistency threshold, it is determined that there are interferers among the collaborators, and selective identification is performed to obtain only the identification results of the collaborators.

[0084] After the recognition is completed, the node where the recognition result is located is determined, and its position between the current node and the next node. If the recognition result is the next node, the node process is updated, and the original next node is used as the current node to continue the recognition. In the n node processes, the nodes are selected through the operation behavior, and the paths that do not conform to the selected node sequence are eliminated, and nm node processes that conform to the current operation process are obtained; m represents the number of eliminated paths, m<n.

[0085] What needs to be said is that according to the action recognition results, the status of the current node and the next node are dynamically judged to ensure that the node process is consistent with the actual operation behavior. Flexible adaptation of the operation process is achieved to avoid recognition errors due to the fixed nature of the node process. In n node processes, the nodes are selected through the operation behavior, and the paths that do not conform to the selected node order are eliminated to reduce unnecessary calculations. After eliminating invalid paths, the remaining node processes are more consistent with the current operation behavior, reducing the calculation complexity and the possibility of mismatching. In the node process, the node order is judged according to the established rules to ensure that the operation behavior conforms to the process logic.

[0086] Furthermore, the node process includes defining each link in the operation process and using it as a node in the process, connecting the nodes in series to obtain a node process after the series connection; if the order of the nodes in the process is replaceable, then according to the replaceable positions, different node orders are connected in series to generate all node processes. The node process corresponding to each operation purpose is constructed, and after selecting the operation purpose, the node process after the series connection can be automatically matched.

[0087] The selective identification includes randomly removing a control branch from all control branches, and re-identifying the degree of fit of the current collaborative operation behavior. If the score is less than the fit threshold, the removed control branch is judged to be an interferer; if the score is not less than the fit threshold, one more control branch is removed until the score is less than the fit threshold.

[0088] In the visual interface, the node corresponding to the interferer is marked, and only participates in the identification process, but does not issue any operation instructions to the control end.

[0089] By randomly removing control branches and re-evaluating the action consistency, interferers that do not meet the collaborative operation requirements are identified, thereby optimizing the recognition results. The presence of interferers will reduce the accuracy of overall recognition. After removing interferers, only control branches that meet the operation requirements are retained to ensure the reliability of the action recognition results. By using the changes in the action consistency score Score, it is dynamically determined which control branches interfere with the current collaborative operation behavior. The dynamic recognition mechanism avoids treating all branches equally, allowing the system to flexibly adjust task allocation according to actual operation conditions. After removing interferers, only valid control branches are analyzed to reduce meaningless calculations and resource consumption. Reduce the system burden caused by interferers and improve recognition efficiency, especially in multi-person complex operation scenarios.

[0090] In the visual interface, the nodes corresponding to the interferers are marked, so that users can intuitively understand which control branches are judged as interferers. This enhances the transparency and explainability of the system and facilitates users to monitor and adjust the operation process.

[0091] The determination of the feasible domain of the moving point includes determining the number of virtual moving points in the blocked part according to the three-dimensional model of each control branch, and obtaining virtual nodes directly connected to the blocked virtual moving points as reference nodes.

[0092] Assume reference node A, determine the position A1 of an obscured virtual moving point according to the angle between reference node A and the connection part and the fixed distance from the adjacent node; determine the feasible domain a2 of the next obscured virtual moving point position according to the rotatable angle of the connection part of A1 and the fixed distance from the adjacent node; randomly select a point in a2 as the position A2 of the obscured virtual moving point, and determine the next virtual moving point until the positions of all obscured virtual moving points are determined, and a position combination of obscured virtual moving points is obtained. Repeat the operation of randomly selecting virtual moving point positions in the feasible domain until no new virtual moving point position combination is generated.

[0093] Dynamically analyze all position combinations, combine any one of the position combinations with other unobstructed virtual moving points as a collaborative state, analyze the action contribution, find the combination with the highest degree of consistency and output it as the optimal result, and color-code the virtual moving point recognition results of the obstructed part in the visualization interface.

[0094] What needs to be said is that in occlusion scenarios, the positions of occluded moving points are recursively generated by referring to the geometric constraints of nodes and connected parts to ensure that occlusion completion conforms to the human body structure or the movement laws of operating equipment. Use geometric and kinematic constraints to generate reasonable combinations of moving point positions to avoid non-physical or unreasonable prediction results. Through random sampling and multiple iterations, all possible combinations of position information of occluded moving points are generated to ensure that the entire solution space is covered. In complex occlusion scenarios, the best solution can be guaranteed to be in the generated solution set in a probabilistic form.

[0095] By combining the position combination with the unobstructed virtual moving point into a coordinated state, evaluating its action contribution and calculating the degree of fit, the combination with the highest matching degree is selected as the final output. The optimal result is selected in combination with the actual operating environment and action logic to avoid recognition errors caused by random selection. The optimal result not only meets the rationality of the occluded moving point, but also enhances the logic and coherence of the overall action coordination.

[0096] It is also important to know that path selection is essentially finding a path from multiple action sequences that best matches the operation target. Different combinations of occluded moving point positions will generate different action feature vectors, which determine the degree of fit between the action and the node process. The prediction results of the occluded moving points are actually part of the dynamic path matching, so the two are completed synchronously. Dynamic analysis calculates the degree of fit between the action contribution of each combination and the node process, and at the same time selects the path that best matches the current operation logic. The optimal combination of occlusion prediction is itself the result of path selection, so no additional steps are required.

[0097] On the other hand, this embodiment also provides a multi-person motion collaborative recognition system based on virtual motion capture technology, which includes:

[0098] The acquisition unit deploys multiple cameras to capture the operator's motion information from different perspectives and perform three-dimensional reconstruction.

[0099] The capture unit captures the moving points of the reconstructed model in three-dimensional space.

[0100] The recognition unit, when an operation occurs, uses game theory to perform collaborative action recognition for operation behaviors without occlusion; for operation behaviors with occlusion, it judges the feasible domain of the moving point, makes predictions based on the judgment results, and selects the optimal output as the recognition result based on the prediction results.

[0101] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0102] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0103] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0104] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0105] Example 2 is an embodiment of the present invention, which provides a multi-person action collaborative recognition method based on virtual moving point motion capture technology. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0106] Five subjects were selected to perform predefined operation tasks, including object transfer, assembly collaboration, and team synchronization. The experimental scene was set up in a simulated smart factory environment, and a multi-view shooting system was built using multiple cameras to capture the operator's action information. The specific steps of the experiment are as follows:

[0107] Ten high-resolution cameras were deployed in the experimental site to synchronously capture the operator's movements from different perspectives. The data collected by the cameras was used to generate a 3D model of each subject using 3D reconstruction technology, focusing on capturing virtual moving points of joints (such as shoulders, elbows, and knees) and extremities (such as fingers and toes).

[0108] According to the 3D model, we constructed virtual moving points for each subject and marked the control branches (such as upper limbs and lower limbs). By tracking the movement trajectory of the virtual moving points in real time, recording their positions, angle changes, and movement speeds, and analyzing the changes in the number of virtual moving points in the control branches before and after the update, we preliminarily determined whether there was occlusion.

[0109] If occlusion is detected, the position of the occluded moving point is completed through the calculation of the feasible region of the moving point. Based on the reference node, combined with the fixed distance and connection angle of the adjacent nodes, the possible position combinations of the occluded moving point are recursively generated, and the combination with the highest degree of fit is selected as the completion result through dynamic analysis.

[0110] Use deep learning models (convolutional neural networks and recurrent neural networks) to extract local motion features and time series characteristics and generate compact motion representation vectors. Combined with game theory, analyze the action contribution of each subject in the collaborative task, dynamically adjust the role division, identify disruptors and optimize collaboration efficiency.

[0111] The subject's actions are identified in real time to determine their positions in the current node and the next node. Paths that do not conform to the node sequence are eliminated, and the node process is dynamically updated to ensure the optimality of the task completion path.

[0112] During the experiment, the virtual moving point trajectory, occlusion processing times, action recognition accuracy, node path selection times, and collaborative efficiency of each subject were recorded as the basis for comparative analysis. The recorded data are shown in Table 1.

[0113] Table 1 Data record table

[0114]

[0115] It can be seen from the table that the multi-person motion collaborative recognition method based on virtual moving point motion capture technology showed significant innovation and advantages in the experiment.

[0116] The average recognition accuracy reached 94.2%, with a maximum of 98.7%, and it could maintain a high accuracy rate even when the action complexity was high (such as subjects 1 and 4). This proves that the virtual moving point analysis method combining multi-view 3D reconstruction with deep learning can effectively improve the accuracy of action recognition.

[0117] In occluded scenes (such as subject 4), the completion success rate of occluded virtual moving points reached more than 95.8% through the feasible domain completion processing of moving points, proving the reliability of the occlusion processing algorithm in complex scenes. Dynamic analysis and multi-solution combination selection further optimized the completion results, making the completed action behavior more consistent.

[0118] The table shows that on average, 1.4 distractors were identified in each task. Especially in complex collaborative scenarios (subject 5), distractor removal significantly optimized collaborative efficiency, with an overall improvement rate of 16.3%. This verifies the effectiveness of the game theory model in dynamic role division and distractor removal.

[0119] In the experiment, invalid paths were eliminated 2.4 times on average, ensuring the rationality of node process selection. Especially in scenes with complex actions (subject 1) or many occlusions (subject 4), the path optimization strategy reduced redundant calculations and improved task completion efficiency.

[0120] Compared with the traditional motion capture system based on markers, this invention overcomes the problems of occlusion and complexity of dynamic division of labor, and has significant improvements in real-time and robustness. The existing technology is prone to recognition errors due to occlusion and role conflicts in complex multi-person scenes, while this invention significantly optimizes the accuracy of collaborative recognition and completion by combining deep learning with game theory, and has strong academic value and practicality.

[0121] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-person action collaborative recognition method based on virtual motion capture technology, characterized in that: include: By deploying multiple cameras, the operator's action information is captured from different perspectives and three-dimensional reconstruction is performed; In three-dimensional space, the reconstructed model is captured in motion; When an operation occurs, game theory is used to identify collaborative actions for operations without occlusion. For operations with occlusion, the feasible domain of the moving point is determined, predictions are made based on the judgment results, and the optimal output is selected as the recognition result based on the prediction results. The moving point capture includes tracking the motion trajectory of the virtual moving point and updating the position information in the three-dimensional space in real time, analyzing the number of virtual moving points in each blocked part, and when the blocked part is only one virtual moving point, completing the blocked part through the positions of other virtual moving points connected to each other and the angle of the connected part, and treating the current operation behavior as an operation behavior without blocking; When the occluded part is multiple virtual moving points, the current operation behavior is regarded as an operation behavior with occlusion; During the update process, the number of virtual moving points in each control branch before and after the update is analyzed. If the number decreases, it is preliminarily judged that there is occlusion.

2. The method for collaborative recognition of multi-person actions based on virtual moving point motion capture technology according to claim 1, characterized in that: The three-dimensional reconstruction includes constructing a three-dimensional model of the part where the operation occurs based on the information collected by the camera; and constructing a virtual moving point according to the three-dimensional model; The virtual moving point includes identifying the points at the joints and limb ends of the operation-occurring part in the three-dimensional model; According to the three-dimensional model, virtual moving points that are in a connection relationship are marked as the same control branch, and a fixed distance between every two mutually connected virtual moving points in the control branch and a rotatable angle of the connection part are determined.

3. The method for collaborative recognition of multi-person actions based on virtual moving point motion capture technology according to claim 2, characterized in that: The game theory includes matching the node flow of the operation process according to the selected operation purpose; Assuming that the number of initially matched node processes is n, each control branch is regarded as a player, and according to the motion characteristics of the virtual moving point and the action contribution of each player to the success of the collaborative operation, the roles of different players in the collaborative operation are identified; The action contribution includes, in each node process path, analyzing the current node and the next node respectively, and the degree of consistency with the current collaborative operation behavior; The local motion features are extracted through the convolutional neural network, and the time series characteristics of the virtual moving points under the collaborative operation are captured by the recurrent neural network, and the output is the feature vector ; The fully connected layer reduces the dimension of the feature vector to , generating a compact action representation vector ; in, represents the d-dimensional real number space, represents k-dimensional real number space; Network output: Each node The action representation vector is ; The feature vector of the current collaborative operation behavior is ; The degree of coincidence includes, for each action representation vector, determining its characteristic subspace through principal component analysis, and the basis matrix of the characteristic subspace is ; Will Projected onto the feature subspace of the node process superior: in, for exist Projection in subspace; Calculate projection error To quantify whether the feature vector belongs to the node feature subspace: Analyzing the projection vector The direction and target vector The degree of fit : in, express and Directional consistency; Combining projection error and direction consistency, define the action consistency score : in, is the maximum allowable value of the projection error; , is the weight parameter; If the score is less than the matching threshold, all players are considered collaborators and identified normally; If the score is not less than the matching threshold, it is determined that there is an interferer between the collaborators, and selective identification is performed to obtain only the identification results of the collaborators; After the recognition is completed, the node where the recognition result is located is determined, and its position between the current node and the next node. If the recognition result is the next node, the node process is updated, and the original next node is used as the current node to continue the recognition; In the n node processes, the paths that do not conform to the selected node sequence are eliminated through the selection of nodes by operation behavior, and nm node processes that conform to the current operation process are obtained; m represents the number of eliminated paths, m<n; The node process includes defining each link in the operation process and using it as a node in the process, connecting the nodes in series to obtain a node process after the connection; if the order of the nodes in the process is replaceable, then connecting different node orders in series according to the replaceable positions to generate all the node processes; The node flow corresponding to each operation purpose is constructed, and after the operation purpose is selected, the node flow after the series connection is automatically matched.

4. The method for collaborative recognition of multi-person actions based on virtual moving point motion capture technology according to claim 3, characterized in that: The selective identification includes randomly removing a control branch from all control branches, re-identifying the degree of fit of the current collaborative operation behavior, and if the score is less than the fit threshold, the removed control branch is judged to be an interferer; if the score is not less than the fit threshold, one more control branch is removed until the score is less than the fit threshold; In the visual interface, the node corresponding to the interferer is marked, and only participates in the identification process, but does not issue any operation instructions to the control end.

5. The method for collaborative recognition of multi-person actions based on virtual moving point motion capture technology according to claim 4, characterized in that: The determination of the feasible domain of the moving point includes, according to the three-dimensional model of each control branch, determining the number of virtual moving points in the blocked part, and obtaining virtual nodes directly connected to the blocked virtual moving points as reference nodes; Assume reference node A, determine the position A1 of an obscured virtual moving point according to the angle between reference node A and the connection part and the fixed distance from the adjacent node; determine the feasible domain a2 of the next obscured virtual moving point position according to the rotatable angle of the connection part of A1 and the fixed distance from the adjacent node; randomly select a point in a2 as the position A2 of the obscured virtual moving point, and determine the next virtual moving point, until the positions of all obscured virtual moving points are determined, and a position combination of obscured virtual moving points is obtained; The operation of randomly selecting virtual moving point positions in the feasible domain is continuously repeated until no new virtual moving point position combinations are generated; The optimal output includes dynamically analyzing all position combinations, combining any one of the position combinations with other unobstructed virtual moving points as a collaborative state, analyzing the action contribution, finding the combination with the highest degree of consistency and outputting it as the optimal result, and color-coding the virtual moving point recognition results of the obstructed part in the visualization interface.

6. A multi-person action collaborative recognition system based on virtual moving point motion capture technology using the method according to any one of claims 1 to 5, characterized in that: The acquisition unit deploys multiple cameras to capture the operator's motion information from different perspectives and perform three-dimensional reconstruction; A capture unit captures the moving points of the reconstructed model in three-dimensional space; The recognition unit, when an operation occurs, uses game theory to perform collaborative action recognition on the operation without occlusion; For operations with occlusion, the feasible domain of the moving point is judged, prediction is made based on the judgment result, and the optimal output is selected as the recognition result based on the prediction result.

Images