A robust multi-view skeleton fusion method based on information prioritization and effective joint masking

Abstract

This invention belongs to the field of computer vision and action recognition, specifically relating to a robust multi-view skeleton fusion method based on information priority selection and effective joint masking. It includes: acquiring skeleton data from multiple viewpoints, performing data preprocessing and effective joint extraction; skeleton registration based on information content to obtain a registered skeleton; obtaining the skeleton most suitable for action recognition based on the registered skeleton and a GCN model; obtaining the action category based on the most suitable skeleton using a modern action recognition model; and adjusting model parameters using cross-entropy loss based on the action category results to optimize iterative recognition accuracy. This invention effectively solves the problem of low-quality noise interfering with high-fidelity data in multi-view fusion through an information content evaluation mechanism and a Boolean mask completion strategy; combined with GCN adaptive viewpoint adjustment, it enhances the model's robustness to different camera layouts and significantly improves the accuracy and stability of human action recognition in complex occlusion environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and action recognition, specifically relating to a robust multi-view skeleton fusion method based on information priority selection and effective joint masking. Background Technology

[0002] In the field of human motion recognition, skeletal data-based recognition methods are widely used due to their strong robustness to changes in lighting and background interference. However, when single-view cameras collect skeletal data, the limited field of view, limb self-occlusion, or environmental object occlusion often lead to the loss of joint point information or coordinate drift, which seriously affects the recognition accuracy.

[0003] Multi-view skeletal fusion technology, by integrating sensor data from multiple angles, can theoretically compensate for blind spots in single-view applications. However, existing multi-view fusion methods face the following challenges: first, how to select the optimal reference information from perspectives of varying quality; second, how to efficiently align skeleton data in different coordinate systems; and third, how to handle local joint loss during the fusion process rather than simply performing numerical averaging to avoid noise interference. Therefore, developing a robust fusion method that prioritizes information quality and possesses effective joint masking capabilities is of great significance for improving the accuracy of action recognition in complex environments. Summary of the Invention

[0004] This invention provides a robust multi-view skeleton fusion method based on information priority selection and effective joint masking to solve the technical problems of high difficulty in aligning multi-view skeleton data and low information utilization under occlusion.

[0005] To address the aforementioned technical problems, one technical solution adopted by this invention is: a robust multi-view skeleton fusion method based on information-priority selection and effective joint masking, comprising:

[0006] S1. Collect skeleton data from multiple perspectives, perform data preprocessing and effective joint extraction;

[0007] S2. Obtain the registration skeleton based on information content;

[0008] S3. Based on the registration skeleton and GCN model, obtain the skeleton most suitable for recognizing the action;

[0009] S4. Based on the skeleton best suited for action recognition, obtain the action category using a modern action recognition model;

[0010] S5. Based on the action category results, adjust the model parameters using cross-entropy loss to optimize the iterative recognition accuracy.

[0011] Furthermore, the method in step S1 includes:

[0012] S11. Collect skeleton sequence data from multiple perspectives. Each frame of data contains key point coordinates and confidence scores. The original data dimensions are represented as (C, T, V, M), where C is the coordinate dimension, T is the number of frames, V is the number of key points, and M is the number of people scanned.

[0013] S12. Convert the data dimensions to (T, V, C) and apply them according to the confidence threshold. The criteria for extracting effective joints are: if the confidence level of a joint is greater than 100%. If the three-dimensional coordinates are non-zero, then it is considered a valid joint.

[0014] Furthermore, the skeleton registration method based on information content in step S2 includes:

[0015] S21. Calculate the information content of each viewpoint based on the number of effective joints, and determine the reference viewpoint;

[0016] S22. Solve the rigid body transformation matrix between the reference viewpoint and other viewpoints based on the Kabsch algorithm;

[0017] S23. Use Boolean mask logic to align and fill in the data from each perspective to generate the fused registration skeleton.

[0018] Furthermore, the method in step S21 includes:

[0019] The total number of effective joints in each frame is defined as the information content of that frame. The cumulative information content of each viewpoint over the entire sequence timeline is calculated, and the viewpoint with the largest cumulative information content is selected as the reference viewpoint skeleton. The remaining perspectives are used as the perspectives to be registered.

[0020] Furthermore, the method in step S22 includes:

[0021] (1) Calculate the reference point set separately With the set of points to be registered center of mass and :

[0022] , ;

[0023] in, Represents the reference point set The Middle The three-dimensional coordinate vectors of each effective joint point Represents the set of points to be registered The Middle The three-dimensional coordinate vectors of each effective joint point This represents the total number of valid joints corresponding to the two sets of points.

[0024] (2) Calculate the decentralized covariance matrix :

[0025] ;

[0026] (3) For the matrix Performing singular value decomposition (SVD) yields:

[0027] ;

[0028] (4) Calculate the rotation matrix and translation vector :

[0029] , , .

[0030] Furthermore, the method in step S23 includes:

[0031] Based on the following formula, missing joints are located and fused using a Boolean mask:

[0032] ;

[0033] in, For reference perspective skeleton, This is the transformed skeleton of the viewpoint to be registered. For frame index, It serves as a joint index; information is supplemented only when joint information is missing in the reference view and the corresponding joint is valid in the view to be registered.

[0034] Furthermore, the method in step S3 includes:

[0035] S31. The preliminary registration skeleton obtained in S2. Input the GCN model;

[0036] S32. Predict an adaptive rotation matrix using the GCN model according to the following formula. :

[0037] ;

[0038] in, This represents the feature mapping process of a graph convolutional neural network on a skeleton sequence. This represents mapping high-dimensional features to... Linear transformations of rotation matrices and orthogonalization constraint handling;

[0039] S33. Based on the following formula, the initial registration skeleton and Multiply to obtain the skeleton best suited for recognizing the action. :

[0040] .

[0041] Furthermore, the method in step S4 includes:

[0042] The skeleton sequence obtained from S3 is input into a two-stream rich activation map convolutional network. The Softmax function is used to output the probability distribution of each action category, and the category with the highest probability is selected as the final recognition result.

[0043] The Softmax function is shown in the following equation:

[0044] ;

[0045] in, The first element in the output vector of the fully connected layer of the network Scores for each category The total number of action categories. The probability that the input skeleton belongs to the first type of action.

[0046] Furthermore, the method in step S5 includes:

[0047] Based on predicted class probability and real labels The cross-entropy loss is calculated using the following formula. : ;

[0048] The weight parameters of GCN and action recognition model are updated using the backpropagation algorithm, and the recognition accuracy under occlusion and multi-view interference is improved through iterative training.

[0049] The beneficial effects of this invention are as follows: By employing a dynamic selection mechanism based on information content for the reference perspective, this invention ensures that multi-view data fusion always uses the highest-quality skeletal information as the benchmark, significantly enhancing the overall robustness of the system when facing severe occlusion or sensor interference. Furthermore, the directional fill-in fusion strategy implemented using Boolean masks effectively eliminates the contamination of high-quality coordinate information by low-confidence noise in traditional numerical averaging methods, maximizing the realism of the fused skeleton in physical space and geometric structure. Simultaneously, the system, through the deep integration of physical registration using the Kabsch algorithm and the GCN adaptive adjustment matrix, not only automatically corrects residual errors in physical alignment but also maps the skeleton coordinates to the standard recognition perspective most conducive to feature extraction, greatly improving the model's adaptability to different camera layout scenarios. Finally, through end-to-end cross-entropy loss closed-loop optimization, collaborative training of the fusion strategy and the backend recognition task is achieved, significantly improving the accuracy and stability of human action recognition in complex dynamic environments while effectively controlling computational overhead. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the overall architecture of a robust multi-view skeleton fusion method based on information priority selection and effective joint masking according to an embodiment of the present invention.

[0051] Figure 2 This is a flowchart illustrating a robust multi-view skeleton fusion method based on information priority selection and effective joint masking, according to an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments.

[0053] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0054] Example 1

[0055] See Figure 1 This embodiment provides a robust multi-view skeleton fusion method based on information-priority selection and effective joint masking. Its core logic lies in solving the common problems of limb occlusion and inconsistent data quality in human motion recognition by combining physical registration of multi-view data with deep learning calibration. Logically, the system can be divided into a data preprocessing module, an information-based spatial alignment module, an adaptive view calibration module, and a recognition optimization module.

[0056] See Figure 2 The method execution flow of this embodiment of the invention is as follows:

[0057] S1. Collect skeleton data from multiple perspectives, perform data preprocessing and effective joint extraction.

[0058] Specifically, the method in step S1 includes:

[0059] S11. Acquire skeleton sequence data from multiple perspectives. In a specific application scenario of this invention, each frame of data includes joint coordinates and confidence level. The original data dimensions are represented as (C, T, V, M), where C is the coordinate dimension, T is the frame number, V is the number of joints, and M is the number of people scanned. In this embodiment of the invention, the feature dimensions of the skeleton sequence cover the original coordinates of nodes, inter-frame motion velocity, and bone length information.

[0060] S12. Convert the data dimensions to (T, V, C) and apply them according to the confidence threshold. Extracting valid joints. In a preferred embodiment of the invention, the filtering threshold of joint confidence is used. The value is set to 0.1. The criterion is: if the confidence level of a joint is greater than 0.1 and the three-dimensional coordinates are not zero, it is considered a valid joint; otherwise, it is considered missing information.

[0061] S2. Obtain the registration skeleton based on information content skeleton.

[0062] Specifically, the skeleton registration method based on information content in step S2 includes:

[0063] S21. Calculate the information content of each viewpoint based on the number of effective joints and determine the reference viewpoint. Specifically, the total number of effective joints in each frame is defined as the information content of that frame. The cumulative information content of each viewpoint over the entire sequence timeline is calculated, and the viewpoint with the largest cumulative information content is selected as the reference viewpoint skeleton. The remaining perspectives are used as the perspectives to be registered. .

[0064] S22. Solve the rigid body transformation matrix between the reference viewpoint and other viewpoints based on the Kabsch algorithm. Specifically, this includes:

[0065] (1) Calculate the reference point set separately With the set of points to be registered center of mass and :

[0066] , ;

[0067] in, Represents the reference point set The Middle The three-dimensional coordinate vectors of each effective joint point Represents the set of points to be registered The Middle The three-dimensional coordinate vectors of each effective joint point This represents the total number of valid joints corresponding to the two sets of points.

[0068] (2) Calculate the decentralized covariance matrix :

[0069] ;

[0070] (3) For the matrix Performing singular value decomposition (SVD) yields:

[0071] ;

[0072] (4) Calculate the rotation matrix and translation vector :

[0073] , , .

[0074] S23. Align and pad the data from each viewpoint using Boolean mask logic to generate the fused registration skeleton. Specifically, based on the following formula, missing joints are located and fused using a Boolean mask:

[0075] ;

[0076] in, For reference perspective skeleton, This is the transformed skeleton of the viewpoint to be registered. For frame index, For joint indexing.

[0077] Information is supplemented only when joint information is missing in the reference view and the corresponding joint is valid in the view to be registered. This strategy avoids the spatial geometric distortion caused by simple averaging fusion.

[0078] S3. Based on the registration skeleton and GCN model, obtain the skeleton most suitable for recognizing the action.

[0079] Specifically, the method in step S3 includes:

[0080] S31. The preliminary registration skeleton obtained in S2. Input the GCN model;

[0081] S32. Predict an adaptive rotation matrix using the GCN model according to the following formula. :

[0082] ;

[0083] in, This represents the feature mapping process of a graph convolutional neural network on a skeleton sequence. This represents mapping high-dimensional features to... Linear transformations of rotation matrices and orthogonalization constraint handling;

[0084] S33. Based on the following formula, the initial registration skeleton and Multiply to obtain the skeleton best suited for recognizing the action. :

[0085] .

[0086] S4. Based on the skeleton best suited for action recognition, obtain the action category using a modern action recognition model.

[0087] Specifically, the skeleton sequence obtained in S3 is input into a two-stream activation map-rich convolutional network. The Softmax function is used to output the probability distribution of each action category, and the category with the highest probability is selected as the final recognition result. In this embodiment, the number of input channels is set to 9 to make full use of spatial coordinates and kinematic features.

[0088] S5. Based on the action category results, adjust the model parameters using cross-entropy loss to optimize the iterative recognition accuracy.

[0089] In this embodiment, the Adam optimizer is used during training, with an initial learning rate preferably of 0.001 and a batch size of 32. The model weights are updated using the backpropagation algorithm to achieve robust recognition under occlusion and multi-view interference.

[0090] To verify the technical effectiveness of this invention in cases of missing human body parts, a comparative test was conducted on the NTU RGB+D 60 dataset. This dataset contains 60 action categories. By performing spatial part masking on the test set, a subset including missing left arm (Part 1), right arm (Part 2), both hands (Part 3), both legs (Part 4), and torso (Part 5) was simulated.

[0091] Specifically, the performance comparison between the method (Ours) and the single-view recognition method (Baseline) provided by this invention is shown in Table 1.

[0092] Table 1. Overall performance of different methods under various occlusion conditions

[0093]

[0094] Experimental results show that the method of this invention exhibits extremely strong stability when faced with missing parts. Especially in the extreme case of "missing both hands", the single-view recognition accuracy is severely compromised, but the method of this invention successfully completes the discriminative features by selecting the viewpoint and using Boolean masking, maintaining an accuracy of 85.41%, demonstrating its superiority in complex occlusion environments.

[0095] To verify the technical contribution of the spatial registration module in step S2 to eliminating geometric bias, the robust fusion method provided by this invention is compared with the naive fusion method without spatial alignment. The results are shown in Table 2.

[0096] Table 2. Performance comparison between the proposed method and single-view recognition methods

[0097]

[0098] Test data reveals that, in all test scenarios, the skeleton fusion accuracy after physical alignment is significantly better than the unaligned version. Particularly under complex occlusion conditions, the alignment module provides an average gain exceeding 17%. This demonstrates that the present invention, by eliminating geometric deviations between viewpoints through the Kabsch algorithm, is a prerequisite for effective feature mining and can significantly improve the accuracy of multi-view recognition tasks.

[0099] Example 2:

[0100] The computer-readable storage medium of this embodiment stores a computer program that, when executed by a processor, implements the steps of the robust multi-view skeleton fusion method based on information priority selection and effective joint masking in Embodiment 1.

[0101] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0102] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0103] Example 3:

[0104] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the robust multi-view skeleton fusion method based on information priority selection and effective joint masking of Embodiment 1.

[0105] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0106] Those skilled in the art will understand that the content disclosed in the embodiments can be provided as a method, system, or computer program product. Therefore, this solution can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this solution can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0107] This solution is described with reference to flowchart illustrations and / or block diagrams of methods and computer program products according to embodiments of this solution. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0111] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A robust multi-view skeleton fusion method based on information-priority selection and effective joint masking, characterized in that, include: S1. Collect skeleton data from multiple perspectives, perform data preprocessing and effective joint extraction; S2. Obtain the registration skeleton based on information content; S3. Based on the registration skeleton and GCN model, obtain the skeleton most suitable for recognizing the action; S4. Based on the skeleton best suited for action recognition, obtain the action category using a modern action recognition model; S5. Based on the action category results, adjust the model parameters using cross-entropy loss to optimize the iterative recognition accuracy.

2. The method according to claim 1, characterized in that, The method in step S1 includes: S11. Collect skeleton sequence data from multiple perspectives. Each frame of data contains key point coordinates and confidence scores. The original data dimensions are represented as (C, T, V, M), where C is the coordinate dimension, T is the number of frames, V is the number of key points, and M is the number of people scanned. S12. Convert the data dimensions to (T, V, C) and apply them according to the confidence threshold. θ The criteria for extracting effective joints are: if the confidence level of a joint is greater than 100%. θ If the three-dimensional coordinates are non-zero, then it is considered a valid joint.

3. The method according to claim 1, characterized in that, The skeleton registration method based on information content in step S2 includes: S21. Calculate the information content of each viewpoint based on the number of effective joints, and determine the reference viewpoint; S22. Solve the rigid body transformation matrix between the reference viewpoint and other viewpoints based on the Kabsch algorithm; S23. Use Boolean mask logic to align and fill in the data from each perspective to generate the fused registration skeleton.

4. The method according to claim 3, characterized in that, The method of step S21 includes: The total number of effective joints in each frame is defined as the information content of that frame. The cumulative information content of each viewpoint over the entire sequence timeline is calculated, and the viewpoint with the largest cumulative information content is selected as the reference viewpoint skeleton. The remaining perspectives are used as the perspectives to be registered.

5. The method according to claim 4, characterized in that, The method of step S22 includes: (1) Calculate the reference point set separately With the set of points to be registered center of mass and : ， ； in, Represents the reference point set The Middle The three-dimensional coordinate vectors of each effective joint point Represents the set of points to be registered The Middle The three-dimensional coordinate vectors of each effective joint point This represents the total number of valid key points corresponding to the two sets of point sets; (2) Calculate the decentralized covariance matrix : ； (3) For the matrix Performing singular value decomposition (SVD) yields: ； (4) Calculate the rotation matrix and translation vector : ， ， 。 6. The method according to claim 5, characterized in that, The method of step S23 includes: Based on the following formula, missing joints are located and fused using a Boolean mask: ； in, For reference perspective skeleton, This is the transformed skeleton of the viewpoint to be registered. For frame index, It serves as a joint index; information is supplemented only when joint information is missing in the reference view and the corresponding joint is valid in the view to be registered.

7. The method according to claim 1, characterized in that, The method of step S3 includes: S31. The preliminary registration skeleton obtained in S2. Input the GCN model; S32. Predict an adaptive rotation matrix using the GCN model according to the following formula. : ； in, This represents the feature mapping process of a graph convolutional neural network on a skeleton sequence. This represents mapping high-dimensional features to... Linear transformations of rotation matrices and orthogonalization constraint handling; S33. Based on the following formula, the initial registration skeleton and Multiply to obtain the skeleton best suited for recognizing the action. : 。 8. The method according to claim 1, characterized in that, The method of step S4 includes: The skeleton sequence obtained from S3 is input into a two-stream rich activation map convolutional network. The Softmax function is used to output the probability distribution of each action category, and the category with the highest probability is selected as the final recognition result. The Softmax function is shown in the following equation: ； in, The first element in the output vector of the fully connected layer of the network Scores for each category The total number of action categories. The probability that the input skeleton belongs to the first type of action.

9. The method according to claim 1, characterized in that, The method of step S5 includes: Based on predicted class probability and real labels The cross-entropy loss is calculated using the following formula. : ； The weight parameters of GCN and action recognition model are updated using the backpropagation algorithm, and the recognition accuracy under occlusion and multi-view interference is improved through iterative training.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the robust multi-view skeleton fusion method based on information-priority selection and effective joint masking as described in any one of claims 1-9.

Images