Four-limb motion posture detection method based on multi-modal multi-view video feature fusion

Abstract

The invention discloses a limb movement posture detection method based on multi-modal multi-view video feature fusion, relates to the field of computer vision, and aims to solve the problems of incomplete single-view data information, limited single-modal feature expression and low posture detection accuracy in the prior art. According to the method, two-dimensional videos and three-dimensional skeleton data of the front face, the left side face and the right side face are synchronously obtained, a double-flow neural network is used for processing different modal data and achieving deep fusion, complementary information of a main view angle and a side view angle is fused through a cross-view-angle attention mechanism, and finally accurate detection of the movement postures of the four limbs is achieved.

Description

Technical Field

[0001] This relates to the field of computer vision, specifically to a method for detecting limb movement postures based on multimodal and multi-view video feature fusion. Background Technology

[0002] Motion posture detection based on limb video data is of great significance in various fields such as sports science, human-computer interaction, and intelligent monitoring. Accurate motion posture detection can provide reliable technical support for applications such as athlete technique analysis, fitness guidance, and behavior monitoring.

[0003] Multi-view refers to acquiring information about the same object from multiple different observation angles. In this context, it specifically refers to collecting human limb movement data from three different perspectives: front, left, and right. Multimodal refers to acquiring data using multiple different types of sensors or devices. In this context, it specifically refers to simultaneously using a monocular camera to acquire two-dimensional image data and a binocular camera to acquire three-dimensional skeletal data.

[0004] Existing computer-aided diagnostic technologies are mostly based on single-view, single-modal data to build models, but their inherent limitations lead to dual bottlenecks in geometric representation and dynamic analysis. From a geometric consistency perspective, a single view can only capture two-dimensional projection information. The same two-dimensional pose may correspond to multiple three-dimensional spatial poses; for example, an arm extending forward and retracting will have the same shape in projection. In addition, due to the lack of parallax information and depth perception, single-view systems cannot distinguish between forward and backward displacements of objects in three-dimensional space. For example, the flexion and extension movements of human joints may present the same projection shape in a two-dimensional plane, leading to the accumulation of errors in the reconstruction of the skeletal chain topology. Traditional single-view reconstruction methods rely on the assumption of rigid objects, but human motion involves non-rigid deformation, which breaks the projection geometric constraints, amplifying the reconstruction error from ±3cm in 2D keypoint localization to ±15° in 3D pose estimation.

[0005] The advantages of monocular vision modality are low equipment cost, convenient data acquisition, and rich texture information, making it suitable for detailed feature extraction; the disadvantages are lack of depth information, susceptibility to changes in viewing angle, and limited accuracy in 3D pose estimation. The advantages of 3D skeletal modality are providing accurate spatial position information, being unaffected by lighting and texture, and being suitable for motion trajectory analysis; the disadvantages are high equipment cost, complex data acquisition, lack of appearance texture information, and susceptibility to occlusion.

[0006] In dynamic scene modeling, single-view models face the dual challenges of motion blur and temporal modeling. When a target is performing high-speed movements such as rotation or jumping, the motion blur in a single frame can lead to inaccurate localization of key points, such as the blurred trajectory of the shoulder joint during rapid arm swings. Traditional CNN models, due to the limitations of their local receptive fields, struggle to capture the spatiotemporal relationships across frames.

[0007] To overcome the shortcomings of existing technologies, a method for limb movement posture detection based on multimodal and multi-view fusion is proposed. This method simultaneously acquires 2D video and 3D skeletal data from the front, left, and right sides. It then uses a two-stream neural network to process the different modal data and achieve deep fusion. Finally, it fuses complementary information from the main view and side view through a cross-view attention mechanism, thereby achieving accurate detection of limb movement posture. Summary of the Invention

[0008] To address the problems of incomplete single-view data, limited single-modal feature representation, and low pose detection accuracy in existing technologies, this paper proposes a limb motion pose detection method based on multimodal and multi-view fusion. This method first performs preprocessing operations on motion images to improve data quality, and then constructs a two-stream neural network to complete multimodal feature fusion training: on one hand, a two-dimensional image convolutional neural network (CNN) is used to extract spatial texture features from motion images, while a three-dimensional image convolutional neural network is simultaneously used to model the spatiotemporal correlation and structural constraints of motion poses, integrating two-dimensional visual information and three-dimensional structural information through a "two-stream network fusion" mechanism; in the multi-view fusion training stage, feature representations from the main view image and the side view image are fused to compensate for the blind spots of single-view observation and enhance feature robustness; finally, motion pose detection is achieved based on the fused features.

[0009] To achieve the above objectives, the following technical solution is adopted:

[0010] Step S1: Motion Image Preprocessing

[0011] Step S11: Collect data on limb movements

[0012] A standardized data acquisition method was established, requiring participants to perform core movements, each repeated three times for 10 seconds (this can be dynamically adjusted according to the experimental objectives). This ensured coverage of the movement states of key joints in the limbs (such as the shoulder, elbow, wrist, hip, knee, and ankle). The sampling frequency formula was used to lock the device frame rate at 25fps, ensuring that the temporal resolution of the data acquisition met the motion capture requirements. Furthermore, to fully record the movement states of key joints in the limbs, a key joint coverage rate was introduced to quantify and evaluate the completeness of the data acquisition, ensuring comprehensive recording of the movement states of these joints.

[0013] Step S12: Synchronize multi-view data

[0014] After inputting multi-view 2D video and 3D skeletal sequences, lens distortion may exist. This distortion is eliminated by calibrating the intrinsic parameters of a monocular camera, establishing a mapping relationship between pixel coordinates and physical coordinates. Simultaneously, to better facilitate spatial transformation between binocular cameras, the extrinsic parameters of the binocular cameras are calibrated, providing a foundation for stereo vision. Then, by introducing parallax dB, the 2D image coordinates are converted into 3D spatial coordinates to reconstruct the human pose. The output includes a corrected multi-view single-frame image sequence, a synchronized 3D skeletal joint coordinate sequence, and the camera's intrinsic and extrinsic parameter matrices.

[0015] Step S13: Data Quality Enhancement and Standardization

[0016] Enhancing the quality and standardization of data is crucial to the success of the invention. Therefore, based on steps S11 and S12, the corrected multi-view images and synchronized three-dimensional skeleton data are input. The collected image data is then subjected to illumination quality assessment, motion blur detection, and removal of abnormal movements. Finally, a high-quality multi-view two-dimensional image set, standardized three-dimensional skeleton sequence data, and a quality assessment report are output.

[0017] Step S2: Multimodal Neural Network Fusion

[0018] Step S21: Extract two-dimensional planar features

[0019] Convolutional Neural Networks (CNNs) are feedforward neural networks that incorporate convolutional computations and have a deep structure. The convolutional structure reduces the memory footprint of deep networks. Using CNNs as the basic structure, this project takes a high-quality, multi-view 2D image set as input, extracts planar features, and outputs multi-view 2D feature vectors (64 dimensions per view) and intermediate layer feature maps.

[0020] Step S22: Extract 3D spatial features

[0021] Graph Convolutional Networks (GCNs) are neural network models specifically designed for processing graph data. They extract relationships and features between nodes through graph convolution operations. Taking standardized 3D skeleton sequence data as input, a hierarchical graph convolutional network is used for feature extraction, outputting 3D skeleton feature vectors and joint correlation weight matrices.

[0022] Step S23: Cross-modal feature fusion and enhancement.

[0023] A 2D CNN is used to capture local textures, and a 3D GCN is used to model global motion, forming spatiotemporally complementary features. Inputting multi-view 2D feature vectors and 3D skeletal feature vectors, the 2D feature dimension is first expanded from 64 to 128 to align with the 3D features; the 2D texture feature weight is 0.4, and the 3D skeletal feature weight is 0.6. Finally, a weighted fusion is performed to preserve the original feature complementarity and avoid information overload, outputting a multimodal fusion feature vector (128 dimensions) and a modality contribution analysis report.

[0024] Step S3: Multi-view dynamic fusion based on multimodal neural network

[0025] There are certain limitations to fusing multimodal neural networks with the main viewpoint early on. By further fusing the two-dimensional and three-dimensional feature dimensions of the main viewpoint and the side viewpoint, and inputting the multimodal fused feature vector, the spatiotemporal interaction relationship across viewpoints can be modeled, which can effectively make up for the blind spots of single-view observation and enhance feature robustness. Finally, based on the fused features, motion posture detection is realized, and the motion posture detection results, anomaly confidence scores, and assessment of the degree of anomaly of key joints are output. Attached Figure Description

[0026] Figure 1 It is a method flowchart. Detailed Implementation

[0027] Step S1: Motion Image Preprocessing

[0028] Step S11: Collect data on limb movements

[0029] For a set of motion data, let N be the number of repetitions of a single action and T be the duration of a single action. Using the formula... Calculate the sampling frequency for this operation to ensure that the temporal resolution of the data acquisition meets the requirements for motion capture.

[0030] To ensure the integrity of the acquired data, a key joint coverage rate is set. Multi-view 2D video and 3D skeleton sequences are input, and the corrected multi-view single-frame image sequence, synchronized 3D skeleton joint coordinate sequence, and camera intrinsic and extrinsic parameter matrix are calculated and output. The specific formulas are as follows:

[0031]

[0032] Where A i Let A be the pixel coverage area of ​​the i-th joint (shoulder, elbow, wrist, etc.) in the image. total C represents the total effective area of ​​the image, and C represents the coverage rate of key joints (%).

[0033] Step S12: Synchronize multi-view data

[0034] To better eliminate lens distortion and establish a mapping relationship between pixel coordinates and physical coordinates, a monocular camera intrinsic parameter matrix K is set. The inputs are multi-view 2D video and 3D skeleton sequences. The outputs are a corrected multi-view single-frame image sequence, a synchronized 3D skeleton joint coordinate sequence, and the camera intrinsic and extrinsic parameter matrices. The specific details are as follows:

[0035]

[0036] Among them, f x fy c is the focal length. x c y The coordinates of the main point.

[0037] After collecting the image data, it is necessary to perform a conversion between two-dimensional and three-dimensional space to provide a foundation for stereo vision. The extrinsic parameters of the binocular camera are calibrated, and the PnP algorithm is introduced to solve the camera pose using n 3D-2D point pairs. The specific formula is as follows:

[0038] [R|T]=solvePnP(objPoints,imgPoints,K,D)

[0039] Where R represents the rotation attitude of the camera coordinate system relative to the world coordinate system, T represents the translation position of the camera coordinate system relative to the world coordinate system, and the two are concatenated to form the extrinsic parameter matrix [R|T]. objPoints are the calibration points in the 3D world coordinate system, imgPoints are the 2D image coordinates, D is the distortion coefficient, and K is the monocular camera intrinsic parameter matrix.

[0040] To recover three-dimensional human pose or motion structure from two-dimensional images, 3D reconstruction is achieved through depth calculation based on parallax dB, thus solving the parallax problem. The specific calculation formula is as follows:

[0041]

[0042] Where Z is the target point depth, B is the baseline distance of the binocular camera, and dB is the parallax between the left and right images.

[0043] Step S13: Data Quality Enhancement and Standardization

[0044] To assess the sufficiency of lighting in images depicting limb movements, thereby filtering out invalid frames from low-light environments and ensuring image sharpness, a lighting quality evaluation formula is established. This formula involves accessing each pixel (i,j) in the image and summing the results. The sum is then divided by the total number of pixels M·N in the image to obtain the average lighting intensity L of the image. avg The specific details are as follows:

[0045]

[0046] Among them, L avg Let I be the average illumination intensity (lux) of the image, and let I(i,j) be the brightness value of the pixel in the i-th row and j-th column. M and N represent the number of rows and columns of the two-dimensional array I, respectively.

[0047] During limb movements, blurring can occur due to motion. To quantify the degree of blur in the data, a motion blur detection formula is introduced, which evaluates image sharpness by analyzing changes in image gradients. First, the gradient of the image is calculated to obtain I.grad Then, the image gradient variance is obtained using the variance calculation function Var(), as detailed below:

[0048]

[0049] Among them, 255 2 The maximum possible squared value of the pixel value is used as a normalization factor to ensure that B score Within the range of 0-1, Var (Igrad) Let B be the image gradient variance. score For fuzzy fractions, when B score A frame is considered valid if its value is ≥0.5.

[0050] Finally, to eliminate abnormal actions, a reasonable threshold for abnormal actions is dynamically set based on the monitored speed threshold using statistical methods, thereby improving data reliability. The specific formula for speed monitoring is as follows:

[0051] vthresh=μv+3σv

[0052] Where vthresh is the anomaly detection threshold, μv is the average historical motion speed, and σv

[0053] This represents the standard deviation of the velocity.

[0054] By combining multiple formulas for filtering based on illumination, ambiguity, and motion speed, the high quality of the input data for this invention is ensured.

[0055] Step S2: Fusion of two-stream neural networks

[0056] Step S21: Extract two-dimensional planar features

[0057] Based on the image data processed in step S1, a high-quality multi-view 2D image set is input, and 2D planar features are extracted. First, features are extracted from convolutional layer 1 (3×3 convolution). An image with a height of 500, a width of 250, and 64 channels is input with a stride of 1. ReLU is used as the activation function, and the specific formula is as follows:

[0058] F c1 =σ(W c1 *I+b c1 )

[0059] Where σ is the ReLU activation function, Wc1 is the convolution kernel weight, I is the input image, and b c1 ∈R k For the offset top, F c1 This is the output feature map of convolutional layer 1.

[0060] The feature map is then reduced in dimensionality using pooling layer 1 (2×2 max pooling) with a stride of 2, resulting in a feature map of size 250×125×64. This preserves salient features while reducing spatial dimensionality, thus improving computational efficiency. The specific dimensionality reduction process is as follows:

[0061] F p1 =MaxPool(F c1 (kernel_size=2, stride=2)

[0062] Where F p1 The pooling layer outputs a feature map; MaxPool is the maximum pooling operation; F c1 This is the output feature map of convolutional layer 1, kernel_size=2 is the pooling kernel size, and stride=2 is the stride.

[0063] Repeat the above steps, passing through convolutional layer 2 (3×3 convolution) with a stride of 1, outputting feature map 2 with dimensions of 250×125×128. Then, pass through pooling layer 2 (2×2 max pooling) to reduce the dimensionality of the feature map, with a stride of 2, outputting feature map with dimensions of 125×63×128. To eliminate position sensitivity and enhance feature invariance, the two-dimensional feature map is compressed into a global one-dimensional feature vector, with the specific formula as follows:

[0064]

[0065] Among them, F 2D The global average pooling outputs a one-dimensional feature vector, where H′ and W′ are the feature maps F. c2 Spatial dimension, F c2 To output a two-dimensional feature map for convolutional layer 2, F c2 (i,j,:) represents all channel features at the i-th row and j-th column of the feature map.

[0066] Step S22: Extract 3D spatial features

[0067] By capturing long-range dynamic constraints between joints through spectral graph convolution, global features are compressed and fed back to local features to enhance the trunk-limb coordinated movement pattern. Multi-scale feature extraction based on a three-layer graph convolutional network establishes a "local-global" feature foundation, constructs a cross-layer semantic association matrix Θ, aggregates features from the graph convolutional layers, and performs deep feature learning on the graph structure data of skeletal joints through activation functions. The specific aggregation process is as follows:

[0068]

[0069] Among them, H (l+1) The feature matrix of the (l+1)th layer nodes, D is the degree matrix, W (l) Let θ be the learnable weight matrix, Θ be the cross-layer semantic association matrix, and σ be the activation function.

[0070] To further enhance the correlation of cross-joint movement patterns, such as the temporal dependency of shoulder-elbow-wrist, a cross-layer feature transfer mechanism is established. This mechanism integrates multi-scale features by concatenating the output features of different graph convolutional layers (GCN1, GCN2, GCN3), thereby enhancing the expressive power of the features. The specific formula is as follows:

[0071] F gcn(3) =[GCN1(F skel ); GCN2(F skel ),GCN3(F skel ))

[0072] Among them, F gcn(3) GCNi represents the multi-scale features after fusion, and GCNi is the output of the convolution of the i-th layer graph.

[0073] Finally, global average pooling is used to compress the temporal dimension, transforming the temporal sequence features into a fixed-length global feature vector. This extracts statistically invariant features of actions, suppresses temporal noise, and preserves details and texture. The specific details are as follows:

[0074]

[0075] Among them, F 3D Here, T represents the global feature vector of the 3D skeleton, and T is the duration of the skeleton sequence. The sum of output features of the third layer graph convolution at time t.

[0076] Step S23: Cross-modal feature fusion and enhancement

[0077] To align the two-dimensional and three-dimensional feature dimensions and facilitate further fusion and enhancement of cross-modal features, the 1×1 convolution dimension in the two dimensions is expanded, as shown in the following formula:

[0078]

[0079] in, Expand the weights for the channels. For the extended two-dimensional features, Let F be the convolution weight matrix. 2D For the original two-dimensional features, It is a 1×1 convolution bias vector.

[0080] After aligning the feature dimensions of the two features, the high-dimensional concatenated features are transformed into intermediate features W that can generate dynamic weights by learning the association patterns of cross-modal features. g This allows the model to flexibly adjust the importance of 2D texture features and 3D skeletal features according to different motion scenarios. The 0-1 weights are generated using the Sigmoid function, as detailed below:

[0081]

[0082] Among them, For the extended two-dimensional features, For three-dimensional skeletal features, W g Let g be the weight projection matrix, g be the dynamic weight vector, and σ be the Sigmoid activation function. Then, by adding the expanded 2D visual features to the dynamically weighted 3D skeletal features, the optimal fusion of the two modalities is achieved, preserving their respective advantages and complementing their shortcomings. The specific formula is as follows:

[0083]

[0084] Among them, F fusion The fused feature vector For the extended two-dimensional features, g is a dynamic weight vector whose generation depends on the collaborative information of the two modal features.

[0085] Step S3: Multi-view dynamic fusion based on multimodal neural network

[0086] Step S31: Dynamic fusion of main view and side view

[0087] To more effectively fuse multiple perspectives, a linear transformation is used to map 2D features to a unified semantic space, eliminating perspective differences. The multimodal fusion feature vector is input, and the features of the main viewpoint and side viewpoints are encoded separately, as follows:

[0088]

[0089] Among them, Q main Q side These are the encoded main-view and side-view spatial-semantic features, respectively. For learnable query weight matrix, These are 2D feature maps of the main view and the side view, respectively.

[0090] The first-view (frontal view) image is input into a two-stream network to generate initial pose estimation results, and the coordinates of key joints are located using heatmap regression. Then, the second-view input image is used to extract the coordinates of human skeletal joints, constructing a skeletal graph structure with joints as nodes and bone connections as edges. The ST-GCN network is then used to process the skeletal sequence, outputting the second-view skeletal features. The specific formula is as follows:

[0091] Q side =ST-GCN side (I side )

[0092] Among them, Q sideFor side view, ST-GCN side (I side ) is a spatiotemporal graph convolutional network specifically for side-view applications, I side Side-view input image.

[0093] The Softmax formula, which calculates attention weights across viewpoints, first calculates the feature similarity between the main viewpoint and the side viewpoint, and then normalizes it using the Softmax formula. A larger weight indicates that the features of the two joints are more similar. The specific formula is as follows:

[0094]

[0095] Where αij represents the attention weight of the j-th joint in the side view to the i-th joint in the main view.

[0096] sim(·) is the cosine similarity function, exp is the exponential function, and ∑ k Summing over all k, F 2k The k-th joint feature of the main perspective, F 3l This is the l-th joint feature from the side view.

[0097] The weighted contribution of side-view features is calculated through a cross-view attention mechanism. Based on the reliability of the main and side-view features, the cross-view fusion coefficient is calculated in real time. Then, the fusion ratio of the two view features is controlled by Across to achieve dynamic weighted fusion. The specific content is as follows:

[0098] F fusion =Across·Q side +(1-Across)·Q main

[0099] Where Across is the cross-view fusion coefficient, which adapts to the scene, F fusion Multi-view features after fusion.

[0100] Through the above operations, the dominant features of the main perspective can be effectively preserved, while supplementary information from the side perspective can be integrated to achieve dynamic fusion of multiple perspectives.

[0101] Step S32: Attitude detection

[0102] To address the issue of pose detection relying on post-processing, anomaly detection is integrated into network training. Based on the pose estimation branch, a binary classification branch (e.g., normal / abnormal) is added, sharing underlying convolutional features. A hybrid loss function is then introduced to jointly optimize the pose estimation error and the anomaly classification loss. The primary task is assigned a larger weight, while the secondary task is assigned a smaller weight. Through cross-validation, the optimal weight combination was determined to be 0.6, 0.3, and 0.1. The specific details are as follows:

[0103] Ltotal =λ1L pose +λ2L cls +λ3L reg

[0104] Where λ1, λ2, λ3 are loss weight coefficients of 0.6, 0.3, and 0.1, respectively, and L reg This is a regularization term to prevent overfitting.

[0105] During the inference phase, if the confidence level of the pose estimation in a certain frame is lower than the threshold, an early warning mechanism is triggered and a prompt is made to recollect data.

[0106] Potential future application areas include: (1) expansion into the fields of medical diagnosis and rehabilitation, such as diagnosis of neuromuscular diseases, monitoring of rehabilitation treatment, and assessment of child development; (2) sports science and the sports industry, such as technical analysis and injury prevention for professional athletes, intelligent guidance systems for mass fitness, and research tools for sports biomechanics; (3) human-computer interaction and virtual reality, such as intelligent gesture control and posture recognition, VR / AR games and training systems, and humanoid robot control.

Claims

1. A method for limb motion pose detection based on multimodal and multi-view video feature fusion, characterized in that, Includes the following steps: Step S1: Motion Image Preprocessing Step S11: Collect data on limb movements Establish standardized data collection methods to record the movement status of key joints in the limbs; Step S12: Synchronize multi-view data After inputting multi-view 2D video and 3D skeleton sequence, lens distortion may exist. This distortion is eliminated by calibrating the intrinsic parameters of a monocular camera and establishing a mapping relationship between pixel coordinates and physical coordinates. At the same time, in order to better complete the spatial transformation between binocular cameras, the extrinsic parameters of the binocular cameras are also calibrated. Then, by introducing parallax dB, the 2D image coordinates are converted into 3D spatial coordinates to reconstruct the human posture. The output is a corrected multi-view single-frame image sequence, a synchronized 3D skeleton joint coordinate sequence, and camera intrinsic and extrinsic parameter matrices. Step S13: Data Quality Enhancement and Standardization Step S2: Multimodal Neural Network Fusion Step S21: Extract two-dimensional planar features Using CNN as the basic structure, the system takes a multi-view two-dimensional image set as input, extracts planar features, and outputs multi-view two-dimensional feature vectors and intermediate layer feature maps. Step S22: Extract 3D spatial features Graph Convolutional Network (GCN) extracts relationships and feature information between nodes through graph convolution operations. Input standardized 3D skeleton sequence data, extract features through a hierarchical graph convolutional network, and output 3D skeleton feature vector and joint correlation weight matrix; Step S23: Cross-modal feature fusion and enhancement; A 2D CNN is used to capture local textures, and a 3D GCN is used to model global motion, forming spatiotemporally complementary features. The inputs are multi-view 2D feature vectors and 3D skeleton feature vectors. First, the 2D feature dimension is expanded from 64 to 128 to align with the 3D features. The 2D texture feature weight is 0.4, and the 3D skeleton feature weight is 0.

6. Finally, the features are weighted and fused to retain the original feature complementarity and avoid information overload. The output is a 128-dimensional multimodal fusion feature vector and a modal contribution analysis report. Step S3: Multi-view dynamic fusion based on multimodal neural network After the multimodal neural network is fused with the main viewpoint, the two-dimensional and three-dimensional feature dimensions of the main viewpoint and the side viewpoint are further fused and the multimodal fused feature vector is input to model the spatiotemporal interaction relationship across viewpoints. This can effectively make up for the blind spots of single-viewpoint observation and enhance the robustness of features. Finally, based on the fused features, motion posture detection is realized, and motion posture detection results, anomaly confidence scores, and assessment of the degree of anomaly of key joints are output.

2. The method according to claim 1, characterized in that: Step S1: Motion Image Preprocessing Step S11: Collect data on limb movements For a set of motion data, let N be the number of repetitions of a single action and T be the duration of a single action. Using the formula... Calculate the sampling frequency for this data acquisition to ensure that the temporal resolution of the data acquisition meets the motion capture requirements. The input consists of multi-view 2D video and 3D skeleton sequence. The output consists of a corrected multi-view single-frame image sequence, a synchronized 3D skeleton joint coordinate sequence, and a camera intrinsic and extrinsic parameter matrix. The specific formulas are as follows: Where A i Let A be the pixel coverage area of ​​the i-th joint in the image. total C represents the total effective area of ​​the image, and C represents the coverage rate of key joints (%). Step S12: Synchronize multi-view data Define the intrinsic parameter matrix K of a monocular camera, input multi-view 2D video and 3D skeleton sequence, and output corrected multi-view single-frame image sequence, synchronized 3D skeleton joint coordinate sequence, and camera intrinsic and extrinsic parameter matrix through calculation. The specific content is as follows: Among them, f x f y c is the focal length. x c y Principal point coordinates; After collecting the image data, it is necessary to perform a conversion between two-dimensional and three-dimensional space to provide a foundation for stereo vision. The extrinsic parameters of the binocular camera are calibrated, and the PnP algorithm is introduced to solve the camera pose using n 3D-2D point pairs. The specific formula is as follows: [R|T]=solvePnP(objPoints,imgPoints,K,D) Where R is the rotation attitude of the camera coordinate system relative to the world coordinate system, T is the translation position of the camera coordinate system relative to the world coordinate system, and the two are matrix-concatenated to form the extrinsic parameter matrix [R|T], objPoints are the calibration object points in the three-dimensional world coordinate system, imgPoints are the two-dimensional image coordinates, D is the distortion coefficient, and K is the monocular camera intrinsic parameter matrix. 3D reconstruction is achieved through depth calculation based on parallax dB, solving the parallax problem. The specific formula is as follows: Where Z is the target point depth, B is the baseline distance of the binocular camera, and dB is the parallax between the left and right images; Step S13: Data Quality Enhancement and Standardization To assess the sufficiency of lighting in images depicting limb movements, thereby filtering out invalid frames from low-light environments and ensuring image sharpness, a lighting quality evaluation formula is established. This formula involves accessing each pixel (i,j) in the image and summing the results. The sum is then divided by the total number of pixels M·N in the image to obtain the average lighting intensity L of the image. avg The specific details are as follows: Among them, L avg Let I be the average illumination intensity (lux) of the image, I(i,j) be the brightness value of the pixel in the i-th row and j-th column, and M and N represent the number of rows and columns of the two-dimensional array I, respectively. During limb movements, blurring can occur due to motion. To quantify the degree of blur in the data, a motion blur detection formula is introduced, which evaluates image sharpness by analyzing changes in image gradients. First, the gradient of the image is calculated to obtain I. grad Then, the image gradient variance is obtained using the variance calculation function Var(), as detailed below: Among them, 255 2 The maximum possible squared value of the pixel value is used as a normalization factor to ensure that B score Within the range of 0-1, Var (Igrad) Let B be the image gradient variance. score For fuzzy fractions, when B score A frame is considered valid if its value is ≥0.

5. The formula for dynamically setting an abnormal action threshold using statistical methods is as follows: vthresh=μv+3σv Where vthresh is the anomaly detection threshold, μv is the average historical motion speed, and σv This represents the standard deviation of the velocity.

3. The method according to claim 1, characterized in that: Step S2: Fusion of two-stream neural networks Step S21: Extract two-dimensional planar features Based on the image data processed in step S1, a high-quality multi-view 2D image set is input, and 2D planar features are extracted. First, the features of convolutional layer 1 are extracted. An image with a height of 500, a width of 250, and 64 channels is input, with a stride of 1. ReLU is used as the activation function, and the specific formula is as follows: F c1 =σ(W c1 *I+b c1 ) Where σ is the ReLU activation function, W c1 Here, I represents the convolution kernel weights, and b represents the input image. c1 ∈R k For the offset top, F c1 This is the output feature map of convolutional layer 1; The feature map is then reduced in dimensionality using pooling layer 1 with a stride of 2, resulting in a feature map of size 250×125×64. This preserves salient features while reducing spatial dimensionality, thus improving computational efficiency. The specific dimensionality reduction process is as follows: F p1 =MaxPool(F c1 ,kernel_size=2,stride=2) Where F p1 The pooling layer outputs a feature map; MaxPool is the maximum pooling operation; F c1 This is the output feature map of convolutional layer 1, kernel_size=2 is the pooling kernel size, stride=2 is the stride; Repeat the above operations, passing through convolutional layer 2 with a stride of 1, outputting feature map 2 with dimensions of 250×125×128, then passing through pooling layer 2 to reduce the dimensionality of the feature map with a stride of 2, outputting feature map with dimensions of 125×63×128; compressing the two-dimensional feature map into a global one-dimensional feature vector, the formula is: Among them, F 2D The global average pooling outputs a one-dimensional feature vector, where H′ and W′ are the feature maps F. c2 Spatial dimension, F c2 To output a two-dimensional feature map for convolutional layer 2, F c2 (i,j,:) represents all channel features at the i-th row and j-th column of the feature map; the convolution is a 3×3 convolution and the pooling is a 2×2 max pooling; Step S22: Extract 3D spatial features By capturing long-range dynamic constraints between joints through spectral graph convolution, global features are compressed and fed back to local features to enhance the trunk-limb coordinated movement pattern. Multi-scale feature extraction based on a three-layer graph convolutional network establishes a "local-global" feature foundation, constructs a cross-layer semantic association matrix Θ, aggregates features from the graph convolutional layers, and performs deep feature learning on the graph structure data of skeletal joints through activation functions. The specific aggregation process is as follows: Among them, H (l+1) The feature matrix of the (l+1)th layer nodes, D is the degree matrix, W (l) Θ is the learnable weight matrix, θ is the cross-layer semantic association matrix, and σ is the activation function; A cross-layer feature transfer mechanism is established, which concatenates the output features of different graph convolutional layers (GCN1, GCN2, and GCN3) to integrate multi-scale features and enhance their expressive power; the specific formula is as follows: F gcn(3) =[GCN1(F skel );GCN2(F skel ),GCN3(F skel )) Among them, F gcn(3) For the fused multi-scale features, GCNi is the output of the i-th layer graph convolution; Finally, global average pooling is used to compress the temporal dimension, transforming the temporal sequence features into a fixed-length global feature vector. This extracts statistically invariant features of actions, suppresses temporal noise, and preserves details and texture. The specific steps are as follows: Among them, F 3D Here, T represents the global feature vector of the 3D skeleton, and T is the duration of the skeleton sequence. (t,:) represents the sum of output features of the third-layer graph convolution at time t; Step S23: Cross-modal feature fusion and enhancement To align the two-dimensional and three-dimensional feature dimensions and facilitate further fusion and enhancement of cross-modal features, the 1×1 convolution dimension in the two dimensions is expanded, as shown in the following formula: in, Expand the weights for the channels. For the extended two-dimensional features, Let F be the convolution weight matrix. 2D For the original two-dimensional features, It is a 1×1 convolution bias vector; After aligning the feature dimensions of the two features, the high-dimensional concatenated features are transformed into intermediate features W that can generate dynamic weights by learning the association patterns of cross-modal features. g This allows the model to flexibly adjust the importance of 2D texture features and 3D skeletal features according to different motion scenarios; 0-1 weights are generated using the Sigmoid function, as detailed below: Among them, For the extended two-dimensional features, For three-dimensional skeletal features, W g Let g be the weight projection matrix, g be the dynamic weight vector, and σ be the Sigmoid activation function. Then, by adding the expanded 2D visual features to the dynamically weighted 3D skeletal features, the optimal fusion of the two modalities is achieved, preserving their respective advantages and complementing their shortcomings. The specific formula is as follows: Among them, F fusion The fused feature vector For the extended two-dimensional features, g is a dynamic weight vector whose generation depends on the collaborative information of the two modal features.

4. The method according to claim 1, characterized in that: Step S3: Multi-view dynamic fusion based on multimodal neural network Step S31: Dynamic fusion of main view and side view To more effectively fuse multiple perspectives, a linear transformation is used to map 2D features to a unified semantic space, eliminating perspective differences. The multimodal fusion feature vector is input, and the features of the main viewpoint and side viewpoints are encoded separately, as follows: Among them, Q main Q side These are the encoded main-view and side-view spatial-semantic features, respectively. For learnable query weight matrix, These are 2D feature maps of the main view and the side view, respectively. The first-view (frontal) image is input into a two-stream network to generate initial pose estimation results, and the coordinates of key joints are located by heatmap regression. Then, the coordinates of human skeletal joints are extracted from the second-view input image to construct a skeletal graph structure, with joints as nodes and bone connections as edges. The ST-GCN network is then used to process the skeletal sequence and output the second-view skeletal features. The specific formula is as follows: Q side =ST-GCN side (I side ) Among them, Q side For side view, ST-GCN side (I side (I) is a spatiotemporal graph convolutional network specifically for side-view applications. side Side-view input image; The Softmax formula, which calculates attention weights across viewpoints, first calculates the feature similarity between the main viewpoint and the side viewpoint, and then normalizes it using the Softmax formula. A larger weight indicates that the features of the two joints are more similar. The specific formula is as follows: Where αij represents the attention weight of the j-th joint in the side view to the i-th joint in the main view. sim(·) is the cosine similarity function, exp is the exponential function, and ∑ k Summing over all k, F 2k The k-th joint feature of the main perspective, F 3l This is the feature of the l-th joint from a side view. The weighted contribution of side-view features is calculated through a cross-view attention mechanism. Based on the reliability of the main and side-view features, the cross-view fusion coefficient is calculated in real time. Then, the fusion ratio of the two view features is controlled by Across to achieve dynamic weighted fusion. The specific content is as follows: F fusion =Across·Q side +(1-Across)·Q main Where Across is the cross-view fusion coefficient, which adapts to the scene, F fusion Multi-view features after fusion; Step S32: Attitude detection To address the issue of pose detection relying on post-processing, anomaly detection is integrated into network training. Based on the pose estimation branch, a binary classification branch (normal or anomaly) is added, sharing underlying convolutional features. Furthermore, a hybrid loss function is introduced to jointly optimize the pose estimation error and the anomaly classification loss. Through cross-validation, the optimal weight combinations were finally determined to be 0.6, 0.3, and 0.

1. The specific details are as follows: L total =λ1L pose +λ2L cls +λ3L reg Where λ1, λ2, λ3 are loss weight coefficients of 0.6, 0.3, and 0.1, respectively, and L reg This is a regularization term.

Images