Space interaction accurate identification method and system based on multi-modal fusion

Abstract

The invention provides a space interaction accurate recognition method and system based on multi-modal fusion, and relates to the technical field of artificial intelligence, and the method comprises the steps: collecting human skeleton, visual image and voice information, generating a space-time attention map through employing a feature extraction network, enhancing visual features, and executing feature complementary correction. Fusing the three types of modal information to obtain unified feature representation; when the recognition confidence is low, the space-time convolution generative adversarial network guided based on the action causal relationship graph complements the missing image frame, and the accuracy and robustness of space interaction recognition are improved.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and particularly to a method and system for accurate recognition of spatial interaction based on multimodal fusion. Background Art

[0002] With the development of human-computer interaction technology, spatial interaction technology, as an important human-computer interaction method, has been widely applied in fields such as smart home, intelligent driving, and virtual reality. Traditional spatial interaction recognition methods mainly rely on single-modal data, such as visual images or skeletal information, for action recognition. However, single-modal information often has difficulty accurately recognizing complex human actions, especially in practical application environments, where it is easily affected by factors such as lighting and occlusion.

[0003] Currently, the main multimodal fusion methods include two ways: feature-level fusion and decision-level fusion. Feature-level fusion is to fuse multimodal features at the feature extraction stage, while decision-level fusion is to make a comprehensive decision on the results of individual recognition of each modality. The existing spatial interaction recognition technology based on multimodal fusion has the following defects and deficiencies: The multimodal feature fusion method in the existing technology lacks an effective spatio-temporal attention mechanism and cannot make full use of the spatio-temporal features of joint points in human skeletal information to enhance visual image information, resulting in limited expression ability of the fused features and difficulty in accurately recognizing complex spatial interaction actions.

[0004] When the existing multimodal fusion method processes different modal data, it often ignores the complementarity and consistency between modalities and lacks an effective feature complementary correction mechanism, resulting in large information loss during the fusion process and affecting the final recognition accuracy.

[0005] The existing technology lacks an effective solution to the problem of data loss in practical application environments. Especially when there is a missing frame phenomenon in visual image information, it is unable to perform effective data completion, resulting in low confidence in the recognition results and affecting the reliability and stability of the system. Summary of the Invention

[0006] The embodiments of the present invention provide a method and system for accurate recognition of spatial interaction based on multimodal fusion, which can solve the problems in the existing technology.

[0007] In the first aspect of the embodiments of the present invention, a method for accurate recognition of spatial interaction based on multimodal fusion is provided, including: Collect human skeletal information, visual image information, and voice information; Input the human body bone information and visual image information into a feature extraction network. The feature extraction network generates a spatio-temporal attention map based on the spatio-temporal features of the joint points in the human body bone information, and applies the spatio-temporal attention map to the visual image information to obtain enhanced visual features; perform feature complementary correction on the human body bone information and the enhanced visual features; based on temporal modeling, feature alignment, and cross-modal attention mechanism, fuse the corrected human body bone features and visual features with the speech information features to obtain a unified feature representation; Perform action recognition on the unified feature representation. When the confidence level of the recognition result is lower than the preset confidence threshold, for the missing image frames in the visual image information, complete them based on the spatio-temporal convolutional generative adversarial network guided by the action causal relationship graph to obtain the completed visual image information; re-fuse the completed visual image information with the human body bone information and the speech information to obtain the final action recognition result.

[0008] In an alternative embodiment, The feature extraction network generates a spatio-temporal attention map based on the spatio-temporal features of the joint points in the human body bone information, and applying the spatio-temporal attention map to the visual image information to obtain enhanced visual features includes: Obtain a bone sequence containing T frames. Each frame of the bone sequence contains N joint points, and each joint point includes two-dimensional coordinates and confidence information. The two-dimensional coordinates are used to represent the position of the joint point in the image; calculate the displacement vector of the joint point based on the two-dimensional coordinates of adjacent frames; Construct the bone sequence into a graph structure. The graph structure includes a vertex set and an edge set. The vertex set corresponds to the joint points in the bone sequence, and the edge set corresponds to the bone connections in the bone sequence; input the graph structure into a graph convolutional network to enhance the features of the joint points based on the set of neighbor nodes of the joint points to obtain enhanced joint point features; perform multi-scale spatio-temporal feature extraction on the enhanced joint point features, generate spatio-temporal position encoding and fuse it with the spatio-temporal features to obtain encoded features, and calculate the spatio-temporal self-attention features; perform weight fusion and constraint optimization on the spatio-temporal self-attention features to obtain a spatio-temporal attention map; Calculate the motion amplitude value in the spatio-temporal attention map based on the displacement vector. The motion amplitude value is determined by the Euclidean norm of the displacement vector at the corresponding position; normalize the motion amplitude value to obtain an enhancement coefficient, and the product of the enhancement coefficient and the spatio-temporal attention map acts on the visual image information to enhance the feature expression of the target action area to obtain enhanced visual features.

[0009] In an alternative embodiment, Generating a spatio-temporal attention map includes: Perform multi-scale downsampling on the enhanced joint point features to obtain a multi-scale spatial feature map, and perform temporal convolution operations on the multi-scale spatial feature map within a preset time window to obtain temporal dynamic features; Generate spatio-temporal position encoding, and fuse the spatio-temporal position encoding with the temporal dynamic features to obtain encoded features; calculate spatio-temporal self-attention features based on the encoded features, and the spatio-temporal self-attention features are used to characterize the spatio-temporal dependence relationship between joint points; Perform global average pooling on the spatio-temporal self-attention features to obtain feature importance weights, and apply the feature importance weights to the multi-scale spatial feature map and the temporal dynamic features to obtain an initial attention map; Optimize the initial attention map based on temporal smoothness constraints, spatial structure constraints, and sparsity constraints, and perform boundary enhancement to obtain a spatio-temporal attention map.

[0010] In an alternative embodiment, Performing feature complementary correction on the human body bone information and enhanced visual features includes: Based on the human body bone information and enhanced visual features, calculate the reliability of bone features and the reliability of visual features; the reliability of bone features is obtained by calculating the geometric consistency, motion smoothness, and structural integrity of joint points, and the reliability of visual features is obtained by calculating temporal consistency and scene context; When it is detected that the reliability of bone features is lower than the first threshold, correct based on the features at the corresponding positions in the visual image information; when it is detected that the reliability of visual features is lower than the second threshold, correct the features based on the joint point positions in the human body bone information; when the reliability of bone features is not lower than the first preset threshold and the reliability of visual features is not lower than the second preset threshold, then use the human body bone information and enhanced visual features as the corrected bone features and visual features.

[0011] In an alternative embodiment, Performing feature complementary correction includes: Generate a complementary correction weight matrix based on the reliability of bone features and the reliability of visual features; perform two-way correction on the bone features and enhanced visual features using the complementary correction weight matrix, including: weighting the appearance information of the enhanced visual features with the first correction weight to obtain a spatial correction map, and fusing the spatial correction map with the bone features to obtain corrected bone features; weighting the motion trajectory information of the bone features with the second correction weight to obtain a motion correction map, and fusing the motion correction map with the enhanced visual features to obtain corrected visual features; Perform temporal consistency and cross-modal consistency verification on the corrected skeletal features and visual features to generate a residual correction term, and fuse the residual correction term with the corresponding features to obtain the final corrected skeletal features and visual features.

[0012] In an alternative embodiment, Perform action recognition on the unified feature representation. When the confidence of the recognition result is lower than the preset confidence threshold, for the missing image frames in the visual image information, complete them based on the spatio-temporal convolutional generative adversarial network guided by the action causal relationship graph, and the obtained completed visual image information includes: Perform action recognition on the unified feature representation, extract spatio-temporal features through a three-dimensional convolutional network and perform feature fusion by combining a temporal attention mechanism to obtain the action recognition result and its confidence; when the confidence of the action recognition result is lower than the preset confidence threshold, construct an action causal relationship graph based on hierarchical action decomposition and temporal dependence fusion; Construct a causal attention mechanism based on the action causal relationship graph, and integrate the causal attention mechanism into the generative adversarial network. The generative adversarial network includes a generator and a discriminator. The generator adopts an encoder-decoder structure and introduces the causal attention mechanism, and the discriminator performs spatio-temporal feature discrimination on the generated result; For the missing image frames in the visual image information, complete them using the generative adversarial network, and optimize by combining reconstruction loss, adversarial loss, and causal consistency constraint during the completion process. The causal consistency constraint is obtained by calculating the deviation between the generated result and the predicted action sequence based on the action causal relationship graph; perform temporal smoothness constraint on the completed visual image information to generate the final completion result.

[0013] In an alternative embodiment, Constructing an action causal relationship graph based on hierarchical action decomposition and temporal dependence fusion includes: Obtain an action sequence based on the unified feature representation, decompose the action sequence into an atomic action set, identify the action combination patterns in the atomic action set through action combination rules to generate a combined action set; calculate the action transition probabilities in the atomic action set and the combined action set to obtain the atomic action layer transition probability and the combined action layer transition probability, and calculate the mapping probability from atomic actions to combined actions and the decomposition probability from combined actions to atomic actions; Construct a dynamic time window that includes adjacent actions before and after the target action, calculate the local temporal dependency score based on the dynamic time window; count the co-occurrence times of actions in the action sequence to construct an action co-occurrence matrix, and calculate the global temporal dependency score; perform weighted fusion on the local temporal dependency score and the global temporal dependency score to obtain a fused dependency score, and update the transition probabilities of the atomic action layer, the combined action layer, the mapping probability, and the decomposition probability based on the fused dependency score; Use the actions in the atomic action set and the combined action set as nodes, and perform online adjustment on the updated transition probabilities of the atomic action layer, the combined action layer, the mapping probability, and the decomposition probability based on the action recognition result, and perform normalization post-processing as the weights of the edges to construct an action causal relationship graph with a hierarchical structure.

[0014] In the second aspect of the embodiments of the present invention, a spatial interaction precise recognition system based on multimodal fusion is provided, including: A first unit for collecting human body skeleton information, visual image information, and voice information; A second unit for inputting the human body skeleton information and visual image information into a feature extraction network. The feature extraction network generates a spatio-temporal attention map based on the spatio-temporal features of the joint points in the human body skeleton information, and applies the spatio-temporal attention map to the visual image information to obtain enhanced visual features; perform feature complementary correction on the human body skeleton information and the enhanced visual features; based on temporal modeling, feature alignment, and cross-modal attention mechanism, fuse the corrected human body skeleton features and visual features with the voice information features to obtain a unified feature representation; A third unit for performing action recognition on the unified feature representation. When the confidence level of the recognition result is lower than the preset confidence threshold, for the missing image frames in the visual image information, complete them based on the spatio-temporal convolutional generative adversarial network guided by the action causal relationship graph to obtain the completed visual image information; re-fuse the completed visual image information with the human body skeleton information and voice information to obtain the final action recognition result.

[0015] In the third aspect of the embodiments of the present invention, an electronic device is provided, including: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0016] In the fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0017] The present invention realizes the effective fusion of multi-modal information and improves the accuracy of spatial interaction action recognition by collecting three types of modal data, namely human bone information, visual image information, and voice information, and enhancing visual features based on spatio-temporal attention maps.

[0018] The present invention adopts a feature complementary correction mechanism to complement and correct human bone features and visual features, and combines time series modeling, feature alignment, and cross-modal attention mechanisms to solve the heterogeneity problem between different modal information, enabling each modal information to complement and enhance each other, and improving the robustness of action recognition.

[0019] The present invention introduces a spatio-temporal convolutional generative adversarial network guided by an action causal relationship graph to perform visual image frame completion processing for low recognition confidence cases, effectively solving problems such as image loss and occlusion that may occur in practical applications, and further improving the recognition performance and adaptability of the system in complex environments. Brief Description of the Drawings

[0020] Figure 1 It is a schematic flow chart of the spatial interaction precise recognition method based on multi-modal fusion according to an embodiment of the present invention; Figure 2 It is a comparison chart of attention precision for multi-scale spatio-temporal feature fusion and triple constraint optimization; Figure 3 It is a performance comparison chart of action prediction accuracy. Detailed Embodiments

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0022] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0023] Figure 1 It is a schematic flow chart of the spatial interaction precise recognition method based on multi-modal fusion according to an embodiment of the present invention, as Figure 1 shown, the method includes: Collect human bone information, visual image information, and voice information; Input the human body bone information and visual image information into a feature extraction network. The feature extraction network generates a spatio-temporal attention map based on the spatio-temporal features of the joint points in the human body bone information, and applies the spatio-temporal attention map to the visual image information to obtain enhanced visual features; perform feature complementary correction on the human body bone information and the enhanced visual features; based on temporal modeling, feature alignment, and cross-modal attention mechanism, fuse the corrected human body bone features and visual features with the speech information features to obtain a unified feature representation; Perform action recognition on the unified feature representation. When the confidence level of the recognition result is lower than the preset confidence threshold, for the missing image frames in the visual image information, complete them based on the spatio-temporal convolutional generative adversarial network guided by the action causal relationship graph to obtain the completed visual image information; re-fuse the completed visual image information with the human body bone information and the speech information to obtain the final action recognition result.

[0024] In an optional implementation manner, the feature extraction network generates a spatio-temporal attention map based on the spatio-temporal features of the joint points in the human body bone information, and applying the spatio-temporal attention map to the visual image information to obtain enhanced visual features includes: Obtain a bone sequence containing T frames. Each frame of the bone sequence contains N joint points, and each joint point includes two-dimensional coordinates and confidence information, where the two-dimensional coordinates are used to represent the position of the joint point in the image; calculate the displacement vector of the joint point based on the two-dimensional coordinates of adjacent frames; Construct the bone sequence into a graph structure, where the graph structure includes a vertex set and an edge set. The vertex set corresponds to the joint points in the bone sequence, and the edge set corresponds to the bone connections in the bone sequence; input the graph structure into a graph convolutional network to enhance the features of the joint points based on the set of neighbor nodes of the joint points to obtain enhanced joint point features; perform multi-scale spatio-temporal feature extraction on the enhanced joint point features, generate spatio-temporal position encoding and fuse it with the spatio-temporal features to obtain encoded features, and calculate spatio-temporal self-attention features; perform weight fusion and constraint optimization on the spatio-temporal self-attention features to obtain a spatio-temporal attention map; Calculate the motion amplitude value in the spatio-temporal attention map based on the displacement vector, where the motion amplitude value is determined by the Euclidean norm of the displacement vector at the corresponding position; normalize the motion amplitude value to obtain an enhancement coefficient, and the product of the enhancement coefficient and the spatio-temporal attention map acts on the visual image information to enhance the feature expression of the target action area to obtain enhanced visual features.

[0025] Exemplarily, human skeleton sequence data containing T frames is obtained. T can be set to 64 frames, and each frame contains N joint points, where N can be 18 standard human joint points, such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankle joints, etc. Each joint point contains two-dimensional coordinates (x, y) and a confidence value c ranging from 0 to 1, which is used to represent the reliability of the detection of this joint point. For the i-th joint point between the t-th frame and the (t + 1)-th frame, its displacement vector is calculated as the current frame coordinates minus the previous frame coordinates. For example, for a waving action, there will be a large horizontal displacement of the wrist joint point between consecutive frames.

[0026] The skeleton sequence is constructed into a graph structure G=(V, E), where V is the vertex set and E is the edge set. The vertex set V corresponds to N joint points, and the edge set E corresponds to the bone connections. For example, there is an edge between the shoulder and the elbow, and an edge between the elbow and the wrist. In actual implementation, these connections can be predefined, such as the shoulder (number 1) is connected to the elbow (number 2), and the elbow is connected to the wrist (number 3), etc. The constructed graph structure is input into the graph convolutional network (GCN). GCN enhances the joint point features through the following steps: For each joint point i, determine its neighbor node set N(i), including the joint points directly connected to it. For example, for the elbow node, its neighbor node set includes the shoulder and the wrist. Then, calculate the spatial weight matrix W between the neighbor nodes, and the weight value w ij is calculated based on the Euclidean distance between joint points i and j: w ij = exp(-||p i - p j ||2 / σ2), where p i and p j are the spatial positions of nodes i and j respectively, and σ is an adjustable scaling parameter. In this embodiment, a 2-layer GCN structure is adopted, the input feature dimension is 3 (x, y, c), the middle layer feature dimension is 64, and the output feature dimension is 128. Taking the first layer of GCN as an example, for joint point i, its updated feature is the weighted sum of its own feature and the features of all connected joint points. In this way, the feature representation of each joint point not only contains its own information, but also fuses the structure and motion information of the neighbor nodes, so as to obtain enhanced joint point features.

[0027] Multi-scale spatio-temporal feature extraction is performed on the enhanced joint point features, generating spatio-temporal position encoding and fusing it with the spatio-temporal features, and finally obtaining the spatio-temporal attention map.

[0028] For the $i$-th joint point in the $t$-th frame, its motion amplitude value is determined by calculating the Euclidean norm of the displacement vector between two adjacent frames of this joint point. Specifically, the square of the difference in the abscissa of this joint point between the current frame $t$ and the next frame $t + 1$ is added to the square of the difference in the ordinate, and then the square root is taken to obtain the motion amplitude value. All motion amplitude values are subjected to min-max normalization to obtain an enhancement coefficient in the range of 0 to 1. The normalization method is as follows: subtract the minimum value among all motion amplitude values from the current motion amplitude value, and then divide by the difference between the maximum value and the minimum value among all motion amplitude values. To prevent the denominator from being zero, a very small constant, such as 0.00001, is additionally added to the denominator. Multiply the enhancement coefficient element-wise with the spatio-temporal attention map, and then apply it to the visual features extracted from the input video. The dimension of the visual features includes the time dimension $T$, the height $H$ of the feature map, the width $W$, and the number of channels $C$. For the region around each joint point, it is located in the feature map according to its two-dimensional coordinates, and then the features of this region are weighted and enhanced by the product of the corresponding enhancement coefficient and the attention value. For example, if the enhancement coefficient calculated for a certain joint point is 0.8 and the attention value at the corresponding position is 0.7, then the two values are multiplied to obtain 0.56 as the final weight coefficient for enhancing the features of this region. In this way, the finally obtained enhanced visual features prominently express the features of the target action region.

[0029] The present invention realizes effective feature enhancement of the target action region through the method of generating a spatio-temporal attention map based on the spatio-temporal features of skeletal joint points and acting on visual image information; constructs the skeletal sequence as a graph structure and extracts features through a graph convolutional network, combines multi-scale spatio-temporal feature extraction and self-attention mechanism to accurately capture the spatial relationship and temporal change between joint points; calculates the motion amplitude through the displacement vector and combines it with the spatio-temporal attention map to realize the adaptive enhancement of visual features, significantly improves the perception ability of the dynamic action region, enhances the discriminability and robustness of feature expression, and provides a more accurate feature representation for subsequent action recognition tasks.

[0030] In an optional implementation manner, generating the spatio-temporal attention map includes: Performing multi-scale downsampling on the enhanced joint point features to obtain multi-scale spatial feature maps, and performing temporal convolution operations on the multi-scale spatial feature maps within a preset time window to obtain temporal dynamic features; Generating spatio-temporal position encoding, fusing the spatio-temporal position encoding with the temporal dynamic features to obtain encoded features; calculating spatio-temporal self-attention features based on the encoded features, and the spatio-temporal self-attention features are used to characterize the spatio-temporal dependence relationship between joint points; Performing global average pooling on the spatio-temporal self-attention features to obtain feature importance weights, and applying the feature importance weights to the multi-scale spatial feature maps and the temporal dynamic features to obtain an initial attention map; Optimize the initial attention map based on temporal smoothing constraints, spatial structure constraints, and sparsity constraints, and perform boundary enhancement to obtain a spatio-temporal attention map.

[0031] Exemplarily, downsample the joint point features using convolutional kernels of different sizes. Use three different-sized convolutional kernels, such as 3×3, 5×5, and 7×7, to perform convolutional operations on the joint point features respectively. The stride corresponding to each convolutional kernel is set to 2, and the padding parameter is set to 1, thereby obtaining three feature maps of different scales respectively. For example, for joint point features with an input size of 64×25 (where 64 represents the feature dimension and 25 represents the number of joint points), after downsampling, three spatial feature maps with sizes of 32×13, 16×7, and 8×4 can be obtained. Adjust these three feature maps of different scales to the same spatial size, i.e., 32×13, through bilinear interpolation, and then perform weighted fusion based on feature importance. Higher weights are assigned to larger-scale feature maps, and the weights of the three scales can be set to 0.4, 0.35, and 0.25 respectively.

[0032] Perform temporal convolution operations on the multi-scale spatial feature maps within a preset time window to obtain temporal dynamic features. Set the length of the time window to T (e.g., T = 10 frames), perform one-dimensional temporal convolution on the multi-scale spatial feature maps within each time window, use a one-dimensional convolution with a kernel size of 3, a stride of 1, and a padding of 1 to keep the time dimension unchanged. For each scale of spatial feature map, the corresponding temporal dynamic features are obtained after temporal convolution.

[0033] Spatio-temporal position encoding consists of two parts: temporal position encoding and spatial position encoding. Temporal position encoding is used to represent the relative position relationship of different time frames, which is generated through a combination of sine and cosine functions. For each time step, a position encoding vector with the same dimension as the feature is generated, enabling features at different time steps to distinguish their temporal order relationships. Spatial position encoding is used to represent the relative position relationship of different joint points in the human skeleton. First, construct an adjacency matrix representing the joint point connection relationship, calculate the shortest path distance between any two joint points, and then, based on this distance information, generate an encoding vector for each joint point reflecting its relative position in the skeleton structure. Finally, concatenate the temporal position encoding and the spatial position encoding along the feature dimension to obtain the complete spatio-temporal position encoding.

[0034] Fuse the spatio-temporal position encoding with the temporal dynamic features to obtain the encoded features. The fusion operation is achieved through element-wise addition, that is, for each temporal dynamic feature, add it to the spatio-temporal position encoding of the corresponding dimension to obtain the encoded features. Calculate the spatio-temporal self-attention features based on the encoded features to represent the spatio-temporal dependence relationship between joint points. Specifically, pass the encoded features of each scale through three independent linear transformation layers to obtain the query vector, key vector, and value vector respectively. The output dimension of the linear transformation layer can be set to one-eighth of the input dimension. Then, perform matrix multiplication on the query vector and the key vector, and divide by the scaling factor (e.g., 8), and obtain the attention weight matrix after Softmax normalization; finally, multiply the attention weight matrix by the value vector to obtain the self-attention features. Multiple attention heads can be set, and each head independently calculates the attention features. Finally, concatenate the features of all heads and obtain the final spatio-temporal self-attention features through a linear transformation.

[0035] Perform average pooling operations on the spatio-temporal self-attention features in the temporal and spatial dimensions to obtain the channel-level feature vector, and then map it to between 0 and 1 through the sigmoid function to form the feature importance weight. For example, for the spatio-temporal self-attention features with a size of 64×T×25, a 64-dimensional weight vector is obtained after global average pooling. Multiply the feature importance weights with the multi-scale spatial feature maps and the temporal dynamic features at the channel level respectively, and then perform weighted summation on the results to form the initial attention map.

[0036] The temporal smoothness constraint is achieved by calculating the difference between adjacent temporal frame attention maps. The difference calculation uses the Euclidean distance metric. Calculate the difference values for each pair of adjacent frame attention maps and sum them up to obtain the temporal smoothness loss. The spatial structure constraint considers the natural connection relationship of the human body skeleton. For each pair of adjacent joint points, calculate the attenuation coefficient according to their distance in the skeleton map. The closer the joint points are, the greater the penalty for the difference in their attention values, and thus the spatial structure loss is obtained. The sparsity constraint is achieved by calculating the L1 norm of the attention map, which promotes the attention values to concentrate on a few key joint points. Combine the loss terms of these three constraints with weights that can be set to 0.3, 0.5, and 0.2 respectively to obtain the total constraint loss, which is used to optimize the initial attention map.

[0037] Boundary enhancement is achieved through the following steps: Apply Sobel operators in the horizontal and vertical directions to the optimized attention map for edge detection, obtaining edge response maps in the horizontal and vertical directions; Take the square root of the sum of the squares of the edge response maps in the two directions to obtain an edge intensity map; Normalize the edge intensity map so that its value range is between 0 and 1; Perform weighted fusion of the normalized edge intensity map and the original attention map, with the fusion weights set to 0.85 and 0.15. In this way, the boundaries between the joint regions in the attention map are enhanced, making the attention distribution of different joints clearer, and finally obtaining a spatio-temporal attention map with enhanced boundaries.

[0038] Figure 2 It is a comparison graph of the attention accuracy for multi-scale spatio-temporal feature fusion and triple-constraint optimization. In the graph, the horizontal axis represents the action complexity, with four levels from low to extremely high; the vertical axis represents the percentage of attention accuracy, ranging from 50% to 100%. The three comparison methods are as follows: The present invention (multi-scale + triple constraint): Adopt multi-scale feature extraction and triple optimization strategies of temporal smoothing constraint, spatial structure constraint, and sparsity constraint; Single-scale + single constraint: Only use single-scale feature extraction and a single constraint condition; Multi-scale + single constraint: Adopt multi-scale feature extraction but only use a single constraint condition; It can be clearly seen from the graph that under all complexities, the method of the present invention achieves the highest attention accuracy, especially in high-complexity and extremely high-complexity action scenarios. In the low-complexity scenario, the method of the present invention reaches an accuracy of 94.7%, while the single-scale + single constraint method is only 82.2%; in the extremely high-complexity scenario, the method of the present invention can still maintain a high accuracy of 83.2%, while the single-scale + single constraint method drops to 56.7%. This result fully demonstrates the superiority and stability of multi-scale spatio-temporal feature fusion and triple-constraint optimization in complex action scenarios. Especially as the action complexity increases, the performance gap between the method of the present invention and traditional methods becomes more significant.

[0039] The spatio-temporal attention map generation method proposed by the present invention extracts multi-scale spatio-temporal features of joints through multi-scale downsampling and temporal convolution operations, combines spatio-temporal position encoding to enhance the spatio-temporal expression ability of features; Introduces a spatio-temporal self-attention mechanism to capture complex dependencies between joints, and generates feature importance weights through global average pooling to achieve adaptive enhancement of features; Designs a triple-constraint optimization strategy and a boundary enhancement technique to ensure the temporal smoothness, spatial structure rationality, and regional focus of the attention distribution, effectively improving the accuracy and discriminability of the attention map, and providing high-quality guiding information for subsequent visual feature enhancement.

[0040] In an optional implementation manner, performing feature complementary correction on the human body bone information and enhanced visual features includes: Calculate the reliability of bone features and the reliability of visual features based on human bone information and enhanced visual features; the reliability of bone features is obtained by calculating the geometric consistency, motion smoothness, and structural integrity of joint points, and the reliability of visual features is obtained by calculating temporal consistency and scene context; When it is detected that the reliability of bone features is lower than the first threshold, correct based on the features at the corresponding position in the visual image information; when it is detected that the reliability of visual features is lower than the second threshold, correct the features based on the joint positions in the human bone information; when the reliability of bone features is not lower than the first preset threshold and the reliability of visual features is not lower than the second preset threshold, then use the human bone information and enhanced visual features as the corrected bone features and visual features.

[0041] Exemplarily, when calculating the reliability of bone features, the evaluation of geometric consistency mainly examines whether the distance between adjacent joint points conforms to the anatomical characteristics of the human bones. The system pre-stores the standard length ratio data of each bone segment of the human body. For example, the length ratio of the upper arm to the lower arm is about 1.1:1, and the length ratio of the thigh to the lower leg is about 1:1.1. During the detection process, calculate the distance between the actually detected adjacent joint points and compare it with the expected bone length. If the deviation between the actual length and the expected length of a certain bone segment exceeds 30%, the geometric consistency score of this part will be reduced. For example, if the detected length of the upper arm is 40 cm and the length of the lower arm is 60 cm, which significantly violates the proportional relationship of the normal human bones, the system will reduce the geometric consistency score of this part. For the evaluation of motion smoothness, analyze the motion trajectories of joint points in multiple consecutive frames. In normal human motion, the position changes of joint points should be smooth and continuous without sudden changes. The system calculates the change rate of each joint point position in 5 consecutive frames. If the position change of a certain joint point between adjacent frames exceeds the preset threshold (such as the distance change exceeds 30 cm or the angle change exceeds 45 degrees), it is considered that the motion smoothness of this joint point is low. For example, if it is detected that the elbow joint suddenly moves 50 cm between two consecutive frames while other parts of the human body change little, the system will reduce the motion smoothness score of this joint point. The structural integrity evaluation mainly examines whether the detected bone structure is complete, checks whether all key joint points (such as the head, shoulders, hips, knees, ankles, etc.) are detected, and whether the connection relationship between these joint points is correct. If the system detects the absence of some key joint points, or the connection relationship between joint points is abnormal (such as the left ankle is connected to the right knee), it is considered that the structural integrity of the bones is damaged. For example, in a certain detection, the system identifies all the joint points of the upper body, but only detects the joint points of the left leg in the lower body while the right leg is completely missing. At this time, the system will significantly reduce the structural integrity score.

[0042] Based on the scores from the above three aspects, the final reliability score of the skeletal features is calculated, ranging from 0 to 1. The higher the score, the higher the reliability.

[0043] When calculating the reliability of visual features, the temporal consistency assessment focuses on the changes in visual features in consecutive frames of images. It analyzes the degree of change in the feature vectors at each pixel position in 10 consecutive frames of images. For normal human movements, the visual features in the corresponding areas should have a certain degree of coherence. Calculate the standard deviation of the feature vectors in each area in the time dimension. If the standard deviation is too large (e.g., exceeding 200% of the area average), it is considered that the temporal consistency of this area is low. For example, when a person quickly moves from a bright area to a shaded area, it may cause a sudden change in the clothing color features, and the system will reduce the temporal consistency score of this part of the area. The scene context assessment examines the matching degree between visual features and the scene environment. It evaluates the reliability of visual features by analyzing environmental elements in the image (such as lighting conditions, background complexity). In an environment with a complex background or uneven lighting, the reliability of visual features is usually low. For example, in a strong backlight environment, the human body contour may become blurred, reducing the scene context score of the corresponding area. Another example is when there are objects in the background that are similar in color to the human clothing, it may cause unclear boundary discrimination, also reducing the scene context score of these areas.

[0044] Based on the scores from both the temporal consistency and scene context aspects, the system calculates the final reliability score of the visual features, ranging from 0 to 1.

[0045] When it is detected that the reliability of the skeletal features is lower than the first threshold (e.g., 0.6), the features at the corresponding positions in the visual image information will be used for correction. Locate the position of the corresponding joint point on the visual feature map, extract the visual features of this position and its surrounding area (such as a 20×20 pixel range), and then re - estimate the position of the joint point based on these visual features. For example, if the reliability of the skeletal features of the right knee joint is only 0.45 (lower than the threshold 0.6), the visual features of this area (such as clear leg edge lines) will be used to re - locate the accurate position of the right knee joint.

[0046] When it is detected that the reliability of the visual features is lower than the second threshold (e.g., 0.7), the features will be corrected based on the joint point positions in the human skeletal information. According to the reliable joint point positions, adjust the feature weights in the corresponding areas on the visual feature map, increase the weights of the features that are consistent with the skeletal structure, and reduce the weights of the inconsistent features. For example, if the system detects that the reliability of the visual features in a person's arm area is 0.65 (lower than the threshold 0.7), and the skeletal features in this area (such as the positions of the shoulder, elbow, and wrist joints) are relatively reliable, the system will adjust the visual features in the arm area according to these joint point positions to make them more consistent with the constraints of the skeletal structure.

[0047] The feature complementary correction method proposed by the present invention realizes the complementary advantages of multi-modal information by accurately evaluating the reliability of skeletal features and visual features; evaluates the reliability of skeletal features from three dimensions: geometric consistency, motion smoothness, and structural integrity, evaluates the reliability of visual features from two aspects: temporal consistency and scene context, and establishes a comprehensive feature quality evaluation system; the adaptive correction mechanism based on reliability scores can intelligently introduce complementary modal information for correction when single-modal information is unreliable, effectively solves the problem of feature degradation in complex scenarios, significantly improves the robustness and accuracy of feature representation, and provides high-quality feature input for subsequent action recognition tasks.

[0048] In an alternative embodiment, performing feature complementary correction includes: Generating a complementary correction weight matrix based on the reliability of skeletal features and the reliability of visual features; performing bidirectional correction on the skeletal features and the enhanced visual features by using the complementary correction weight matrix, including: weighting the appearance information of the enhanced visual features with the first correction weight to obtain a spatial correction map, and performing feature fusion on the spatial correction map and the skeletal features to obtain corrected skeletal features; weighting the motion trajectory information of the skeletal features with the second correction weight to obtain a motion correction map, and performing feature fusion on the motion correction map and the enhanced visual features to obtain corrected visual features; Performing temporal consistency and cross-modal consistency verification on the corrected skeletal features and visual features, generating a residual correction term, and fusing the residual correction term with the corresponding features to obtain the final corrected skeletal features and visual features.

[0049] Exemplarily, when generating the complementary correction weight matrix based on the reliability of skeletal features and the reliability of visual features, the reliability scores of skeletal features and visual features are first normalized to the range of 0 to 1. For each joint point position, if its skeletal feature reliability is 0.8 and its visual feature reliability is 0.6, the first correction weight is set to 0.6 and the second correction weight is set to 0.8, which reflects the supplementary effect of the high-reliability modality on the low-reliability modality.

[0050] During the bidirectional correction process, it is necessary to process the appearance information of the enhanced visual features. Taking the arm area as an example, if the visual features of this area contain 64 channels, each channel represents different appearance features, such as edges, textures, etc. Multiply these features by the first correction weight, and the areas with higher weight values (such as the elbow area with a reliability of 0.9) will retain the original features, while the areas with lower weight values (such as the wrist area with a reliability of 0.3) will be suppressed. The generated spatial correction map is then fused with the skeletal features, and the fusion method is weighted summation, and the fusion weights can be set to 0.6 and 0.4.

[0051] For the processing of the motion trajectory information of skeletal features, extract the position change sequence of each joint point within 10 consecutive frames. For example, for the wrist joint, record its displacements in the horizontal and vertical directions within 10 consecutive frames. Multiply these trajectory information by the second correction weight. For the joint points with reliable motion trajectories (such as the shoulder with a reliability of 0.85), retain their complete motion information; for the joint points with unreliable motion trajectories (such as the ankle with a reliability of 0.4), reduce their influence. The generated motion correction map is then fused with the enhanced visual features, and the fusion weights are set to 0.55 and 0.45.

[0052] In the temporal consistency verification step, perform a time window analysis on the corrected features, and set the window size to 15 frames. Analyze the feature change trends of each joint point in these frames, including position changes, appearance changes, etc. If the features of a certain joint point change drastically between consecutive frames, such as the position mutation exceeding 25 cm or the feature value change exceeding 70% of the original value, it is considered that there is temporal inconsistency. For example, if it is detected that the left hand suddenly has a displacement of 40 cm in the 8th frame, while the displacements in other frames are all within 5 cm, a correction term is generated for the features of the 8th frame. The cross-modal consistency verification mainly checks the matching degree between the skeletal features and the visual features. At the position of each joint point, compare the skeletal positioning result and the position indicated by the visual features. If the difference between the two exceeds the preset threshold (such as 15 pixels), a corresponding residual correction term is generated. For example, when the skeletal features show that the right elbow is located at the image coordinates (200, 300), while the position indicated by the visual features is (180, 320), calculate the position deviation and generate a correction amount. The generation of the residual correction term takes into account the verification results of temporal consistency and cross-modal consistency. For the right elbow in the above example, if the temporal consistency verification shows a position deviation of 10 pixels and the cross-modal consistency verification shows a position deviation of 20 pixels, a residual correction term with a size of 15 pixels is generated comprehensively. Fuse the residual correction term with the corresponding features, and the fusion ratio can be set to 0.85 for the original features and 0.15 for the residual correction term.

[0053] The present invention effectively utilizes the complementary advantages of the two modalities to make up for the deficiencies through the complementary correction mechanism, improving the reliability and accuracy of the features; through the bidirectional correction and multi-level verification mechanism, the quality and reliability of the features are significantly improved, providing a better feature basis for subsequent action recognition and behavior analysis tasks.

[0054] In an alternative embodiment, for the action recognition of the unified feature representation, when the confidence level of the recognition result is lower than the preset confidence threshold, for the missing image frames in the visual image information, complete them based on the spatio-temporal convolutional generative adversarial network guided by the action causal relationship graph, and the obtained completed visual image information includes: Perform action recognition on the unified feature representation, extract spatio-temporal features through a 3D convolutional network and combine a temporal attention mechanism for feature fusion to obtain the action recognition result and its confidence; when the confidence of the action recognition result is lower than the preset confidence threshold, construct an action causal relationship graph based on hierarchical action decomposition and temporal dependence fusion; Construct a causal attention mechanism based on the action causal relationship graph, and integrate the causal attention mechanism into a generative adversarial network. The generative adversarial network includes a generator and a discriminator. The generator adopts an encoder-decoder structure and introduces the causal attention mechanism, and the discriminator discriminates the spatio-temporal features of the generated result; For the missing image frames in the visual image information, use the generative adversarial network for completion, and optimize by combining reconstruction loss, adversarial loss and causal consistency constraint during the completion process. The causal consistency constraint is obtained by calculating the deviation between the generated result and the predicted action sequence based on the action causal relationship graph; perform temporal smoothness constraint on the completed visual image information to generate the final completion result.

[0055] Exemplarily, perform temporal modeling on the corrected bone features, visual features and speech features. For the bone feature sequence, use a bidirectional long short-term memory network to process 32 consecutive frames of data, set the number of hidden units to 256, and extract the motion feature information at each time step. For the visual feature sequence, use a temporal convolutional network to process 32 consecutive frames of image features, set the convolutional kernel size to 3, and the number of channels to 128, and extract the temporal dependence of the visual features. For the speech features, divide the audio signal into time segments aligned with the video frames, and extract 32-dimensional Mel-frequency cepstral coefficient features for each segment, and perform feature extraction through a one-dimensional convolutional network. During the feature alignment process, map the features of the three modalities to the same feature space, and uniformly set the feature dimension to 256 dimensions. Adopt a time window mechanism for feature alignment, and set the window size to 8 frames. Within each time window, calculate the similarity matrix between different modality features to guide feature alignment. For example, when it is detected that in a certain action segment, the movement of the hand joint points is highly correlated with the energy change of the speech signal, strengthen the alignment weight between these two modality features. The cross-modal attention mechanism realizes feature enhancement by calculating the cross-correlation between different modality features. For each modality, construct two feature transformation branches to generate query features and key-value features respectively. For example, the query features generated by the bone features can calculate the attention weight with the key-value features of the visual features to realize the visual feature enhancement guided by the bones. In order to balance the importance of different modalities, set learnable modality importance weights, and the initial values are all set to one-third. In this way, the features of the three modalities are fused into a unified feature representation, and the feature dimension remains 256 dimensions.

[0056] When performing action recognition on unified feature representations, a three-dimensional convolutional network is used to extract spatio-temporal features. The network consists of five convolutional modules, each of which is composed of two three-dimensional convolutional layers. The input of the first convolutional module is a feature sequence of 32 frames × 256 dimensions. Feature extraction is performed using a 3×3×3 convolutional kernel with a stride of 1, and the output number of feature channels is 64. In subsequent convolutional modules, the number of feature channels is increased to 128, 256, 512, and 512 in sequence, and at the same time, the spatio-temporal resolution of the features is reduced through a max-pooling layer with a stride of 2. To capture long-term dependencies in the action sequence, a temporal attention mechanism is introduced based on the convolutional features. The feature sequence is divided into 4 time segments, each segment containing 8 frames. For each time segment, the correlation score between it and other segments is calculated. The calculation of the correlation score takes into account the semantic similarity of the features and the time position information. The closer the time distance and the higher the similarity, the greater the attention weight corresponding to the segment. For example, in the waving action, the two key segments of the arm lifting and lowering will obtain a higher cross-correlation attention weight. The features with temporal attention weights added are mapped to the action category space through two fully connected layers. The output dimension of the first fully connected layer is 1024, and the output dimension of the second layer is the number of action categories. The probability distribution of each action category is obtained through the softmax function, and the category with the highest probability is used as the recognition result, and its probability value is used as the confidence level.

[0057] When the confidence level of the action recognition result is lower than the preset confidence threshold (such as 0.75), the visual image completion module is activated. An action causal relationship graph is constructed based on hierarchical action decomposition and temporal dependence fusion.

[0058] The construction process of the causal attention mechanism is as follows: Extract the set of previous action nodes and the set of possible subsequent action nodes at the current moment from the action causal relationship graph, each containing 5 most relevant action nodes. Perform feature embedding on these nodes, mapping each action node to a 64-dimensional vector, and at the same time mapping the edge relationship (transition probability) between nodes to a 32-dimensional vector. Process these node and edge features through a graph attention network, which contains 3 layers of graph attention layers, with 8 attention heads in each layer, and outputs a 128-dimensional graph feature vector. To achieve interaction with visual features, a cross-attention module is designed. This module interacts the visual feature map F (with a dimension of 512×8×8) output by the encoder with the graph feature vector G. First, expand G to a 512-dimensional vector through linear projection, calculate the dot product similarity between G and the features at each spatial position in F to form an attention map A (with a dimension of 8×8); finally, apply A to F to obtain a causally weighted feature map F' (with a dimension of 512×8×8). In addition, a dynamic gating mechanism is introduced to adaptively adjust the influence intensity of causal attention according to the current confidence. Specifically, map the confidence to a weight α between 0 and 1 through the sigmoid function, and the final feature is F'' = α·F' + (1 - α)·F, realizing the adaptive fusion of causal information and visual information. This causal attention mechanism effectively introduces action semantic knowledge into the visual generation process, ensuring that the generated results conform to action logic.

[0059] The generative adversarial network consists of two parts: a generator and a discriminator. The generator adopts an encoder-decoder structure. The encoder processes the input image sequence and maps it to a latent feature space; the decoder then reconstructs the latent features into the target image. The encoder contains 6 downsampling convolutional blocks, and each convolutional block consists of a convolutional layer, an instance normalization layer, and a LeakyReLU activation function. The convolutional layer uses a 4×4 convolutional kernel, with a stride of 2 and a padding of 1, and the number of channels is 64, 128, 256, 512, 512, 512 in sequence. The decoder contains 6 upsampling transposed convolutional blocks, and each block consists of a transposed convolutional layer, an instance normalization layer, and a ReLU activation function. The transposed convolutional layer uses a 4×4 convolutional kernel, with a stride of 2 and a padding of 1, and the number of channels is 512, 512, 256, 128, 64, 3 (RGB channels) in sequence.

[0060] The causal attention mechanism is integrated into the bottleneck layer between the encoder and the decoder, and fuses the feature map (with a dimension of 512×8×8) output by the encoder with the encoding result of the action causal relationship. Extract the nodes and edges related to the current action from the action causal relationship graph, and encode them into a 128-dimensional vector through a graph convolutional network; map this vector to the same dimension as the number of channels of the feature map (512) through a fully connected layer; calculate the similarity between this vector and each spatial position of the feature map to generate an attention map; apply the attention map to the original feature map to obtain a weighted feature map, which is used as the input of the decoder.

[0061] The discriminator adopts the PatchGAN structure to discriminate the spatio-temporal features of the generated results. This discriminator not only judges the authenticity of a single image frame but also evaluates the temporal coherence. The input of the discriminator is a continuous sequence of 3 image frames, and spatio-temporal features are extracted through 3D convolution. The specific structure includes 5 3D convolutional blocks, and each convolutional block consists of a 3D convolutional layer, an instance normalization layer, and a LeakyReLU activation function. The convolutional layer uses a 4×4×4 convolutional kernel (time × height × width), with a stride of (1,2,2), and the number of channels is 64, 128, 256, 512, and 1 in sequence. The final output is a discriminant score map, indicating the authenticity scores of each part of the input sequence.

[0062] For the missing image frames in visual image information, the above generative adversarial network is used for completion. The missing frames in the video sequence are detected, and then 2 - 3 frames before and after the missing frame are extracted as context information and input into the generation network. For example, for a video with 30 frames per second, if the 15th frame is missing, the system will extract the 12th - 14th frames and the 16th - 18th frames as conditional inputs to generate the completion result of the 15th frame. During the completion process, three loss functions are combined for optimization: reconstruction loss, adversarial loss, and causal consistency constraint. The reconstruction loss measures the pixel-level difference between the generated image and the real image. The L1 loss is adopted, which calculates the sum of the absolute differences of the corresponding pixel values of the generated frame and the real frame, and then takes the average. The adversarial loss comes from the feedback of the discriminator and adopts the form of WGAN-GP (Wasserstein GAN with gradient penalty) to improve the authenticity and temporal coherence of the generated image. The causal consistency constraint calculates the deviation between the generated result and the predicted action sequence based on the action causal relationship graph. The generated image frame is input into a pre-trained action recognition model to obtain the action prediction result of this frame; at the same time, according to the action causal relationship graph and the known previous actions, the most likely action of the current frame is predicted; then the difference between these two results is calculated as the causal consistency constraint. In practice, the cross-entropy is used to calculate the difference between the two action probability distributions. The three loss functions are combined with different weights. In this embodiment, the weight of the reconstruction loss is 0.6, the weight of the adversarial loss is 0.2, and the weight of the causal consistency constraint is 0.2. These weights can be adjusted according to specific application requirements.

[0063] Perform temporal smoothness constraint on the completed visual image information to ensure that the generated image frames smoothly transition visually with the previous and subsequent frames, avoiding flickering or incoherence. Calculate the optical flow for three consecutive frames (the previous frame, the completed frame, and the subsequent frame), and then perform inter-frame interpolation smoothing based on the optical flow information. Adopt a bidirectional optical flow calculation method, with the forward optical flow from the previous frame to the completed frame and the backward optical flow from the subsequent frame to the completed frame, and then perform weighted averaging on the two according to the time position to obtain the final flow field. Perform local adjustment on the completed frame based on this flow field to ensure the naturalness of the action transition.

[0064] Re-perform feature fusion on the completed visual image information, human body skeleton information, and speech information to obtain the final action recognition result. When the confidence level of the recognition result is not lower than the preset confidence threshold, then use the action recognition result as the final action recognition result.

[0065] The present invention effectively solves the limitations of traditional methods in dealing with information loss problems in action recognition by introducing an action causal relationship graph to guide the visual image completion process; not only considers the visual quality of the image, but also emphasizes the logical coherence of the action, significantly improving the accuracy and robustness of action recognition; especially in complex environments and scenarios with rapid action changes, it can effectively recover the missing information, providing reliable technical support for fields such as intelligent monitoring, human-computer interaction, and virtual reality, and having broad application prospects.

[0066] In an optional implementation manner, constructing an action causal relationship graph based on hierarchical action decomposition and temporal dependence fusion includes: Obtain an action sequence based on the unified feature representation, decompose the action sequence into an atomic action set, identify the action combination patterns in the atomic action set through action combination rules, and generate a combined action set; calculate the action transition probabilities in the atomic action set and the combined action set to obtain the atomic action layer transition probability and the combined action layer transition probability, and calculate the mapping probability from the atomic action to the combined action and the decomposition probability from the combined action to the atomic action; Construct a dynamic time window, where the dynamic time window contains the adjacent actions before and after the target action, and calculate the local temporal dependence score based on the dynamic time window; count the action co-occurrence times in the action sequence to construct an action co-occurrence matrix, and calculate the global temporal dependence score; perform weighted fusion on the local temporal dependence score and the global temporal dependence score to obtain a fusion dependence score, and update the atomic action layer transition probability, the combined action layer transition probability, the mapping probability, and the decomposition probability based on the fusion dependence score; Taking the actions in the atomic action set and the combined action set as nodes, and based on the action recognition results, online adjust the updated transition probabilities of the atomic action layer, the transition probabilities of the combined action layer, the mapping probabilities, and the decomposition probabilities, and perform normalization post-processing as the edge weights to construct an action causal relationship graph with a hierarchical structure.

[0067] Exemplarily, an action sequence is obtained based on unified feature representation. The continuous video is segmented by a sliding window method. The window size is 2 seconds and the step size is 0.5 seconds. The action category of the video segment within each window is recognized by a pre-trained action recognition model. For example, for a 30-second video, the system extracts a total of 57 windows, and the action sequence obtained is like "standing - bending - picking up an object - standing upright - walking - putting down an object", etc.

[0068] Decompose the action sequence into an atomic action set. An atomic action refers to the basic action unit that cannot be further decomposed and is the basis for constructing complex actions. The system has predefined 50 types of atomic actions, including "bending the elbow", "raising the arm", "leaning forward the torso", "rotating the head", etc. In the decomposition process, a pre-trained pose estimation model is used to extract key points, and then the atomic actions are determined through rule matching and template comparison. For example, the action of "picking up an object" may be decomposed into atomic actions such as "extending the arm", "clenching the fingers", "bending the elbow", "retracting the arm", etc. Identify the action combination patterns in the atomic action set through action combination rules, and generate a combined action set. The action combination rules are designed based on the temporal pattern mining algorithm to identify frequently occurring atomic action sequences, set the minimum support as 0.05, the minimum confidence as 0.6, and apply the improved PrefixSpan algorithm to mine frequent sequence patterns. The mined patterns form combined actions after semantic filtering. For example, the system may find that the combination of "extending the arm", "clenching the fingers", "bending the elbow" appears frequently and define it as the combined action of "grasping". In this embodiment, the system has identified 120 common combined actions.

[0069] Calculate the action transition probabilities in the atomic action set and the combined action set. For the atomic action set, count the number of transitions between adjacent atomic actions and divide it by the total number of occurrences of the previous action to obtain the transition probability at the atomic action layer. For example, if "extend arm" is followed by "clench fingers" 200 times, and "extend arm" appears a total of 250 times, then the transition probability is 0.8. Similarly, calculate the transition probability at the combined action layer. For the mapping probability from atomic actions to combined actions, count the number of times a certain atomic action belongs to a specific combined action and divide it by the total number of occurrences of that atomic action. For example, if "clench fingers" appears 350 times in "grasp", and the total number of occurrences is 400 times, then the mapping probability is 0.875. The decomposition probability from combined actions to atomic actions is the probability that a certain combined action contains a specific atomic action, calculated by dividing the number of times the combined action contains that atomic action by the total number of combined actions.

[0070] The dynamic time window size is adaptively adjusted according to the action duration and usually includes the previous 2 and the next 2 actions of the target action. Within the window, the system considers the time interval and sequential relationship between actions and calculates the local temporal dependency score. The time interval is processed by an exponential decay function, and the shorter the interval, the higher the weight; the sequential relationship is assigned different weights according to the relative position, with a weight of 1 for directly adjacent actions, 0.7 for actions separated by one action, and 0.4 for actions separated by two actions. The local temporal dependency score is the product of the time interval weight and the sequential relationship weight. For example, if actions A and B are closely connected (time interval of 0.2 seconds), then the time interval weight is 0.98; if A is directly followed by B, the sequential relationship weight is 1, and the final local dependency score is 0.98. At the same time, count the co-occurrence times of actions in the action sequence to construct an action co-occurrence matrix and calculate the global temporal dependency score. Action co-occurrence means that two actions appear in the same sequence regardless of their specific positions. Count the co-occurrence times of each pair of actions in the training data to construct an N×N co-occurrence matrix (N is the total number of actions). The global temporal dependency score is calculated by point mutual information to measure the non-randomness degree of the co-occurrence of two actions. Specifically, calculate the logarithm of the ratio of the number of times two actions co-occur to the product of their respective independent occurrence probabilities. The higher this value, the stronger the dependency. For example, if the actions "pick up object" and "put down object" co-occur 850 times in 1000 sequences, and their respective occurrence probabilities are 0.3 and 0.25, then their global dependency score is relatively high, about 1.55.

[0071] The local temporal dependence score and the global temporal dependence score are weighted and fused to obtain the fused dependence score. The fusion adopts a linear weighting method, and the weights are set according to the application scenario. In this embodiment, the weight of the local dependence score is 0.7, and the weight of the global dependence score is 0.3. For example, for the action pair "pick up an object - put down an object", if the local dependence score is 0.85 and the global dependence score is 1.55, the fused dependence score is 0.85×0.7 + 1.55×0.3 = 1.06.

[0072] Based on the fused dependence score, the transition probabilities of the atomic action layer, the combined action layer, the mapping probability, and the decomposition probability are updated. The update process uses a weighted average method, with the original probability weight being 0.6 and the dependence score influence weight being 0.4. The dependence score is mapped to the 0 - 1 interval through the sigmoid function and then weighted averaged with the original probability. For example, if the original transition probability from "extend the arm" to "clench the fingers" is 0.8, and the corresponding fused dependence score is 1.06, which is mapped to 0.74 after mapping, the updated transition probability is 0.8×0.6 + 0.74×0.4 = 0.776.

[0073] Taking the actions in the atomic action set and the combined action set as nodes, an action causal relationship graph with a hierarchical structure is constructed. The edges in the graph are determined based on the updated various probabilities, including the edges within the atomic action layer (transitions between atomic actions), the edges within the combined action layer (transitions between combined actions), and the edges between layers (mapping edges from atomic actions to combined actions and decomposition edges from combined actions to atomic actions). Based on the action recognition results, the weights of these edges are adjusted online, and the adjustment amplitude is proportional to the recognition confidence. For example, when the system recognizes the action "pick up an object" with a confidence of 0.9, it will enhance the weights of the edges related to this action by a relatively large amplitude (such as 0.1); while when the confidence is low (such as 0.6), the adjustment amplitude is small (such as 0.03). The adjusted weights are normalized to ensure that the sum of the weights of all edges starting from the same node is 1.

[0074] In practical applications, the graph is stored and index - optimized. The adjacency list structure is used for storage, and the action nodes are organized hierarchically. Each node records its out - edge target nodes and weights. To improve the retrieval efficiency, the system builds a hash mapping table from action names to node IDs. In addition, the graph is updated every 1000 action sequences are processed to ensure that the graph can adapt to new action patterns.

[0075] Figure 3This is a performance comparison graph of action prediction accuracy, showing the comparison results of three different methods in action scenarios of various complexities. The horizontal axis represents the complexity of the action scenario, which from left to right are "single-person simple actions", "single-person complex actions", "two-person interaction actions", "multi-person complex interactions", and "cross-scenario multi-person interactions", with increasing complexity; the vertical axis represents the action prediction accuracy percentage, ranging from 50% to 100%. Three different action prediction methods are compared in the graph: The present invention (hierarchical action causal graph) adopts a two-layer structure of an atomic action layer and a combined action layer, fusing local temporal dependencies and global temporal dependencies; the single-layer action transfer model only uses a single layer to model the action transfer relationship; the traditional Markov model models based on the state transition probability of a fixed time step. It can be clearly seen from the graph that as the complexity of the action scenario increases, the prediction accuracy of all three methods shows a downward trend, but the decline of the method of the present invention is the smallest. In the single-person simple action scenario, the differences among the three methods are small (95.2% for the present invention, 92.7% for the single-layer model, and 90.8% for the Markov model); while in high-complexity scenarios such as cross-scenario multi-person interactions, the accuracy gap between the method of the present invention (79.4%) and the other two methods (63.1% for the single-layer model and 60.7% for the Markov model) expands significantly. This performance difference is mainly attributed to the fact that the hierarchical structure of the present invention can more effectively capture the long-distance dependencies in complex action sequences, and the strategy of fusing local and global temporal dependencies can provide more comprehensive action context information.

[0076] The present invention constructs an action causal relationship graph through hierarchical action decomposition and temporal dependency fusion, effectively solving the problem that traditional methods are difficult to capture both the fine-grained features and high-level semantics of actions at the same time; not only considering the direct transfer relationship between actions, but also fusing local and global temporal dependency information, significantly improving the accuracy of action prediction and understanding; the hierarchical structure design enables the graph to express both the details of basic action units and the overall semantics of combined actions, providing strong knowledge support for tasks such as video completion and action prediction, and having broad application prospects.

[0077] In the second aspect of the embodiments of the present invention, a spatial interaction precise recognition system based on multimodal fusion is provided, including: A first unit for collecting human body bone information, visual image information, and voice information; A second unit is configured to input the human body bone information and visual image information into a feature extraction network. The feature extraction network generates a spatio-temporal attention map based on the joint point spatio-temporal features in the human body bone information, applies the spatio-temporal attention map to the visual image information to obtain enhanced visual features, performs feature complementary correction on the human body bone information and the enhanced visual features, and fuses the corrected human body bone features and visual features with speech information features based on temporal modeling, feature alignment, and cross-modal attention mechanism to obtain a unified feature representation. A third unit is configured to perform action recognition on the unified feature representation. When the confidence level of the recognition result is lower than a preset confidence threshold, for the missing image frames in the visual image information, a spatio-temporal convolutional generative adversarial network guided by an action causal relationship graph is used for completion to obtain the completed visual image information, and the completed visual image information is re-fused with the human body bone information and the speech information to obtain a final action recognition result.

[0078] In a third aspect of the embodiments of the present invention, there is provided an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the method described above.

[0079] In a fourth aspect of the embodiments of the present invention, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0080] The present invention may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions for executing various aspects of the present invention are uploaded.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spatial interaction precise recognition method based on multimodal fusion, characterized in that, Including: Collecting human body bone information, visual image information, and voice information; Inputting the human body bone information and visual image information into a feature extraction network. The feature extraction network generates a spatio-temporal attention map based on the spatio-temporal features of the joint points in the human body bone information, and applies the spatio-temporal attention map to the visual image information to obtain enhanced visual features; Performing feature complementary correction on the human body bone information and the enhanced visual features; based on temporal modeling, feature alignment, and cross-modal attention mechanism, fusing the corrected human body bone features and visual features with the voice information features to obtain a unified feature representation; Performing action recognition on the unified feature representation. When the confidence level of the recognition result is lower than the preset confidence threshold, for the missing image frames in the visual image information, complete them based on the spatio-temporal convolutional generative adversarial network guided by the action causal relationship graph to obtain the completed visual image information; Re-performing feature fusion on the completed visual image information, human body bone information, and voice information to obtain the final action recognition result.

2. The method according to claim 1, characterized in that, The feature extraction network generates a spatio-temporal attention map based on the spatio-temporal features of the joint points in the human body bone information, and applying the spatio-temporal attention map to the visual image information to obtain enhanced visual features includes: Obtaining a bone sequence containing T frames, each frame of the bone sequence containing N joint points, and each joint point including two-dimensional coordinates and confidence information, where the two-dimensional coordinates are used to represent the position of the joint point in the image; calculating the displacement vector of the joint point based on the two-dimensional coordinates of adjacent frames; Constructing the bone sequence into a graph structure, the graph structure including a vertex set and an edge set, the vertex set corresponding to the joint points in the bone sequence, and the edge set corresponding to the bone connections in the bone sequence; inputting the graph structure into a graph convolutional network, enhancing the features of the joint points based on the set of neighbor nodes of the joint points to obtain enhanced joint point features; performing multi-scale spatio-temporal feature extraction on the enhanced joint point features, generating spatio-temporal position encoding and fusing it with the spatio-temporal features to obtain encoded features, and calculating spatio-temporal self-attention features; performing weight fusion and constraint optimization on the spatio-temporal self-attention features to obtain a spatio-temporal attention map; Calculating the motion amplitude value in the spatio-temporal attention map based on the displacement vector, where the motion amplitude value is determined by the Euclidean norm of the displacement vector at the corresponding position; normalizing the motion amplitude value to obtain an enhancement coefficient, and the product of the enhancement coefficient and the spatio-temporal attention map acts on the visual image information to enhance the feature expression of the target action area to obtain enhanced visual features.

3. The method according to claim 2, wherein Generating a spatio-temporal attention map includes: Performing multi-scale downsampling on the enhanced joint point features to obtain multi-scale spatial feature maps, and performing temporal convolution operations on the multi-scale spatial feature maps within a preset time window to obtain temporal dynamic features; Generating spatio-temporal position encoding, fusing the spatio-temporal position encoding with the temporal dynamic features to obtain encoded features; calculating spatio-temporal self-attention features based on the encoded features, where the spatio-temporal self-attention features are used to represent the spatio-temporal dependence relationship between joint points; Global average pooling is performed on the spatio-temporal self-attention features to obtain feature importance weights, and the feature importance weights are applied to the multi-scale spatial feature map and the temporal dynamic features to obtain an initial attention map; The initial attention map is optimized based on temporal smoothness constraints, spatial structure constraints, and sparsity constraints, and boundary enhancement is performed to obtain a spatio-temporal attention map.

4. The method according to claim 1, wherein Performing feature complementary correction on the human body bone information and the enhanced visual features includes: Based on the human body bone information and the enhanced visual features, the reliability of the bone features and the reliability of the visual features are calculated; the reliability of the bone features is obtained by calculating the geometric consistency, motion smoothness, and structural integrity of the joint points, and the reliability of the visual features is obtained by calculating the temporal consistency and the scene context; When it is detected that the reliability of the bone features is lower than the first threshold, correction is performed based on the features at the corresponding positions in the visual image information; when it is detected that the reliability of the visual features is lower than the second threshold, correction is performed based on the joint positions in the human body bone information; when the reliability of the bone features is not lower than the first preset threshold and the reliability of the visual features is not lower than the second preset threshold, the human body bone information and the enhanced visual features are used as the corrected bone features and visual features.

5. The method according to claim 4, characterized in that Performing feature complementary correction includes: Generating a complementary correction weight matrix based on the reliability of the bone features and the reliability of the visual features; performing bidirectional correction on the bone features and the enhanced visual features by using the complementary correction weight matrix, including: weighting the appearance information of the enhanced visual features with the first correction weight to obtain a spatial correction map, and fusing the spatial correction map with the bone features to obtain the corrected bone features; weighting the motion trajectory information of the bone features with the second correction weight to obtain a motion correction map, and fusing the motion correction map with the enhanced visual features to obtain the corrected visual features; Performing temporal consistency and cross-modal consistency verification on the corrected bone features and visual features, generating a residual correction term, and fusing the residual correction term with the corresponding features to obtain the final corrected bone features and visual features.

6. The method according to claim 1, characterized in that Performing action recognition on the unified feature representation. When the confidence level of the recognition result is lower than the preset confidence threshold, for the missing image frames in the visual image information, based on the spatio-temporal convolutional generative adversarial network guided by the action causal relationship graph, completion is performed to obtain the completed visual image information, including: Performing action recognition on the unified feature representation, extracting spatio-temporal features through a three-dimensional convolutional network and performing feature fusion by combining a temporal attention mechanism to obtain the action recognition result and its confidence level; when the confidence level of the action recognition result is lower than the preset confidence threshold, constructing an action causal relationship graph based on hierarchical action decomposition and temporal dependence fusion; Constructing a causal attention mechanism based on the action causal relationship graph, integrating the causal attention mechanism into the generative adversarial network, where the generative adversarial network includes a generator and a discriminator, the generator adopts an encoder-decoder structure and introduces the causal attention mechanism, and the discriminator performs spatio-temporal feature discrimination on the generation result; For the missing image frames in the visual image information, the generative adversarial network is used for completion. During the completion process, it is optimized by combining the reconstruction loss, adversarial loss, and causal consistency constraint. The causal consistency constraint is obtained by calculating the deviation between the generated result and the predicted action sequence based on the action causal relationship graph. The temporally smoothed constraint is applied to the completed visual image information to generate the final completion result.

7. The method according to claim 6, wherein Constructing the action causal relationship graph based on hierarchical action decomposition and temporal dependence fusion includes: Based on the unified feature representation, an action sequence is obtained. The action sequence is decomposed into an atomic action set. Action combination patterns in the atomic action set are identified through action combination rules to generate a combined action set. The action transition probabilities in the atomic action set and the combined action set are calculated to obtain the atomic action layer transition probability and the combined action layer transition probability, and the mapping probability from atomic actions to combined actions and the decomposition probability from combined actions to atomic actions are calculated. A dynamic time window is constructed. The dynamic time window contains adjacent actions before and after the target action. The local temporal dependence score is calculated based on the dynamic time window. The action co-occurrence matrix is constructed by counting the action co-occurrence times in the action sequence, and the global temporal dependence score is calculated. The local temporal dependence score and the global temporal dependence score are weighted and fused to obtain the fused dependence score. Based on the fused dependence score, the atomic action layer transition probability, the combined action layer transition probability, the mapping probability, and the decomposition probability are updated. Taking the actions in the atomic action set and the combined action set as nodes, based on the action recognition result, the updated atomic action layer transition probability, the combined action layer transition probability, the mapping probability, and the decomposition probability are adjusted online and post-processed by normalization as the weights of the edges to construct an action causal relationship graph with a hierarchical structure.

8. A spatial interaction precise recognition system based on multimodal fusion, which is used to implement the method described in any one of the foregoing claims 1-7, characterized in that, It includes: The first unit is used to collect human body skeleton information, visual image information, and voice information. The second unit is used to input the human body skeleton information and visual image information into a feature extraction network. The feature extraction network generates a spatio-temporal attention map based on the joint spatio-temporal features in the human body skeleton information, and applies the spatio-temporal attention map to the visual image information to obtain enhanced visual features. Performing feature complementary correction on the human body skeleton information and the enhanced visual features; based on temporal modeling, feature alignment, and cross-modal attention mechanism, fusing the corrected human body skeleton features and visual features with the voice information features to obtain a unified feature representation. The third unit is used to perform action recognition on the unified feature representation. When the confidence level of the recognition result is lower than the preset confidence threshold, for the missing image frames in the visual image information, they are completed based on the spatio-temporal convolutional generative adversarial network guided by the action causal relationship graph to obtain the completed visual image information. The completed visual image information is re-fused with the human body skeleton information and voice information to obtain the final action recognition result.

9. An electronic device, characterized in that, It includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images