Multimodal assembly action recognition method for comparing semantic query

Abstract

The invention discloses a multi-modal assembly action recognition method for comparative semantic query, and relates to the technical field of man-machine cooperation assemblation.The method comprises the steps that a visual sensor is arranged on an assembly workbench to obtain an operator action video, a sampling frame sequence is obtained through random frame sampling, a skeleton sequence is obtained through human body posture estimation, and a skeleton sequence is obtained through human body posture estimation; and inputting an assembly action recognition model to complete recognition. The model comprises an image coding module, a skeleton coding module, a feature fusion module, a text coding module and a semantic comparison module which are used for extracting image and skeleton features, fusing features, coding preset category text description, comparing action features with category text features and outputting a result with the highest similarity, and a comparison loss function is adopted during training. According to the method, multi-modal information is fused, the problems of single-modal limitation and multi-modal semantic segmentation are solved, category text semantics are fully utilized, the fine-grained action recognition precision is improved, the over-fitting risk is reduced, and the generalization and task migration ability of the model in a dynamic industrial scene is enhanced.

Description

Technical Field

[0001] This application relates to the field of human-machine collaborative assembly technology, and in particular to a multimodal assembly action recognition method based on comparative semantic query. Background Technology

[0002] Against the backdrop of the manufacturing industry shifting from the traditional mass production and standardized production model to multi-variety, small-batch, and personalized customization, traditional fully automated assembly lines, which rely on preset processes and fixed structures, are difficult to adapt to flexible and ever-changing assembly needs, and their applicability and scalability are limited.

[0003] To enhance the flexibility and intelligence of manufacturing systems, Human-Robot Collaborative Assembly (HRCA) systems have emerged. HRCA systems combine the flexibility of human operation with the stability of machine control by introducing robots into the human workspace, thereby improving assembly efficiency and system adaptability to a certain extent. As collaborative models evolve from static division of labor to dynamic shared spaces, collaborative robots are required to possess stronger environmental perception and behavioral understanding capabilities, enabling them to recognize and respond to operator actions and intentions in real time in unstructured scenarios, thereby achieving proactive collaborative functions such as grasping, assistance, and error correction.

[0004] In existing technologies, motion recognition for HRCA mainly employs three types of perception methods: First, methods based on biosignals such as inertia or electromyography, while offering rapid response, are limited by high wearable costs and significant individual differences, making widespread deployment in complex industrial environments difficult. Second, visual recognition methods based on RGB images, while possessing rich spatial information, lack robustness under the influence of lighting changes and occlusion interference. Third, methods based on human skeletal sequences, with three-dimensional joint point trajectories as the core, possess advantages such as structured representation, lightweight design, and strong robustness to background interference, and have become the mainstream solution in current industrial motion recognition. All these methods, in the final step, feed the extracted features into a unified classifier, using a softmax function to generate a probability distribution. The N categories are represented by numbers from 0 to N-1, and the category number corresponding to the highest probability is the output recognition result.

[0005] Existing technologies still have the following shortcomings in practical HRCA scenarios: 1. Limited data representation capability of a single modality: skeletal information can only describe the geometric structure and movement trajectory of the human body itself, and cannot capture the semantic interaction between the person and the tool or assembly object, leading to ambiguity in fine-grained action recognition with high similarity. 2. Imperfect multimodal fusion mechanism: current multimodal recognition methods mostly adopt simple feature splicing or early fusion strategies, without establishing semantic alignment relationships between modalities. 3. The textual descriptions of category labels are not fully utilized, and recognition and judgment still rely on fixed-label classifiers.

[0006] Therefore, existing technologies lack an efficient action recognition method that can integrate skeletal structure, visual information and language semantics, especially a fusion framework with cross-modal semantic alignment capabilities that is suitable for fine-grained assembly action recognition. A new solution is urgently needed to improve the intelligent collaboration level and practical deployment capability of HRCA systems. Summary of the Invention

[0007] The purpose of this application is to provide a multimodal assembly action recognition method based on comparative semantic queries, which can solve the problems of single-modality limitations and multimodal semantic fragmentation, has cross-modal semantic alignment capability, and is suitable for fine-grained assembly action recognition.

[0008] To achieve the above objectives, this application provides the following solution: A multimodal assembly action recognition method based on comparative semantic queries includes the following steps: Visual sensors are installed above the assembly workbench to acquire video data of human operators performing actions.

[0009] Random frame sampling is performed on the video data to obtain the sampled frame sequence data.

[0010] Human pose estimation is performed on video data to obtain human skeletal sequence data.

[0011] The sampled frame sequence data and human skeleton sequence data are input into the trained assembly action recognition model for feature extraction and semantic comparison query to obtain the assembly action recognition result.

[0012] The assembly action recognition model includes an image encoding module, a skeleton encoding module, a feature fusion module, a text encoding module, and a semantic comparison module. The image encoding module extracts features from the sampled frame sequence data to obtain image features. The skeleton encoding module extracts features from the human skeleton sequence data to obtain skeleton features. The feature fusion module fuses image features and skeleton features into action features. The text encoding module encodes the pre-defined category text descriptions for each assembly action to obtain several category text features. The semantic comparison module performs semantic comparisons between the action features and each category text feature, and outputs the category text description with the highest similarity as the assembly action recognition result. When training the assembly action recognition model, a contrastive loss function is used to train the model.

[0013] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a multimodal assembly action recognition method based on comparative semantic queries. The method includes: deploying a vision sensor above an assembly workbench to synchronously collect visual information, providing a foundation for subsequent multimodal fusion. Compared to a single skeletal modality, this method can supplement details of the working environment and the interaction between people, tools, and assembly objects. Random frame sampling is performed on video data to obtain sampled frame sequence data, and human pose estimation is performed on the video data to obtain human skeletal sequence data. Random frame sampling can reduce data redundancy while retaining key action information, while human pose estimation accurately extracts skeletal motion trajectories. The combination of these two forms a complementary multimodal data structure—the skeletal sequence captures the dynamic trajectory of the human body, and the sampled frame sequence records environmental and interactive object features, jointly enhancing the distinguishability of highly similar actions and reducing ambiguity in fine-grained recognition. Subsequently, the sampled frame sequence data and human skeletal sequence data are input into a trained assembly action recognition model for feature extraction and semantic comparison queries to obtain the assembly action recognition result. The assembly action recognition model used in this step includes an image encoding module, a skeleton encoding module, a feature fusion module, a text encoding module, and a semantic comparison module. Image features and skeleton features are extracted separately, fused to generate action features, and then cross-modal semantic matching is used to directly associate action features with text semantics, significantly improving the recognition accuracy of fine-grained actions. When training the assembly action recognition model, a contrastive loss function is used to train the model. By narrowing the distance between matching pairs and widening the distance between non-matching pairs, the model is prompted to learn semantically consistent cross-modal representations, effectively reducing the risk of overfitting and improving the model's generalization ability and task transfer ability in dynamic industrial scenarios. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a multimodal assembly action recognition method based on comparative semantic query, provided in one embodiment of this application.

[0016] Figure 2 This is a schematic diagram illustrating the technical framework of an assembly action recognition model applied in a multimodal assembly action recognition method based on comparative semantic query, provided in an embodiment of this application.

[0017] Figure 3 This is a flowchart illustrating the construction and training of an assembly action recognition network in a multimodal assembly action recognition method based on comparative semantic queries, as provided in an embodiment of this application.

[0018] Figure 4 This is a flowchart of step B1 in a multimodal assembly action recognition method based on comparative semantic query, provided in an embodiment of this application.

[0019] Figure 5 This is a schematic diagram of the technical framework for training an assembly action recognition model in a multimodal assembly action recognition method based on comparative semantic queries, provided in an embodiment of this application.

[0020] Figure 6 This is a schematic diagram of the ST-Block structure in a multimodal assembly action recognition method for comparative semantic query provided in an embodiment of this application.

[0021] Figure 7 This is a schematic diagram of the ATT-Block structure in a multimodal assembly action recognition method for comparative semantic query provided in an embodiment of this application.

[0022] Figure 8 This is a schematic diagram of the feature fusion module in a multimodal assembly action recognition method based on comparative semantic query, provided in an embodiment of this application.

[0023] Figure 9 This is a schematic diagram of the adapter structure in a multimodal assembly action recognition method for comparative semantic query provided in an embodiment of this application.

[0024] Figure 10 This is a schematic diagram of a working scenario for a human-machine collaborative speed reducer assembly example provided in one embodiment of this application.

[0025] Figure 11This is a schematic diagram of the skeleton model used in a multimodal assembly action recognition method based on comparative semantic query provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] This application provides a multimodal assembly action recognition method based on comparative semantic queries. In one exemplary embodiment, such as... Figure 1 As shown, it includes the following steps: A1. Install vision sensors above the assembly workbench to acquire video data of human operators performing actions.

[0029] A2. Perform random frame sampling on the video data to obtain a sampled frame sequence data. Specifically, divide the video data into several segments of equal length; randomly select one frame from each segment to form the sampled frame sequence data.

[0030] A3. Perform human pose estimation on the video data to obtain human skeleton sequence data.

[0031] A4. Input the sampled frame sequence data and human skeleton sequence data into the trained assembly action recognition model for feature extraction and semantic comparison query to obtain the assembly action recognition result.

[0032] In the method provided in this embodiment, the assembly action recognition model includes an image encoding module, a skeleton encoding module, a feature fusion module, a text encoding module, and a semantic comparison module. For example... Figure 2The diagram illustrates the technical framework for model application. The model input consists of a sequence of sampled frames and a skeleton sequence based on RGB image sampling. An image encoding module extracts features from the sampled frame sequence data to obtain image features. A skeleton encoding module extracts features from the human skeleton sequence data to obtain skeleton features. A feature fusion module fuses the image and skeleton features into action features. A text encoding module encodes the pre-defined category text descriptions for each assembly action, resulting in several category text features. A semantic comparison module performs semantic comparisons between the action features and each category text feature, outputting the category text description with the highest similarity as the assembly action recognition result. The assembly action recognition model is trained using a contrastive loss function. After obtaining the assembly action recognition results, the human operator's assembly intention can be determined, allowing the collaborative robot to perform corresponding cooperative operations.

[0033] In another exemplary embodiment of this application, the method further includes a process of constructing and training an assembly action recognition network, such as... Figure 3 As shown, the process includes the following steps: B1. For various assembly actions of human operators during the assembly process, video data is collected and processed to construct a training sample set and a validation sample set. The training and validation sample sets each include several action samples; each action sample includes sampled frame sequence data, human skeleton sequence data, and a category text description. In this embodiment, as... Figure 4 As shown, step B1 specifically includes the following steps: B11. Collect several video data points of N human operators performing K types of assembly actions during the assembly process; each human operator repeats each type of assembly action M times, resulting in a total of N×M×K video data points.

[0034] B12. For any video data, perform human pose estimation on the video data to obtain human skeleton sequence data.

[0035] B13. Randomly sample the video data to obtain a sampled frame sequence data. In this embodiment, step B13 specifically involves: dividing the video data into several segments of equal length; randomly selecting one frame from each segment to form the sampled frame sequence data.

[0036] B14. Generate category text descriptions for the video data based on the assembly actions and category text description templates corresponding to the video data.

[0037] B15. Based on the human skeleton sequence data, sampled frame sequence data and category text description of the video data, construct the motion samples corresponding to the video data.

[0038] B16. Divide several action samples into a training sample set and a validation sample set according to a preset ratio.

[0039] B2. Construct an assembly action recognition network, and train and validate the network based on the training and validation sample sets to obtain a trained assembly action recognition model. The technical framework diagram for training the assembly action recognition network described above is shown below. Figure 5 As shown.

[0040] Figure 5 The left side illustrates the assembly work scenario and data collection. During the assembly process, a RealSense camera continuously monitors the work site, collecting RGB video data. Corresponding skeletal keypoint sequences are extracted through human pose estimation. The category labels for assembly actions are constructed into semantically complete category text descriptions using a category text description template prompt. This creates a multimodal data input that integrates spatial and motion information. Figure 5 The upper right section illustrates the feature extraction and fusion process for the skeletal sequence and sampled frame sequences, including the encoding of the skeletal sequence and the sampled frame sequences. The skeletal sequence extracts action features through a spatiotemporal graph convolutional network, while the sampled frame sequences obtain scene information through an R(2+1)D ResNet module. The two types of features are subsequently fused to characterize the temporal dynamics and semantic scene of the action samples. Figure 5 The lower right half shows the feature extraction process for the category description text. The category description text is passed through a pre-trained text encoding module and an adapter module to generate an embedding vector of the category text description, which serves as the text feature.

[0041] During the training phase, each batch contains B pairs of action features and their corresponding text features. A contrastive learning framework is used to narrow the distance between matching pairs while distancing mismatched samples, achieving cross-modal representation alignment. During the inference phase, all candidate category text descriptions are first processed by a text encoding module to generate their respective category text features. For a new input action sample, action features are generated via an image encoding module, a skeleton encoding module, and a feature fusion module. These features are then matched with the text features of each category, and the category text description with the highest similarity is retrieved using a nearest neighbor search method as the assembled action recognition result.

[0042] The skeleton encoding module uses a Spatial-Temporal Graph Convolutional Network (ST-GCN) as its backbone network. It performs spatiotemporal neighborhood feature aggregation on each node through multi-layer graph convolution to extract spatiotemporal context-aware representation features. At the end of the skeleton encoding module, a lightweight attention block ATT-Block is integrated. The attention block includes a joint temporal attention unit and a channel attention unit.

[0043] In ST-GCN, the human skeletal sequence is modeled as a spatiotemporal graph. ,in V Represents the set of all key points. E S Represents spatial edges (topological connections between joints). E T This represents a temporal edge (a connection between adjacent time frames of the same joint). Each node... Indicates the first t The first frame i Each joint. The input feature can be represented as a tensor. , where C represents the number of input channels (e.g., coordinate dimension). T Indicates the number of time frames. V This indicates the number of key points in each frame.

[0044] ST-GCN performs graph convolution operations in both spatial and temporal directions, such as Figure 6 As shown, in each ST-Block, the edge first passes through an ATT (Adjacency Tolerance) layer, which contains a learnable adjacency weight mask to adjust the importance of the edges; then, it passes through a spatial graph convolutional network (GCN) to extract and aggregate spatial features, as mathematically described in Equation (1): (1) in, W d For spatial convolution parameters, M d This is the adjacency weight mask, and ⊙ represents element-wise multiplication. d It is the graphical distance between key points. D It is the maximum distance, where It is an adjacency matrix. Let be the degree matrix of the nodes. Then, after passing through TCN, one-dimensional convolution is performed on the features of each node in the temporal dimension to model dynamic changes. ST-GCN aggregates features from the spatiotemporal neighborhood of each node through multi-layer graph convolution, extracting spatiotemporal context-aware representation features. Spatial convolution captures spatial dependencies between joints, while temporal convolution models the dynamic evolution features of each node in the sequence, thus achieving the modeling of complex action patterns. The entire ST-Block consists of nine stacked ST-GCN networks, capable of extracting high-order semantic features of spatial structure and temporal dynamics layer by layer.

[0045] In the process of human action recognition and understanding, the importance of different key points varies significantly at different times. To more accurately focus on time frames (temporal dimension), key points (spatial dimension), and semantic channels (channel dimension) that are highly relevant to the task, while reducing redundant information interference and improving the abstraction ability of high-order semantic features, this embodiment introduces a globally fusion-based attention module. Considering the model's adaptability to small-scale datasets, this attention module is designed to be lightweight and integrated only at the end of the skeletal feature extractor, in order to improve the effectiveness of feature representation while controlling the number of parameters. Figure 7 As shown, the attention module includes the Joint-temporal attention module (JTAM) and the Channel attention module (CAM).

[0046] Specifically, regarding the spatiotemporal characteristics of ST-GCN output The JTAM part is used to capture the importance of which joints at which times. Average pooling and max pooling are performed in the channel direction to extract the overall distribution features and the strongest local response features in the joint and time dimensions, respectively. These features are concatenated to enhance the expressive power, and then the spatiotemporal attention map is obtained through convolution, as shown in equations (2) to (4): , (2) (3) (4) The CAM part is used to capture different channels and assign them equal importance to the final prediction. Average pooling and max pooling are performed in the joint and time dimensions, and then channel attention is calculated using MLP mapping. The above process is shown in equations (5) to (7): , (5) (6) (7) Spatiotemporal characteristics of ST-GCN output Passing through JTAM and CAM in sequence, , This order can achieve better results. Finally, spatiotemporal global average pooling is used to obtain the final features, which are then mapped to the specified dimension through a linear layer. The final output of the skeleton encoding module is... d is the feature dimension (e.g., 512).

[0047] The image encoding module uses R(2+1)D ResNet. Operationally, R(2+1)D ResNet decomposes the 3D convolution operation into two stages: one-dimensional temporal convolution and two-dimensional spatial convolution. Internally, it uses a residual connection mechanism to build a deep structure. Structurally, R(2+1)D ResNet includes an initial 3D convolutional layer, multiple sets of R(2+1)D residual layers, and a global average pooling layer. Each set of (2+1)D residual layers contains multiple R(2+1)D residual blocks, and each R(2+1)D residual block contains multiple R(2+1)D convolution operations.

[0048] For input tensor Where C is the number of input channels (e.g., RGB channels), G is the number of time frames (e.g., 16 frames), and H×W is the spatial size of the frame (e.g., 480×640), the R(2+1)D ResNet network decomposes the 3D convolution operation into a 2D spatial convolution and a 1D temporal convolution. In each R(2+1) convolution operation, the spatial convolution kernel is... ,in, It is the number of intermediate channels. The spatial convolution kernel has a time dimension of 1, processes spatial features only within a single frame, does not span frames, and its output is... (The spatial dimensions vary due to stride or padding). The temporal convolution kernel is... ,in t It is the temporal receptive field (e.g., 3 frames). It is the number of output channels, only aggregating timing information, without changing the spatial resolution, and the output is... (Time dimension varies depending on step size or fill).

[0049] To improve training stability and the network's deep representation capabilities, the R(2+1)D network employs a residual connection mechanism to construct a deep structure. Specifically, Figure 1The R(2+1)D ResNet module's network structure consists of an initial 3D convolutional layer, multiple R(2+1)D residual layers, and a global average pooling layer. Each R(2+1)D residual layer contains multiple R(2+1)D residual blocks, and each residual block contains multiple R(2+1)D convolutional operations. The first block of each residual layer performs spatial downsampling through a convolution with a spatial stride of 2, halving the height and width of the feature map. Finally, a global average pooling layer aggregates the temporal and spatial dimensions, and the output of the RGB feature extraction is... (d is the feature dimension).

[0050] The feature fusion module employs low-rank bilinear attention pooling (LBAP) to fuse skeletal and image features, and generates modal attention based on the fused representation to guide the enhancement and weighted fusion of the original features. This embodiment introduces LBAP to fuse skeletal and RGB features, and generates modal attention based on the fused representation to guide the enhancement and weighted fusion of the original features, effectively leveraging the complementary information in the spatiotemporal structure of the image and skeletal sequences. Figure 8 As shown, given RGB features and skeletal sequence features First, cross-modal interaction is performed using low-rank bilinear pooling as shown in equation (8): (8) in, and Let be the projection matrix of the modes, and ⊗ denote the Hadamard product. Given a low-rank projection matrix (k < 512), output fused features. The fused feature is then used to generate an attention weight vector through a fully connected layer and a one-dimensional convolution, as shown in Equation (9): (9) Obtained using two channels respectively and After cross-multiplying each feature with the original feature, the final feature representation is obtained by weighted summation of features according to equation (10): (10) In this embodiment, by introducing a bilinear attention pooling mechanism, the model can explicitly model the high-order interaction relationship between the image and the skeleton, improve the consistency and discriminativeness of cross-modal features, and thus effectively improve the performance of multimodal action recognition.

[0051] Based on the assembly actions and category text description templates corresponding to the video data, when generating category text descriptions for the video data, various semantically equivalent but differently expressed sentence structures are used to construct natural language descriptions with clear semantics as category text descriptions for assembly actions.

[0052] In multimodal tasks, compared to the traditional method of using semantically meaningless discrete category numbers as labels, this invention fully utilizes the name information of action sequence samples to construct natural language descriptions with clear semantics, thereby enhancing the correspondence between text features and action semantics. To fully stimulate the semantic expression capabilities of the pre-trained text encoding module, a diverse design of templated sentence structures is introduced when constructing the natural language descriptions of action categories. Compared to a single, fixed prompt template, using multiple semantically equivalent but differently expressed sentence structures (such as "A person is performing the action of {}", "An operator is executing {}", "The action being performed is {}", etc.) helps activate different semantic subspaces in the text encoding module, enhancing the perception of category semantics. Furthermore, sentence diversity improves the robustness of the model during training, mitigating overfitting caused by a single language structure. The encoded vectors are averaged during training or inference.

[0053] The text encoding module is used to generate category text features in the form of embedded vectors for several constructed category text descriptions. An adapter is inserted at the end of the pre-trained Transformer text encoding module. The adapter is a two-layer feedforward neural network. During training, only the parameters in the adapter are updated, and the backbone weights of the text encoding module remain frozen. The adapter is an efficient and lightweight structure that adapts to new tasks with a minimal number of parameters. The text features are adjusted for the task without modifying the weights of the original text encoding module. The adapter structure is a two-layer feedforward neural network, which includes intermediate dimensionality compression and nonlinear transformation. The mathematical expression is shown in Equation (11): (11) in, W and b These represent the weights and biases for each layer.

[0054] like Figure 9 As shown, it first goes through a linear layer The high-dimensional text features are compressed to a lower dimension (from 512 to 64), passed through the nonlinear activation function ReLU, and then through another linear layer. Restored to the original dimensions. βThe residual connection weights are represented by [value]. This "compression-reduction" structure allows the model to capture semantic transformations of new tasks in a low-dimensional space while maintaining computational efficiency. Furthermore, to enhance the adapter's expressive power for semantic transfer, residual connections are used to dynamically mix the fine-tuned knowledge with the original knowledge in the text encoding module's backbone, improving the model's performance stability in new tasks. During training, only the parameters in the adapter need to be updated, while the backbone text weights of the text encoding module remain frozen. This approach not only significantly reduces computational resource consumption but also effectively mitigates the risk of overfitting in few-sample tasks. The adapted text representation is used to align with action sequence features, thereby achieving an efficient combination of semantic consistency enhancement and knowledge transfer in multimodal retrieval and understanding tasks.

[0055] To enhance the alignment between text modalities and skeleton modalities, a contrastive loss function is used to train the assembly action recognition model. This function narrows the feature distance between positive sample pairs while widening the distance between negative sample pairs, enabling the model to learn discriminative cross-modal semantic representations. The contrastive loss function includes loss terms for symmetrical action-to-text and text-to-action directions.

[0056] For text features and action characteristics ,in , B It refers to the batch size, which is the number of matched samples in each group during training. It is called the right side, and For negative pairs, the goal of contrastive loss is to minimize the distance between positive pairs in the embedding space and maximize the distance between negative pairs. Cosine similarity is used as the similarity metric. Specifically, the loss term for the action-to-text direction is shown in equation (12): (12) in, The loss term is for the action-to-text direction. and These are text features and action features, respectively. , B This refers to the batch size during training. sim ( , ) is the cosine similarity function, and τ is the temperature parameter used to control the sharpness of the distribution. In this embodiment, the temperature parameter τ is set to 0.07. The loss term from text to action direction is shown in Equation (13): (13) in, This is the loss term from text to action direction; The contrast loss function is shown in equation (14): (14) in, L contrast The contrastive loss function value is used for comparison. This loss includes the two preceding symmetric loss terms, ensuring that the model learns consistent aligned representations in both directions. By minimizing the contrastive loss function, the model can learn semantically consistent, modality-independent representation structures in the shared embedding space, significantly improving the accuracy and robustness in downstream multimodal retrieval and matching tasks.

[0057] In another exemplary embodiment of this application, a human-machine collaborative speed reducer assembly case is used as an implementation case. The working scenario of this case is as follows: Figure 10 As shown, the overall system includes a collaborative robot 1, a vision sensor 2, a host computer 3, a parts storage area 4, a tool storage area 5, a buffer zone 6, and a finished product storage area 7. The host computer perceives and understands the scene status and operational behaviors based on data collected by the vision sensor, and issues control commands to the collaborative robot in real time to achieve human-robot collaborative operation. The buffer zone is used to store parts or tools to be operated. Human operators perform operations in the work area, and assembled products are moved to the finished product area.

[0058] The entire assembly process is as follows: First, the operator checks the work area to ensure all necessary parts and tools are available. Then, the operator places the reducer base in the work area, and the robot simultaneously provides the worm gear. The operator inserts the worm gear into the housing. The robot provides the worm gear shaft cover, which the operator installs. The robot picks up the worm gear shaft end cap screws, which the operator inserts, and the robot provides a first wrench to assist in tightening. Next, the robot sequentially provides the turbine, large end cap, and end cap screws. After the operator completes the assembly, the robot provides a second wrench to complete the final tightening. Finally, the robot places the assembled reducer onto the conveyor belt, completing the machine delivery. It is worth noting that different operators may have different assembly habits, and some operational steps, such as the installation sequence of the turbine and end caps, can be flexibly adjusted, reflecting the actual openness and dynamism of this task process.

[0059] In this case, there are 8 assembly actions that need to be identified: ① placing the gearbox housing on the assembly area table, ② installing the input shaft into the gearbox, ③ installing the output shaft into the gearbox, ④ removing and installing the top cover, ⑤ removing and installing the side cover, ⑥ inserting the screws into the designated holes, ⑦ tightening the screws using the designated tools, and ⑧ installing the oil plugs into the corresponding ports.

[0060] An Intel RealSense D455 camera was mounted above and to the side of the experimental setup to capture the entire scene from an overhead view. The video was captured at a resolution of 680x480 and a frame rate of 30fps. The assembly of the reducer was performed by four operators to capture worker heterogeneity among the subjects. To reflect the randomness of human behavior, each operator performed each of the eight assembly actions 30 times based on individual operating habits, generating a total of 960 temporal action sequence samples. Each sample included a synchronously acquired RGB video stream and the corresponding operator skeleton data. The operator's skeletal keypoint detection was achieved using the MMPose algorithm. The original skeletal model output by this algorithm contained 17 keypoints. To focus on upper limb and trunk movements, keypoints representing the legs and feet were removed in this case, ultimately using 13 keypoints, such as... Figure 11 As shown, the skeletal nodes are explained below: p 1. Left hip, p 2. Right hip, p 3. Left shoulder p 4. Right shoulder p 5 left elbow, p 6. Left hand p 7 right elbow, p 8. Right hand p 9 left ear, p 10 Left eye, p 11 Right ear, p 12 Right eye, p 13 Nose. The sample set is divided into 80%-20% portions for model training and validation.

[0061] The method proposed in this embodiment is compared with the classic ST-GCN method. The comparison results of the assembly action recognition accuracy are shown in Table 1.

[0062] Table 1 Comparison of Assembly Action Recognition Accuracy

[0063] As can be seen from the results in Table 1, the method proposed in this embodiment has higher accuracy in assembly action recognition.

[0064] To further verify the effectiveness of the proposed multimodal fusion contrastive learning framework in this embodiment, comparisons were made with several representative human action recognition methods on the NTU RGB+D dataset. NTU RGB+D is one of the most widely used large-scale public datasets in the field of human action recognition, covering 60 categories of everyday actions and containing 56,880 action samples. Each frame provides the RGB, depth, and 25 joint skeletons for each object. This dataset provides two standard evaluation benchmarks: cross-subject (xsub) and cross-viewpoint (xview). Experiments were conducted on both benchmarks in this embodiment, comparing the prediction accuracy of the proposed method with several mainstream models. Simultaneously, ablation experiments verified the effectiveness of each module.

[0065] The selected baseline methods cover different structural designs and information processing strategies, including the standard spatiotemporal graph convolutional network ST-GCN, typical multi-stream multi-scale improvement methods (such as 2s-AGCN, MS-G3D, 2s-MST-GCN, etc.), and the recently proposed multimodal fusion model SGM-Net. This reflects the breadth and typicality of the comparative methods in terms of structural types and modal fusion strategies. The experimental results are shown in Table 2.

[0066] Table 2. Accuracy comparison of different recognition methods on the NTU RGB+D dataset.

[0067] As shown in Table 2, under both X-sub and X-view evaluation settings, the proposed method in this embodiment significantly outperforms classic graph neural network models in terms of recognition accuracy and surpasses the latest multimodal fusion methods. This result demonstrates that the contrastive learning framework for multimodal fusion proposed in this application possesses stronger spatiotemporal modeling performance.

[0068] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal assembly action recognition method based on comparative semantic queries, characterized in that, include: Visual sensors are installed above the assembly workbench to acquire video data of human operators performing actions. Random frame sampling is performed on the video data to obtain sampled frame sequence data; Human pose estimation is performed on the video data to obtain human skeleton sequence data; The sampled frame sequence data and the human skeleton sequence data are input into a trained assembly action recognition model for feature extraction and semantic comparison query to obtain the assembly action recognition result. The assembly action recognition model includes an image encoding module, a skeleton encoding module, a feature fusion module, a text encoding module, and a semantic comparison module. The image encoding module is used to extract features from the sampled frame sequence data to obtain image features. The skeleton encoding module is used to extract features from the human skeleton sequence data to obtain skeleton features. The feature fusion module is used to fuse the image features and the skeleton features into action features. The text encoding module is used to encode the preset category text descriptions of each assembly action to obtain several category text features. The semantic comparison module is used to perform semantic comparison between the action features and each category text feature, and outputs the category text description with the highest similarity as the assembly action recognition result. When training the assembly action recognition model, a contrastive loss function is used to train the model.

2. The multimodal assembly action recognition method based on comparative semantic query according to claim 1, characterized in that, Also includes: For various assembly actions of human operators during the assembly process, video data is collected and processed to construct a training sample set and a validation sample set; the training sample set and the validation sample set include several action samples; each action sample includes sampled frame sequence data, human skeleton sequence data and category text description; An assembly action recognition network is constructed, and the assembly action recognition network is trained and validated based on the training sample set and the validation sample set to obtain a trained assembly action recognition model.

3. The multimodal assembly action recognition method based on comparative semantic query according to claim 2, characterized in that, For each assembly action performed by a human operator during the assembly process, video data was collected and processed to construct training and validation sample sets, specifically including: Collect several video data points of N human operators performing K types of assembly actions during the assembly process; each human operator performs each type of assembly action M times, resulting in a total of N×M×K video data points; For any given video data, human pose estimation is performed on the video data to obtain human skeleton sequence data; Random frame sampling is performed on the video data to obtain sampled frame sequence data; Generate a category text description for the video data based on the assembly action and category text description template corresponding to the video data; Based on the human skeleton sequence data, sampled frame sequence data and category text description of the video data, motion samples corresponding to the video data are constructed. The action samples are divided into a training sample set and a validation sample set according to a preset ratio.

4. The multimodal assembly action recognition method based on comparative semantic query according to claim 1, characterized in that, Random frame sampling is performed on the video data to obtain sampled frame sequence data, specifically including: The video data is divided into several segments of equal length; One frame is randomly selected from each segment to form a sampled frame sequence data.

5. The multimodal assembly action recognition method based on comparative semantic query according to claim 1, characterized in that, The skeleton encoding module uses the Spatiotemporal Graph Convolutional Network (ST-GCN) as its backbone network. In ST-GCN, the human skeleton sequence is modeled as a spatiotemporal graph. Multi-layer graph convolution is used to aggregate features of the spatiotemporal neighborhood of each node to extract spatiotemporal context-aware representation features. A lightweight attention block (ATT-Block) is integrated at the end of the skeleton encoding module. The attention block includes a joint temporal attention unit and a channel attention unit.

6. The multimodal assembly action recognition method based on comparative semantic query according to claim 1, characterized in that, The image encoding module employs R(2+1)D ResNet. Operationally, R(2+1)D ResNet decomposes the 3D convolution operation into two stages: a one-dimensional temporal convolution and a two-dimensional spatial convolution. Internally, it uses a residual connection mechanism to construct a deep structure. Structurally, R(2+1)D ResNet includes an initial 3D convolutional layer, multiple sets of R(2+1)D residual layers, and a global average pooling layer. Each set of (2+1)D residual layers contains multiple R(2+1)D residual blocks, and each R(2+1)D residual block contains multiple R(2+1)D convolution operations.

7. The multimodal assembly action recognition method based on comparative semantic query according to claim 1, characterized in that, The feature fusion module uses low-rank bilinear attention pooling to fuse the skeletal features and the image features, and generates modal attention based on the fused representation to guide the enhancement and weighted fusion of the original features.

8. The multimodal assembly action recognition method based on comparative semantic query according to claim 1, characterized in that, Based on the assembly action and category text description template corresponding to the video data, when generating the category text description of the video data, multiple semantically equivalent but differently expressed sentence structures are used to construct a natural language description with clear semantics as the category text description of the assembly action; the text encoding module is used to generate category text features in the form of embedded vectors for the constructed category text descriptions; an adapter is inserted at the end of the pre-trained Transformer text encoding module, the adapter being a two-layer feedforward neural network, during training only updating the parameters in the adapter, while the backbone weights of the text encoding module remain frozen.

9. The multimodal assembly action recognition method based on comparative semantic query according to claim 1, characterized in that, When training the assembly action recognition model, the contrastive loss function is used to train the model. By narrowing the feature distance between positive sample pairs and widening the distance between negative sample pairs, the model learns a discriminative cross-modal semantic representation. The contrastive loss function includes loss terms for symmetrical action-to-text direction and text-to-action direction, and the temperature parameter τ is set to 0.

07.

10. The multimodal assembly action recognition method based on comparative semantic query according to claim 9, characterized in that, The loss term for the action-to-text direction is shown in the following formula: ； in, The loss term is for the action-to-text direction. and These are text features and action features, respectively. , B This refers to the batch size during training. sim ( , ) is the cosine similarity function, and τ is the temperature parameter used to control the sharpness of the distribution; The loss term from text to action direction is shown in the following formula: ； in, This is the loss term from text to action direction; The contrastive loss function is shown in the following equation: ； in, L contrast To compare the loss function values.

Images