A Multimodal Assembly Action Recognition Method Based on Comparative Semantic Queries

Abstract

This application discloses a multimodal assembly action recognition method based on contrastive semantic query, relating to the field of human-machine collaborative assembly technology. The method includes: acquiring operator action videos by deploying visual sensors on an assembly workbench; obtaining a sampled frame sequence through random frame sampling and a skeletal sequence through human pose estimation; and inputting these sequences into an assembly action recognition model to complete the recognition. The model includes image encoding, skeletal encoding, feature fusion, text encoding, and semantic comparison modules. These modules extract image and skeletal features, fuse features, encode preset category text descriptions, compare action features with category text features, and output the result with the highest similarity. A contrastive loss function is used during training. This method integrates multimodal information, solving the limitations of single-modality recognition and the problem of multimodal semantic fragmentation. It fully utilizes category text semantics, improves fine-grained action recognition accuracy, reduces the risk of overfitting, and enhances the model's generalization and task transfer capabilities in dynamic industrial scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-robot collaborative assembly, in particular to a multi-modal assembly action recognition method for contrastive semantic query. BACKGROUND

[0002] Under the background that manufacturing industry is transforming from traditional mass production and standardization to multi-variety, small batch and individual customization, traditional fully automatic assembly line relies on preset process and fixed structure, and is difficult to adapt to flexible and variable assembly requirements, and its applicability and expansion ability are limited.

[0003] To improve the flexibility and intelligence level of manufacturing system, human-robot collaborative assembly system (HRCA) emerges as the times require. The HRCA system introduces robots into human work space, realizes the combination of flexibility of human operation and stability of machine control, and improves the assembly efficiency and system adaptability to a certain extent. With the evolution of collaborative mode from static division of labor to dynamic sharing space, collaborative robots are required to have stronger environmental perception and behavior understanding ability, and can identify and respond to the actions and intentions of operators in unstructured scenes in real time, so as to realize the functions of active cooperation such as grasping, assisting and error correction.

[0004] In the prior art, action recognition for HRCA mainly adopts three types of perception methods: one is the recognition method based on inertia or electromyography, which is fast in response, but is difficult to be widely deployed in complex industrial environments due to high wearing cost, large human body difference and other factors; the second is the visual recognition method based on RGB image, which has rich spatial information, but is not robust under the influence of factors such as light change and occlusion interference; the third is the method based on human body skeleton sequence, which takes three-dimensional joint trajectory as the core, has the advantages of structured, lightweight and strong robustness to background interference, and has become the mainstream scheme in current industrial action recognition. These methods all send the extracted features into a unified classifier in the last step, use softmax function to generate probability distribution, use 0 to N-1 numbering instead of N classes, and the class number corresponding to the maximum probability is the output recognition result.

[0005] The prior art still has the following deficiencies in the actual HRCA scene: 1. The data expression ability of a single modality is limited, and the skeleton information can only describe the geometric structure and motion trajectory of the human body itself, and cannot capture the interaction semantics between the human and the tool and the assembly object, resulting in ambiguity in high-similarity fine-grained action recognition. 2. The multi-modal fusion mechanism is imperfect, and the current multi-modal recognition method mainly adopts simple feature splicing or early fusion strategy, and the semantic alignment relationship between the modes is not established. 3. The text description of the class label is not fully utilized, and still relies on a fixed label classifier for recognition and judgment.

[0006] Therefore, the prior art still lacks an efficient action recognition method that can fuse skeleton structure, visual information and language semantics, especially lacks a fusion framework with cross-modal semantic alignment capability suitable for fine-grained assembly action recognition, and a new solution is urgently needed to improve the intelligent collaboration level and practical deployment capability of the HRCA system. SUMMARY

[0007] The purpose of the present application is to provide a multi-modal assembly action recognition method for contrastive semantic query, which can solve the problems of single modality limitation and multi-modal semantic fragmentation, and has cross-modal semantic alignment capability and is suitable for fine-grained assembly action recognition.

[0008] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0009] A multi-modal assembly action recognition method for contrastive semantic query, comprising the following steps:

[0010] A visual sensor is arranged above the assembly workbench to obtain video data when a human operator performs an action.

[0011] Random frame sampling is performed on the video data to obtain sample frame sequence data.

[0012] Human body pose estimation is performed on the video data to obtain human body skeleton sequence data.

[0013] The sample frame sequence data and the human body skeleton sequence data are input into a trained assembly action recognition model for feature extraction and semantic contrastive query to obtain an assembly action recognition result.

[0014] The assembly action recognition model comprises an image coding module, a skeleton coding module, a feature fusion module, a text coding module and a semantic comparison module; the image coding module is configured to extract features from the sampled frame sequence data to obtain image features; the skeleton coding module is configured to extract features from the human skeleton sequence data to obtain skeleton features; the feature fusion module is configured to fuse the image features and the skeleton features into action features; the text coding module is configured to code the preset category text description of each assembly action to obtain a plurality of category text features; the semantic comparison module is configured to compare the action features with each category text feature respectively, and output the category text description with the highest similarity as the assembly action recognition result; when training the assembly action recognition model, the model is trained by using a contrast loss function.

[0015] According to the specific embodiments provided in the present application, the following technical effects are disclosed:

[0016] The present application provides a multi-modal assembly action recognition method for contrast semantic query, which comprises: arranging a visual sensor above an assembly workbench, synchronously collecting visual information to provide a basis for subsequent multi-modal fusion, and compared with a single skeleton modality, the work environment and the interaction details of the human and the tool and the assembly object can be supplemented; random frame sampling is performed on video data to obtain sampled frame sequence data, and human pose estimation is performed on the video data to obtain human skeleton sequence data; random frame sampling can reduce data redundancy while retaining key action information, and human pose estimation accurately extracts skeleton motion trajectories, and the combination of the two forms a complementary multi-modal data structure, i.e., the skeleton sequence captures the dynamic trajectory of the human body, and the sampled frame sequence records the characteristics of the environment and the interactive objects, which together enhance the distinguishability of high similarity actions and reduce ambiguity in fine-grained recognition. Then, the sampled frame sequence data and the human skeleton sequence data are input into a trained assembly action recognition model for feature extraction and semantic comparison query to obtain an assembly action recognition result. The assembly action recognition model used in this step comprises an image coding module, a skeleton coding module, a feature fusion module, a text coding module and a semantic comparison module; image features and skeleton features are extracted respectively, and after the action features are generated by fusion, the action features and the text semantics are directly associated through cross-modal semantic matching, which significantly improves the recognition accuracy of fine-grained actions; when training the assembly action recognition model, the model is trained by using a contrast loss function, the distance of the matching pairs is pulled closer and the distance of the non-matching pairs is pushed farther apart, which promotes the model to learn cross-modal representations with consistent semantics, effectively reduces the risk of overfitting, and improves the generalization ability and task migration ability of the model in dynamic industrial scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0018] Figure 1 A flowchart of a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0019] Figure 2 A technical framework diagram of applying an assembly action recognition model in a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0020] Figure 3 A flowchart of constructing an assembly action recognition network and training in a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0021] Figure 4 A flowchart of step B1 in a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0022] Figure 5 A technical framework diagram of training an assembly action recognition model in a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0023] Figure 6 A structure diagram of ST-Block in a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0024] Figure 7 A structure diagram of ATT-Block in a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0025] Figure 8 A structure diagram of a feature fusion module in a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0026] Figure 9 A structure diagram of an adapter in a multi-modal assembly action recognition method for comparative semantic query provided by an embodiment of the present application.

[0027] Figure 10 A working scene diagram of a human-machine cooperation speed reducer assembly case provided by an embodiment of the present application.

[0028] Figure 11A skeleton model diagram adopted in a multi-modal assembly action recognition method for a contrastive semantic query provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0030] In order to make the above objectives, characteristics and advantages of the present application more apparent, further detailed description of the present application will be given below with reference to the drawings and specific embodiments.

[0031] A multi-modal assembly action recognition method for a contrastive semantic query provided by an embodiment of the present application includes the following steps in an exemplary embodiment, as shown in Figure 1

[0032] A1, a visual sensor is arranged above an assembly workbench to obtain video data of human operators performing actions.

[0033] A2, random frame sampling is performed on the video data to obtain sample frame sequence data. Specifically, the video data is divided into several segments of equal length; a frame is randomly taken in each segment to form the sample frame sequence data.

[0034] A3, human body posture estimation is performed on the video data to obtain human body skeleton sequence data.

[0035] A4, the sample frame sequence data and the human body skeleton sequence data are input into a trained assembly action recognition model for feature extraction and semantic contrastive query to obtain an assembly action recognition result.

[0036] In the method provided in the present embodiment, the assembly action recognition model includes an image encoding module, a skeleton encoding module, a feature fusion module, a text encoding module and a semantic contrast module. As shown in Figure 2 ​The technical framework diagram when the model is applied is shown. The model input is a sampling frame sequence and a skeleton sequence based on an RGB image sampling; an image coding module is used for feature extraction on the sampling frame sequence data to obtain image features; a skeleton coding module is used for feature extraction on the human skeleton sequence data to obtain skeleton features; a feature fusion module is used for fusing the image features and the skeleton features into action features; a text coding module is used for text coding on the preset category text description of each assembly action to obtain a plurality of category text features; a semantic comparison module is used for respectively comparing the action features with each category text feature in terms of semantics, and outputting the category text description with the highest similarity as the assembly action recognition result; when the assembly action recognition model is trained, the model is trained by using a comparison loss function. After the assembly action recognition result is obtained, the assembly intention of the human operator is judged, and the collaborative robot can be controlled to perform the corresponding collaborative operation.

[0037] In another exemplary embodiment of the present application, the method further includes a process of constructing an assembly action recognition network and training, as shown in Figure 3 The process includes the following steps:

[0038] B1, for various assembly actions of human operators in the assembly process, video data is collected and processed respectively, and a training sample set and a verification sample set are constructed. The training sample set and the verification sample set include a plurality of action samples; any action sample includes sampling frame sequence data, human skeleton sequence data and category text description. In the present embodiment, as shown in Figure 4 The step B1 specifically includes the following steps:

[0039] B11, a plurality of video data of N human operators performing K types of assembly actions in the assembly process is collected; each human operator repeats the performance of each type of assembly action M times, and a total of N×M×K video data is obtained.

[0040] B12, for any video data, human pose estimation is performed on the video data to obtain human skeleton sequence data.

[0041] B13, random frame sampling is performed on the video data to obtain sampling frame sequence data. In the present embodiment, the step B13 specifically includes: dividing the video data into a plurality of segments of equal length; randomly taking a frame in each segment to form the sampling frame sequence data.

[0042] B14, according to the assembly action and the category text description template corresponding to the video data, the category text description of the video data is generated.

[0043] B15, according to the human skeleton sequence data, the sampling frame sequence data and the category text description of the video data, an action sample corresponding to the video data is constructed.

[0044] B16, a plurality of action samples are divided into a training sample set and a validation sample set according to a preset proportion.

[0045] B2, an assembly action recognition network is built, and the assembly action recognition network is trained and verified based on the training sample set and the validation sample set, to obtain a trained assembly action recognition model. The process of training the assembly action recognition network is as shown in the technical framework diagram. Figure 5

[0046] Figure 5 The left side is an assembly work scene and data collection diagram. In the assembly work process, a RealSense camera is used to continuously monitor the work site, collect RGB video data, and extract the corresponding skeleton key point sequence through human pose estimation, and the category label of the assembly action is constructed into a semantic complete category text description through a category text description template prompt. Thus, a multi-modal data input integrating spatial and action information is constructed. Figure 5 The right upper part shows the feature extraction and fusion process of the skeleton sequence and the sampling frame sequence, including the encoding of the skeleton sequence and the sampling frame sequence. The skeleton sequence extracts action features through a spatio-temporal graph convolution network, and the sampling frame sequence obtains scene information through an R(2+1)D ResNet module. The two types of features are fused to represent the temporal dynamics and semantic scene of the action sample. Figure 5 The lower right part shows the feature extraction process of the category description text. The category description text passes through a pre-trained text encoding module and an adapter module in sequence to generate an embedding vector of the category text description as a text feature.

[0047] In the training phase, each Batch contains B pairs of action features and their corresponding text features, and through a contrastive learning framework, the distance between matching pairs is narrowed, while the distance between mismatched samples is pushed away, to realize cross-modal representation alignment. In the inference phase, all candidate category text descriptions are first generated through a text encoding module to generate their respective category text features. For a new action sample input, the action features are first generated through an image encoding module, a skeleton encoding module and a feature fusion module, and then matched with each category text feature. The nearest neighbor search method is used to retrieve the highest similarity category text description as the assembly action recognition result.

[0048] ​The skeleton encoding module adopts a Spatial-Temporal Graph Convolutional Network (ST-GCN) as a backbone network; a multi-layer graph convolution is used to aggregate the features of the spatial and temporal neighborhood of each node, to extract the representation features of the spatial and temporal context awareness, and a lightweight attention block ATT-Block is integrated at the end of the skeleton encoding module; the attention block includes a joint temporal attention unit and a channel attention unit.

[0049] In the ST-GCN, a human skeleton sequence is modeled as a spatio-temporal graph wherein V represents a set of all joint nodes, E S represents a spatial edge (topological connection between joints), E T represents a temporal edge (connection between the same joint in adjacent time frames). Each node represents the t joint in the i frame. The input feature can be represented as a tensor wherein C represents the number of input channels (for example, the coordinate dimension), T represents the number of time frames, V represents the number of joint nodes in each frame.

[0050] The ST-GCN performs a graph convolution operation in both the spatial and temporal directions, as shown in Figure 6 In each ST-Block, first, ATT is performed, which includes a learnable adjacency weight mask for adjusting the importance of the edge; then, spatial graph convolution GCN is performed to extract and aggregate the spatial features, and the mathematical description is shown in equation (1):

[0051] (1)

[0052] wherein W d is a spatial convolution parameter, M d is an adjacency weight mask, and represents an element-wise multiplication. d is the graph distance between the joint nodes, D is the maximum distance, wherein is an adjacency matrix, is the degree matrix of the node. Then, each node feature in the time dimension is one-dimensionally convolved through the TCN to model the dynamic changes. The ST-GCN aggregates the features of the spatio-temporal neighborhood of each node through multi-layer graph convolution to extract the representation features with spatio-temporal context awareness. The spatial convolution captures the spatial dependency between joints, while the temporal convolution models the dynamic evolution characteristics of each node in the sequence, thereby realizing the modeling of complex motion patterns. The entire ST-Block block is composed of 9 ST-GCN networks stacked as described above, which can extract high-order semantic features of spatial structure and temporal dynamics layer by layer.

[0053] In the process of personnel action recognition and understanding, the importance of different joints at different times varies significantly. To more accurately focus on the time frames (time dimension), joints (spatial dimension), and semantic channels (channel dimension) that are highly relevant to the task, while reducing redundant information interference and improving the abstract ability of high-order semantic features, the embodiment introduces a global fusion-based attention module. Considering the adaptability of the model on small-scale datasets, the attention module is designed to be lightweight and integrated only at the end of the skeleton feature extractor to improve the effectiveness of feature representation while controlling the parameter amount. As shown in Figure 7 The attention module includes a joint temporal attention (JTAM) and a channel attention (CAM).

[0054] Specifically, for the spatio-temporal features output by the ST-GCN, the JTAM part is used to capture the importance of which joints at which times. Average pooling and max pooling are performed in the channel direction to extract the overall distribution features and the locally strongest response features in the joint and time dimensions, respectively. The features are then concatenated to enhance the expression ability, and then a spatio-temporal attention map is obtained through convolution, as shown in equations (2)-(4):

[0055] , (2)

[0056] (3)

[0057] (4)

[0058] The CAM part is used to capture the importance of different channels to the final prediction. Average pooling and max pooling are performed in the joint and time dimensions, and then MLP mapping is used to calculate the channel attention. The above process is shown in equations (5)-(7):

[0059] , (5)

[0060] (6)

[0061] (7)

[0062] spatial-temporal features output by the ST-GCN in turn through the JTAM and the CAM, , Such an order can achieve better results. Finally, the final feature is obtained through spatial-temporal global average pooling, and is mapped to a specified dimension through a linear layer. The final output of the skeleton encoding module is , and d is the feature dimension (for example, 512).

[0063] The image encoding module adopts R(2+1)D ResNet. In terms of operation, R(2+1)D ResNet decomposes three-dimensional convolution operation (3D Convolution) into one-dimensional time domain convolution (Temporal Convolution) and two-dimensional space domain convolution (Spatial Convolution) two stages, and adopts a residual connection mechanism to construct a deep structure; in terms of structure, R(2+1)D ResNet includes an initialized three-dimensional convolution layer, multiple groups of R(2+1)D residual layers, and a global average pooling layer; each group of (2+1)D residual layers contains multiple R(2+1)D residual blocks, and each R(2+1)D residual block contains multiple R(2+1)D convolution operations.

[0064] For an input tensor , where C is the number of input channels (such as the number of RGB channels), G is the number of time frames (such as 16 frames), and HxW is the spatial size of the frame (such as 480x640), R(2+1)D ResNet network decomposes three-dimensional convolution operation into a two-dimensional spatial convolution and a one-dimensional time convolution. In each R(2+1) convolution operation, the spatial convolution kernel is , where is the number of intermediate channels, the size of the spatial convolution kernel, the time dimension is 1, only processing the spatial features within a single frame, not across frames, and the output is , (the spatial size changes due to the step or padding). The time convolution kernel is , where t is the time receptive field (such as 3 frames), is the number of output channels, only aggregating time sequence information, not changing the spatial resolution, and the output is (time size changes due to step or padding).

[0065] To improve the training stability and network depth expression ability, a residual connection mechanism is used to build a deep structure in the R(2+1)D network, specifically, Figure 1 The network structure of the R(2+1)D Resnet module in the R(2+1)D network is composed of an initialized three-dimensional convolutional layer, multiple groups of R(2+1)D residual layers, and a global average pooling layer. Each group of (2+1)D residual layers includes multiple R(2+1)D residual blocks, and each residual block includes multiple R(2+1)D convolution operations. The first block of each residual layer realizes spatial downsampling through a convolution with a spatial stride of 2, reducing the height and width of the feature map by half. Finally, the global average pooling layer is used to aggregate the time and spatial dimensions, and the output result of the RGB feature extraction is , (d is the feature dimension).

[0066] The feature fusion module uses a low-rank bilinear attention pooling method to fuse the skeleton features and image features, and generates modality attention based on the fusion representation to guide the enhancement and weighted fusion of the original features. In this embodiment, the low-rank bilinear attention pooling (LBAP) method is used to fuse the skeleton features and RGB features, and generates modality attention based on the fusion representation to guide the enhancement and weighted fusion of the original features, which can fully exploit the complementary information of image sequences and skeleton sequences in the spatio-temporal structure. As shown in Figure 8 , given the RGB feature and the skeleton sequence feature , first cross-modal interaction is performed through the low-rank bilinear pooling shown in equation (8):

[0067] (8)

[0068] where, and are the projection matrices of the modal, and is the Hadamard product, is the low-rank projection matrix (k<512), and the output fusion feature is obtained. The fusion feature is then used to generate an attention weight vector through a fully connected and one-dimensional convolution as shown in equation (9):

[0069] (9)

[0070] Two channels are used to obtain and , respectively, which are cross-multiplied with the original features, and the final feature representation is obtained through feature weighted addition as shown in equation (10):

[0071] (10)

[0072] In this embodiment, by introducing a bilinear attention pooling mechanism, the model can explicitly model the high-order interaction between the image and the skeleton, improve the consistency and discriminativeness of the cross-modal features, and thus effectively improve the performance of multi-modal action recognition.

[0073] According to the assembly action and the category text description template corresponding to the video data, the category text description of the video data is generated, a variety of semantically equivalent but differently expressed sentence patterns are used, and a natural language description with clear semantics is constructed as the category text description of the assembly action.

[0074] In the multi-modal task, compared with the traditional method of using discrete category numbers without semantic meaning as labels, the application makes full use of the name information of the action sequence sample, and constructs it into a natural language description with clear semantics, thereby enhancing the correspondence between the text features and the action semantics. In order to fully stimulate the semantic expression ability of the pre-trained text encoding module, the diversity design of the template sentence is introduced when constructing the natural language description of the action category. Compared with a single and fixed prompt template, using a variety of semantically equivalent but differently expressed sentence patterns (such as "A person is performing the action of {}", "An operator is executing {}", "The action being performed is {}") helps to activate different semantic subspaces in the text encoding module and enhances the perception ability of the category semantics. In addition, the diversity of the sentence pattern can improve the robustness of the model during training and alleviate the overfitting phenomenon caused by the single language structure. The average of their encoding vectors is used during training or reasoning.

[0075] The text encoding module is used to generate category text features in the form of embedding vectors for the constructed category text descriptions; an adapter is inserted at the end of the pre-trained Transformer text encoding module, the adapter is a two-layer feedforward neural network, only the parameters in the adapter are updated during training, and the backbone weights of the text encoding module remain frozen. The adapter is a high-efficiency lightweight structure that adapts to new tasks with minimal parameter amount. Without modifying the original text encoding module weights, the text features are adjusted specifically for the task. The adapter structure is a two-layer feedforward neural network, including dimension compression and nonlinear transformation in the middle, and the mathematical expression is as shown in formula (11):

[0076] (11)

[0077] wherein, W and b are the weights and biases of each layer, respectively.

[0078] AsFigure 9 As shown, first through a linear layer The high-dimensional text features are compressed to a lower dimension (from 512 to 64), and after the Nonlinear activation function ReLU, another linear layer is used Restored to the original dimension. Among them, β Residual connection weight. This "compression-recovery" structure allows the model to capture the semantic transformation of the new task in the low-dimensional space while maintaining computational efficiency. Further, to enhance the expression ability of the adapter to semantic transfer, residual connection is used to dynamically mix the fine-tuned knowledge with the original knowledge in the backbone of the text encoding module, improving the performance stability of the model in the new task. During training, only the parameters in the adapter need to be updated, while the text weights in the backbone of the text encoding module remain frozen. This approach not only significantly reduces the computational resource consumption, but also effectively reduces the risk of overfitting in small sample tasks. The adapted text representation is used to align with the action sequence features, thereby achieving efficient combination of semantic consistency enhancement and knowledge transfer in multi-modal retrieval and understanding tasks.

[0079] To enhance the alignment capability between the text and skeleton modalities, when training the assembled action recognition model, the model is trained using a contrastive loss function, which pulls the feature distance between positive sample pairs closer while pushing the distance between negative sample pairs further apart, prompting the model to learn discriminative cross-modal semantic representations; the contrastive loss function includes symmetric action-to-text and text-to-action loss terms.

[0080] For text features and action features , where , B is the size of the Batch, and during training, each set of matched samples is called a positive pair, while is a negative pair. The goal of the contrastive loss is to make the positive pairs as close as possible in the embedding space, while making the negative pairs as far apart as possible. Cosine similarity is used as a measure of similarity, . Specifically, the action-to-text loss term is shown in equation (12):

[0081] (12)

[0082] where is the action-to-text loss term, and are the text features and action features, , B is the size of the Batch during training, sim(, ) is a cosine similarity function, and τ is a temperature parameter used to control the sharpness of the distribution, in this embodiment, the temperature parameter τ is set to 0.07;

[0083] The loss term of text to action direction is shown in formula (13):

[0084] (13)

[0085] wherein, is the loss term of text to action direction;

[0086] The contrast loss function is shown in formula (14):

[0087] (14)

[0088] wherein, L contrast is the contrast loss function value. The loss contains the previous two symmetrical loss terms, which ensures that the model can learn consistent alignment representations in both directions. By minimizing the contrast loss function, the model can learn semantic consistent and modality independent representation structure in the shared embedding space, which significantly improves the accuracy and robustness in downstream multi-modal retrieval and matching tasks.

[0089] In another exemplary embodiment of the present application, a human-robot collaborative speed reducer assembly case is used as an implementation case, and the work scene structure of the case is as shown in Figure 10 The overall system includes a collaborative robot 1, a visual sensor 2, an upper computer 3, a part storage area 4, a tool storage area 5, a buffer area 6 and a finished product storage area 7. The upper computer perceives and understands the scene state and operation behavior based on the data collected by the visual sensor, and issues control instructions to the collaborative robot in real time to realize human-robot collaborative operation. The buffer area is used to store parts or tools to be operated, human operators operate in the work area, and the assembled products are moved to the finished product area.

[0090] The entire assembly process is as follows: the worker first checks the working area status to confirm that the required parts and tools are ready. Then, the manual operator places the reducer base to the working area, and the robot will provide the worm synchronously. The operator inserts the worm into the housing. The robot provides the worm shaft cover, and the worker installs the worm shaft cover. The robot grabs the worm shaft end cover screw, the operator inserts the screw, and the robot provides the first wrench to assist screw tightening. Then, the robot provides the turbine, large end cover and end cover screw in turn, and after the worker completes the assembly, the robot provides the second wrench to complete the final tightening operation. Finally, the robot places the assembled reducer on the conveyor belt to realize the delivery of the whole machine. It is worth noting that different operators may have different assembly habits, and some operation steps such as the installation order of the turbine and the end cover have flexibility, reflecting the actual openness and dynamics of the task flow.

[0091] In this case, there are 8 assembly actions that need to be identified: ① placing the gearbox housing on the assembly area table, ② installing the input shaft into the gearbox, ③ installing the output shaft into the gearbox, ④ taking the top cover plate, ⑤ taking the side cover plate, ⑥ inserting the screw into the specified hole, ⑦ tightening the screw using the specified tool, and ⑧ installing the oil plug into the corresponding port.

[0092] An Intel Realsense D455 camera was installed on the side above the experimental scene to ensure a bird's-eye view to capture the entire scene. The video capture resolution was 680x480, and the frame rate was 30fps. The assembly task of the reducer was completed by four operators to capture the worker heterogeneity between subjects. To reflect the randomness of human behavior of the subjects, each operator repeated each of the 8 assembly actions 30 times based on individual operation habits, generating a total of 960 time-series action sequence samples. Each sample included a synchronously captured RGB video stream and the corresponding operator skeleton data. The operator's human skeleton key point detection was achieved using the MMPose algorithm. The original skeleton model output by this algorithm included 17 key points. To focus on upper limb and torso actions, the key points representing the legs and feet were removed in this case, and finally 13 key points were used, as shown in Figure 1. Figure 11 p 1 left hip, p 2 right hip, p 3 left shoulder, p 4 right shoulder, p 5 left elbow, p 6 left hand, p 7 right elbow, p 8 right hand, p 9 left ear, p 10 left eye, p 11 right ear, p ​12 Right eye, p 13 Nose. The sample set is divided into 80%-20% portions for model training and validation.

[0093] The method proposed in this embodiment is compared with the classic ST-GCN method. The comparison results of the assembly action recognition accuracy are shown in Table 1.

[0094] Table 1 Comparison of Assembly Action Recognition Accuracy

[0095]

[0096] As can be seen from the results in Table 1, the method proposed in this embodiment has higher accuracy in assembly action recognition.

[0097] To further verify the effectiveness of the proposed multimodal fusion contrastive learning framework in this embodiment, comparisons were made with several representative human action recognition methods on the NTU RGB+D dataset. NTU RGB+D is one of the most widely used large-scale public datasets in the field of human action recognition, covering 60 categories of everyday actions and containing 56,880 action samples. Each frame provides the RGB, depth, and 25 joint skeletons for each object. This dataset provides two standard evaluation benchmarks: cross-subject (xsub) and cross-viewpoint (xview). Experiments were conducted on both benchmarks in this embodiment, comparing the prediction accuracy of the proposed method with several mainstream models. Simultaneously, ablation experiments verified the effectiveness of each module.

[0098] The selected baseline methods cover different structural designs and information processing strategies, including the standard spatiotemporal graph convolutional network ST-GCN, typical multi-stream multi-scale improvement methods (such as 2s-AGCN, MS-G3D, 2s-MST-GCN, etc.), and the recently proposed multimodal fusion model SGM-Net. This reflects the breadth and typicality of the comparative methods in terms of structural types and modal fusion strategies. The experimental results are shown in Table 2.

[0099] Table 2. Accuracy comparison of different recognition methods on the NTU RGB+D dataset.

[0100]

[0101] As shown in Table 2, under both X-sub and X-view evaluation settings, the proposed method in this embodiment significantly outperforms classic graph neural network models in terms of recognition accuracy and surpasses the latest multimodal fusion methods. This result demonstrates that the contrastive learning framework for multimodal fusion proposed in this application possesses stronger spatiotemporal modeling performance.

[0102] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, it should be understood that the application encompasses all possible combinations of the technical features unless such a combination is not technically possible.

[0103] The principles and implementation manners of the present application are described herein by using specific examples, and the above embodiments are only used to help understand the method of the present application and its core idea; meanwhile, according to the idea of the present application, the specific implementation manners and application scopes will be changed by those skilled in the art. In conclusion, the content of the present specification should not be understood as a limitation of the present application.

Claims

1.A method for multi-modal assembly action recognition for contrastive semantic query, characterized in that, The method comprises the following steps: a visual sensor is arranged above an assembly workbench to obtain video data of human operators performing actions; random frame sampling is performed on the video data to obtain sampled frame sequence data; human pose estimation is performed on the video data to obtain human skeleton sequence data; the sampled frame sequence data and the human skeleton sequence data are input into a trained assembly action recognition model for feature extraction and semantic comparison to obtain an assembly action recognition result; the assembly action recognition model comprises an image encoding module, a skeleton encoding module, a feature fusion module, a text encoding module and a semantic comparison module; the image encoding module is used for feature extraction on the sampled frame sequence data to obtain image features; the skeleton encoding module is used for feature extraction on the human skeleton sequence data to obtain skeleton features; the feature fusion module is used for fusing the image features and the skeleton features into action features; the text encoding module is used for text encoding on pre-set category text descriptions of each assembly action to obtain a plurality of category text features; the semantic comparison module is used for semantic comparison between the action features and each category text feature, and outputs a category text description with the highest similarity as the assembly action recognition result; when training the assembly action recognition model, a contrastive loss function is used to train the model; the contrastive loss function comprises symmetric loss terms in the action-to-text direction and the text-to-action direction; the loss term in the action-to-text direction is as follows: ； wherein, is a loss term for action-to-text direction, and are text feature and action feature, respectively, , B is the size of Batch during training, sim (, ) is a cosine similarity function, and τ is a temperature parameter used to control the sharpness of the distribution. the loss term in the text-to-action direction is as follows: ； wherein, is the text-to-action direction loss term; the contrastive loss function is as follows: ； wherein, L contrast For comparison loss function values. 2.The method of claim 1, wherein, The method further comprises the following steps: for various assembly actions of human operators in the assembly process, video data is collected and processed respectively to construct a training sample set and a verification sample set; the training sample set and the verification sample set each comprise a plurality of action samples; any action sample comprises sampled frame sequence data, human skeleton sequence data and a category text description; an assembly action recognition network is built, and the assembly action recognition network is trained and verified based on the training sample set and the verification sample set to obtain a trained assembly action recognition model. 3.The method of claim 2, wherein, For various assembly actions of human operators in the assembly process, video data is collected and processed respectively to construct a training sample set and a verification sample set, specifically comprising the following steps: N human operators perform K types of assembly actions, and a plurality of video data of the N human operators performing the K types of assembly actions are collected; each human operator performs each type of assembly action M times, and N×M×K video data are obtained in total; for any video data, human pose estimation is performed on the video data to obtain human skeleton sequence data; random frame sampling is performed on the video data to obtain sampled frame sequence data; a category text description of the video data is generated according to the assembly action corresponding to the video data and a category text description template; an action sample corresponding to the video data is constructed according to the human skeleton sequence data, the sampled frame sequence data and the category text description of the video data. A plurality of the action samples are divided into a training sample set and a verification sample set according to a preset proportion. 4.The method of claim 1, wherein, Random frame sampling is performed on the video data to obtain sample frame sequence data, specifically including: The video data is divided into a plurality of segments of equal length; A frame is randomly taken in each segment to form sample frame sequence data. 5.The method of claim 1, wherein, The skeleton encoding module adopts a space-time graph convolution network ST-GCN as a backbone network; in the ST-GCN, a human body skeleton sequence is modeled as a space-time graph, and a multi-layer graph convolution is used to aggregate features of a space-time neighborhood of each node, to extract a representation feature with space-time context awareness, and a lightweight attention block ATT-Block is integrated at the end of the skeleton encoding module; the attention block includes a joint time attention unit and a channel attention unit. 6.The method of claim 1, wherein, The image encoding module adopts an R(2+1)D ResNet; in operation, the R(2+1)D ResNet decomposes a three-dimensional convolution operation into two stages of one-dimensional time domain convolution and two-dimensional space domain convolution, and uses a residual connection mechanism to construct a deep structure; in structure, the R(2+1)D ResNet includes an initialized three-dimensional convolution layer, a plurality of groups of R(2+1)D residual layers, and a global average pooling layer; each group of (2+1)D residual layers includes a plurality of R(2+1)D residual blocks, and each R(2+1)D residual block includes a plurality of R(2+1)D convolution operations. 7.The method of claim 1, wherein, The feature fusion module adopts a low-rank bilinear attention pooling method to fuse the skeleton features and the image features, and generates modal attention based on the fused representation, to guide the enhancement and weighted fusion of the original features. 8.The method of claim 1, wherein, When generating the category text description of the video data according to the corresponding assembly action and the category text description template, a plurality of sentence patterns with different expression manners but equivalent semantics are used to construct natural language descriptions with clear semantics as the category text description of the assembly action; the text encoding module is used to generate embedded vector form category text features for the constructed category text descriptions; an adapter is inserted at the end of the pre-trained Transformer text encoding module, the adapter is a two-layer feedforward neural network, only the parameters in the adapter are updated during training, and the backbone weights of the text encoding module remain frozen. 9.The method of claim 1, wherein, When training the assembly action recognition model, the model is trained by using a contrastive loss function, the feature distance between positive sample pairs is narrowed, and the distance of negative sample pairs is pushed away, so that the model learns discriminative cross-modal semantic representations; a temperature parameter τ is set to 0.07.

Images