Industrial assembly action recognition and error correction training method based on deep learning

Abstract

The invention provides an industrial assembly action recognition and error correction training method based on deep learning, and the method comprises the steps: achieving the synchronous collection of the whole assembly process through the deployment of a multi-mode perception collection system; a standard assembly database is constructed, a three-dimensional simulation platform is introduced, random disturbance is applied through a parameterized digital human body model, field disturbance is simulated, and a virtual sample enhancement training set is generated; based on real and virtual samples, constructing a deep neural network model, fusing a graph convolutional network, an LSTM and a convolutional network to extract skeleton, trajectory and mechanical features, and realizing assembly action recognition, anomaly detection and key frame positioning; and an assembly quality scoring function is constructed through the track consistency, the attitude offset and the mechanical anomaly index, multi-stage error correction feedback is generated, and personalized training is guided. The method is high in recognition precision of errors in assembly actions of assembly operators, can achieve timely and intelligent feedback, and is suitable for assembly quality optimization and skill training in an industrial scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an industrial assembly action recognition and error correction training method based on deep learning. Background Art

[0002] Currently, industrial assembly training typically relies on traditional manual demonstrations and static instructions, such as text, drawings, or videos. This training approach lacks real-time and personalized feedback, making it difficult to accurately identify incorrect steps during assembly, impacting assembly quality and production efficiency.

[0003] Although some systems have attempted to introduce sensors and image recognition technology for process monitoring, the following technical bottlenecks still exist: first, there is a lack of continuous capture and high-precision modeling capabilities of workers' movements throughout the entire process; second, there is an inability to automatically classify and trace the types of errors in complex assembly movements; and third, the feedback mechanism is single and cannot generate differentiated training suggestions based on individual differences.

[0004] Therefore, we need to develop a deep learning-based industrial assembly motion recognition and error correction training method that can integrate multi-source perception, deep motion understanding and virtual feedback to achieve accurate recognition and error correction training in industrial assembly scenarios. Summary of the Invention

[0005] The purpose of the present invention is to provide an industrial assembly action recognition and error correction training method based on deep learning to solve the problems mentioned in the above background technology, such as rough modeling of existing assembly actions, difficulty in identifying and classifying complex assembly actions, and a single assembly error feedback mechanism.

[0006] To achieve the above objectives, the present invention provides an industrial assembly action recognition and error correction training method based on deep learning, the method is as follows:

[0007] Step S1: deploying a multimodal sensing and acquisition system, setting up a central processing host and a simulation feedback terminal, and coordinating the multimodal sensing and acquisition system to collect raw multimodal data during the manual assembly process through a unified clock synchronization module to ensure time synchronization and coordinate transformation of multi-source heterogeneous data;

[0008] Step S2: The assembly operation data of senior assembly experts are recorded through a multimodal sensing and acquisition system, a standard assembly database is constructed, and a sample classification strategy is implemented. The standard assembly database is divided into two categories based on the quality of assembly actions: standard samples and error samples, corresponding to standard assembly operation data and error assembly operation data respectively;

[0009] Step S3: Using virtual motion sample synthesis and data augmentation technology, a virtual assembly scene is built in a 3D simulation platform. A parametric digital human body model is introduced, random perturbation functions are applied to skeletal motion parameters and tool operation parameters, and environmental interference factors in the industrial field are simulated to generate large-scale virtual samples.

[0010] Step S4: Based on the standard assembly database and the virtual sample, a deep learning model is constructed and trained. After normalizing the original multimodal data using the deep learning model, skeletal motion features, trajectory evolution features, and mechanical signal features are extracted and fused into multimodal fusion features. Based on the multimodal fusion features, assembly action recognition, anomaly detection, and keyframe positioning are performed to obtain an assembly action determination result.

[0011] Step S5: Based on the assembly action judgment results, an assembly quality evaluation mechanism is established, assembly quality evaluation indicators are defined and calculated, and an assembly quality scoring function is constructed to evaluate the assembly action judgment results, thereby obtaining a comprehensive quality score for the assembly operator when performing the assembly operation, and a feedback plan for improving the assembly quality is obtained based on the comprehensive quality score.

[0012] Based on the above scheme, the parameterized digital human body model is introduced to apply a random perturbation function to the skeletal motion parameters and tool operation parameters, specifically:

[0013] A parametrically customized digital human model is introduced into the virtual assembly scene as the main body of the assembly operation. The digital human model should meet the following basic requirements: the skeletal hierarchy is the same as that of a real person, the joint degrees of freedom match, the kinematic / dynamic characteristics are consistent with the human body mechanism, and the appearance and clothing can be customized;

[0014] The typical assembly movements are collected in three dimensions and processed into a data format suitable for the virtual environment, and the collected typical assembly movement data are combined with the digital human body model in the virtual environment;

[0015] Through random perturbation functions, variation samples are synthesized based on existing typical assembly actions.

[0016] Based on the above scheme, the synthetic variation samples are to introduce random perturbations to map the original typical assembly actions into a cluster of similar but not identical variation samples, specifically including:

[0017] Skeletal motion parameters of typical assembly actions of assembly operators and tool operating parameters of assembly tools Apply random perturbation functions separately , get:

[0018]

[0019]

[0020] in , is the covariance matrix of the random perturbation amplitude of the skeletal motion parameters and the tool operation parameters, which is reasonably set by simulating the execution error of the real assembly action. The time of data collection.

[0021] Based on the above scheme, the environmental interference factors of the simulated industrial site are used to generate large-scale virtual samples, specifically:

[0022] Simulating various environmental interference factors of an industrial site on the change sample to obtain the virtual sample;

[0023] The environmental interference factors include: partial occlusion, ambient light changes, and camera viewing angle deviation;

[0024] For local occlusion, we simulate it by adding movable obstacles in the virtual environment, including robotic arms and assembly lines; for changes in ambient lighting, we simulate it by adjusting the point light source or surface light source parameters of the virtual environment; the camera perspective offset is simulated by adjusting the external parameters of each virtual camera. Add noise to simulate based on ,in and For noise.

[0025] Based on the above solution, the normalization process includes:

[0026] The input module of the deep learning model takes the original multimodal data as input samples Perform normalization to obtain a normalized skeleton sequence , normalized tool trajectory sequence and normalized mechanical signal , the input sample ;in 、 、 They correspond respectively to the three-dimensional bone data, assembly tool trajectory and mechanical signal acquired by the RGB-D camera in the multimodal perception acquisition system, Indicates the The action category to which the first input sample belongs when it is a training sample, or The input sample is the action category to be predicted when the test sample is is the total number of input samples;

[0027] The normalization of the three-dimensional skeleton data is specifically as follows: joints in The three-dimensional coordinates of the moment are , then the normalized coordinates The calculation is as follows: ,in is the number of skeletal joints, is the length of the time series, and Respectively All joint 3D coordinates, labels, The mean and standard deviation of the input samples are used to convert the total number of frames of coordinate features to the origin and normalize the dimensions to the unit scale; for three-dimensional bone data The three-dimensional coordinates of all joints in the above normalization are normalized to obtain the normalized bone sequence .

[0028] Based on the above scheme, the deep learning feature extraction module contains three parallel sub-networks: skeleton sub-network, trajectory sub-network, and mechanical sub-network, which are responsible for extracting skeleton motion features, trajectory evolution features, and mechanical signal features respectively;

[0029] For the extraction of the skeleton action features, a sequence learning paradigm based on graph convolutional neural network is used as the skeleton sub-network to normalize the skeleton sequence As input, construct an undirected bone topology graph , where the node set correspond Skeletal joints, edge sets Represents the physical connection relationship between bones; based on undirected bone topology graph , using multi-level GCN to learn the motion correlation between different skeletal joints and model the temporal evolution of skeletal motion, extracting the skeletal motion features ,in is the learned skeleton feature dimension, is the time step.

[0030] Based on the above scheme, the long short-term memory network is used as the trajectory sub-network to normalize the tool trajectory sequence. Modeling the spatiotemporal dependencies of 、 Feeding into a multi-layer stacked bidirectional long short-term memory network, the temporal semantic information is adaptively aggregated through a gating mechanism to extract the trajectory evolution features of the deep representation ,in As the trajectory feature dimension, this sub-network can effectively capture the key action segments and semantic transformations in assembly tool operations.

[0031] Based on the above scheme, based on the normalized mechanical signal , using a one-dimensional convolutional neural network as a mechanical sub-network to extract mechanical features, including: based on the translation invariance of the one-dimensional convolutional neural network in the time dimension, scanning the input normalized mechanical signal through a multi-scale convolution kernel , automatically mine mechanical change patterns at different time scales and map them into compact mechanical signal features ,in is the mechanical characteristic dimension.

[0032] Based on the above scheme, the assembly quality evaluation indicators include: trajectory consistency, posture offset, and mechanical anomaly index;

[0033] The trajectory consistency is used to measure the deviation between the current assembly operator's tool operation trajectory and the standard trajectory. The trajectory consistency is calculated based on the trajectory sequence of the current input sample and the average trajectory of the standard sample. as follows:

[0034]

[0035] in is the total number of frames of the current input sample trajectory, Respectively represent Tool position and quaternion pose for the frame; , The standard sample trajectory is Mean statistics of the frame; Represents searching for the best time alignment between two trajectories To offset the difference in the start and end times of the action, is the weight balance coefficient of position and posture.

[0036] Based on the above scheme, the assembly quality scoring function is as follows:

[0037]

[0038] in is the posture offset, which is used to measure the degree of standardization of human posture; It is the mechanical anomaly index, which is used to measure the degree of abnormality of the force / torque signal during the assembly process; are the weight coefficients of the three assembly quality evaluation indicators, , is the abnormal penalty coefficient; function If the input condition is true, it takes 1, otherwise it takes 0; Indicates that AssemNet detects an error in the current action, and the overall quality score will be penalized.

[0039] The present invention has the following advantages and effects compared to the prior art:

[0040] (1) Introducing a multimodal perception system and clock synchronization module to achieve spatiotemporal synchronization and high-precision data acquisition during the assembly process, which can accurately capture skeletal movements, tool trajectories, and mechanical signals throughout the entire process, enabling high-precision dynamic modeling;

[0041] (2) Design a multimodal deep neural network that integrates graph convolutional networks, LSTM, and one-dimensional CNN. Combine real and virtual samples to build a training set to achieve action classification, anomaly detection, and key frame positioning. It can accurately identify assembly action types and erroneous operations, and locate the time and type of assembly anomalies.

[0042] (3) Build an assembly quality assessment mechanism, provide feedback solutions based on trajectory consistency, posture offset, and mechanical anomaly index scores, and give differentiated feedback suggestions based on individual assembly levels, thus forming a closed loop of assessment-improvement-reassessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for the specific embodiments or the description of the prior art. Similar elements or parts are generally identified by similar reference numerals throughout the drawings. Elements or parts in the drawings are not necessarily drawn to scale.

[0044] Figure 1 This is a flowchart of a deep learning-based industrial assembly action recognition and error correction training method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The example implementation methods can be implemented in various forms and should not be understood as being limited to the examples described herein. On the contrary, these implementation methods are provided to make the present invention more comprehensive and complete, and to fully convey the concepts of the example implementation methods to those skilled in the art.

[0046] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, it will be appreciated by those skilled in the art that the technical solutions of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0047] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0048] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0049] The present invention will be described in detail below with reference to specific embodiments:

[0050] As attached Figure 1 As shown, embodiment 1 of the present invention provides an industrial assembly action recognition and error correction training method based on deep learning, and the specific steps of the method are as follows:

[0051] Step S1: Deploy a multimodal perception and acquisition system, including installing an RGB-D camera system, an inertial measurement unit, and a force / torque sensor in an industrial assembly environment, and setting up a central processing host and a simulation feedback terminal. The multimodal perception and acquisition system is coordinated by a unified clock synchronization module to collect raw multimodal data during the manual assembly process, ensuring time synchronization and coordinate transformation of multi-source heterogeneous data.

[0052] Step S2: The assembly operation data of senior assembly experts are recorded through a multimodal sensing and acquisition system, a standard assembly database is constructed, and a sample classification strategy is implemented. The standard assembly database is divided into two categories based on the quality of assembly actions: standard samples and error samples, corresponding to standard assembly operation data and error assembly operation data respectively;

[0053] Step S3: Using virtual motion sample synthesis and data augmentation technology, a virtual assembly scene is built in a 3D simulation platform. A parametric digital human body model is introduced, random perturbation functions are applied to skeletal motion parameters and tool operation parameters, and environmental interference factors in the industrial field are simulated to generate large-scale virtual samples.

[0054] Step S4: Based on the standard assembly database and the virtual sample, a deep learning model is constructed and trained. After normalizing the original multimodal data using the deep learning model, skeletal motion features, trajectory evolution features, and mechanical signal features are extracted and fused into multimodal fusion features. Based on the multimodal fusion features, assembly action recognition, anomaly detection, and keyframe positioning are performed to obtain an assembly action determination result.

[0055] Step S5: Based on the assembly action judgment result, an assembly quality evaluation mechanism is established, and assembly quality evaluation indicators including trajectory consistency, posture offset, and mechanical anomaly index are defined and calculated. A comprehensive quality score is obtained through an assembly quality comprehensive scoring function. According to the numerical range of the comprehensive quality score, the assembly quality is divided into four levels, and a multi-level feedback scheme is given to the assembly quality of different quality levels, forming a dynamic optimization closed loop of evaluation, prompting, improvement, and re-evaluation to achieve lean control of the assembly process.

[0056] Preferably, in step S1, the multimodal perception acquisition system includes the following devices: an RGB-D camera system, an inertial measurement unit, a force / torque sensor, a central processing host, and a simulation feedback terminal.

[0057] Specifically, the RGB-D camera system is used to capture the three-dimensional skeleton data and assembly tool trajectory of the assembly operator during the assembly operation, and is composed of several RGB-D cameras. The shooting angles of each camera in the RGB-D camera system must be carefully designed to achieve full coverage of the assembly operation area. In this embodiment, the RGB-D camera system must be placed at least in two positions, directly in front of and directly above the operation area. Indicates the number of RGB-D cameras. The camera is recorded as , its external parameters in the world coordinate system, namely position and posture, are expressed as , Indicates the The rotation matrix of an RGB-D camera, where refers to the special orthogonal group, which represents the set of rotation matrices in three-dimensional space; Indicates the The translation vector of the RGB-D camera, and All RGB-D camera external parameters can be obtained and optimized through a high-precision visual calibration system such as the Optitrack system.

[0058] Several inertial measurement units (IMUs) are attached to key areas of the assembly operator's body, including the wrists, elbows, and key assembly tools. Each IMU measures three-axis angular velocity and acceleration data in real time. To minimize accuracy loss caused by drift, the central processing unit is designed with a unified clock synchronization module to coordinate the sampling frequencies of the RGB-D camera system and the IMUs. For example, the sampling frequency of both sensors can be set to 60Hz.

[0059] During the assembly process, the collection of mechanical signals is also crucial. In this embodiment, force / torque sensors are installed on major assembly tools, such as electric screwdrivers, presses, welding guns, etc., as well as on work surfaces. The assembly actions of assembly operators usually form typical time-series patterns of force and torque, and their data can be used as an important basis for judging whether the actions are standardized. Specifically, the layout position of the force / torque sensor needs to be designed according to the process flow of the specific assembly task, and the minimum range of the sensor needs to cover the mechanical variation range of the assembly operation, and its sampling frequency is generally not less than 100 Hz. The force components of the force / torque sensor on the x, y, and z coordinate axes are respectively recorded as , , , the torque component is recorded as , , .

[0060] The core hardware of the multimodal perception and acquisition system is the central processing unit (CPU), equipped with a high-performance CPU and GPU, and storing massive amounts of standard and field-measured data. The CPU is connected to the RGB-D camera system, inertial measurement unit, and force / torque sensor via high-speed Ethernet. The CPU receives heterogeneous data streams and performs preprocessing on them, including time synchronization, coordinate transformation, and feature extraction. This preprocessed, fused data is then fed into the system's designed deep neural network, AssemNet, for inference and judgment. The CPU's computing power must meet the requirements of real-time inference.

[0061] The human-machine interaction interface is implemented by the simulation feedback terminal, which features a large touchscreen display and is equipped with a voice prompt module. The simulation feedback terminal receives the central processing unit's judgment of the assembly operator's assembly actions, presents the current assembly action execution status in a 3D visualization, and highlights identified errors in red. Furthermore, the simulation feedback terminal can provide voice correction prompts to guide the operator in correcting their movements.

[0062] Preferably, in step S2, in order to train a deep neural network model that can accurately identify assembly action errors, it is necessary to construct a high-quality standard assembly database. The sample source of the standard assembly database is mainly senior assembly experts within the company. Through the multimodal perception and acquisition system described in step S1, the assembly operation data of the senior assembly experts on the production line are recorded, specifically including: three-dimensional skeleton posture sequence captured by RGB-D camera, namely three-dimensional skeleton data, continuous trajectory of the end posture of the assembly tool, namely assembly tool trajectory, three-axis angular velocity and acceleration data of key parts of the body of senior assembly experts measured in real time by inertial measurement unit, mechanical signals collected by multi-channel force / torque sensors; and also including manually annotated action stage information and key frame error type labels.

[0063] The collected assembly operation data needs to be preprocessed to form a unified data organization format; taking the kth assembly sample as an example, the data contained therein can be represented as a triple ,in Indicates the The set of three-dimensional joint positions of the bones in the assembly sample is specifically defined as , , is the total number of joints in the human body; For the The trajectory sequence of the assembly tools in the assembly samples is recorded as , where express The tool end position at the moment, Indicates the posture of the tool end, H refers to the quaternion space, is the total number of frames of the k-th sample; For the The set of multi-channel mechanical data corresponding to the assembly samples is defined as , , ,in and They are The three-dimensional force and torque vectors measured at all times; all assembly samples are organized into a unified data structure in the form of this triplet data.

[0064] Preferably, the sample grading strategy is to implement graded management on the assembly samples in the standard assembly database according to the quality of assembly actions; specifically, the sample quality is judged by the experts who make subjective scores on the coherence and standardization of the action execution, and then combine the objective quality test results of the assembled parts to score and sort the assembly samples. The assembly samples in the standard assembly database are divided into two categories. The first category of samples It is a high-quality standard sample with smooth movement, correct operation steps and stable assembly results; the second type of sample These are error samples that include incorrect operations. These errors mainly come from deliberately set counterexamples and occasional mistakes made by experts in actual operations. Error types can be divided into several subcategories according to differences in assembly processes, such as omissions, reversed order, improper force / angle, etc.

[0065] Preferably, in step S3, in order to further improve the robustness and generalization ability of the deep learning model in subsequent steps, virtual action sample synthesis and data enhancement technology are introduced based on the standard assembly database, and a realistic virtual assembly environment is constructed in a high-fidelity three-dimensional simulation engine. Large-scale virtual samples are automatically generated as a supplementary data set for deep learning model training through digital human body models and programmable assembly objects.

[0066] Specifically, a virtual scene that is consistent with the actual assembly workshop layout is built on mainstream 3D simulation platforms (such as Unity3D and Unreal Engine). The virtual scene includes key elements such as assembly stations, workpieces, tooling, and sensor layout, and its geometric parameters are completely consistent with the real environment.

[0067] Furthermore, a parametrically customized digital human model is introduced into the virtual assembly scene as the main body of the assembly operation; the digital human model should meet the following basic requirements: the skeletal hierarchy is the same as that of a real person, the joint degrees of freedom match, the kinematic / dynamic characteristics are consistent with the human body mechanism, and the appearance and clothing can be customized;

[0068] Furthermore, typical assembly movements are captured in three dimensions and processed into a data format suitable for the virtual environment. Three-dimensional capture methods include optical motion capture systems (OMC) and inertial motion capture systems (IMU-based motion capture). The former uses infrared camera arrays to perform high-precision three-dimensional tracking of reflective markers attached to key parts of the human body, while the latter uses inertial measurement units (IMUs) such as magnetic fields and gyroscopes to reconstruct the relative motion of human limbs. By combining the captured typical assembly movement data with a digital human model in the virtual environment, the typical assembly movement operations of the assembly worker can be reproduced. Furthermore, a 3D model of the assembly object, or workpiece, must also be imported into the virtual environment to support visual assembly simulation. The workpiece model can often be directly derived from the CAD model from the product design phase.

[0069] Preferably, in order to make the virtual samples cover more changes in assembly action patterns, while deterministically reconstructing typical assembly actions, a random perturbation function is used to synthesize variation samples based on existing typical assembly actions;

[0070] Specifically, the skeletal motion parameters of the assembly workers and tool operating parameters for assembly tools Apply random perturbation functions separately , :

[0071]

[0072]

[0073] in , is the covariance matrix of the random perturbation amplitude of the skeletal motion parameters and the tool operation parameters, which is reasonably set by simulating the execution error of the real assembly action. is the data collection moment. After introducing random perturbations, the original typical assembly actions are mapped into a cluster of similar but not identical variation samples, which greatly expands the sample space.

[0074] Preferably, in addition to the changes in the assembly action itself, various environmental interference factors of the industrial site need to be simulated for the change sample to obtain the virtual sample. Common environmental interference factors include: partial occlusion, ambient light changes, camera angle deviation, etc. For partial occlusion, it is simulated by adding movable obstacles in the virtual environment, including robotic arms and assembly lines; for ambient light changes, it is simulated by adjusting the point light source / surface light source parameters of the virtual environment; the camera angle deviation can be simulated by adjusting the external parameters of each virtual camera. Add noise on the basis of ,in and For noise.

[0075] In summary, by controlling various factors in a virtual environment, we generate a large number of highly realistic virtual samples of assembly actions. These samples complement the limited number of real samples in the standard assembly database and together form the assembly action training set used to train the deep learning model. After all virtual samples are uniformly converted into a data format consistent with real samples, they are fed into the deep learning model for end-to-end training. The inclusion of virtual samples helps improve the deep learning model's robustness to complex working conditions, thereby accurately understanding the diverse human-object-environment interactions in real assembly processes.

[0076] Preferably, in step S4, based on the standard assembly database and the virtual sample, a deep learning model is constructed and trained, the original multimodal data is normalized by the deep learning model to obtain a normalized skeleton sequence, a normalized tool trajectory sequence and a normalized mechanical signal, and the skeleton action features, trajectory evolution features and mechanical signal features are respectively extracted by a graph convolutional neural network, a long short-term memory network and a one-dimensional convolutional neural network, and then aggregated into multimodal fusion features using an attention fusion module. Based on the multimodal fusion features, assembly action recognition is performed to output the action category, anomaly detection is performed to determine whether there is an erroneous operation, and key frame positioning is performed to determine the position at the critical moment. These three outputs together constitute the assembly action judgment result;

[0077] Preferably, a deep learning model, AssemNet, is constructed and trained specifically for industrial assembly action recognition and analysis. Compared with traditional action recognition networks, AssemNet has made targeted improvements in perceptual data fusion, spatiotemporal feature extraction, and multi-task learning to adapt to the complexity and variability of assembly scenarios.

[0078] Specifically, during the training process of the deep learning model, differentiated weight assignment strategies are adopted for samples of different categories, that is, different weight assignment strategies are adopted for the first category samples A1 and the second category samples A2 in step S2 and the virtual samples generated in step S3; The deep learning model calculates its cross entropy loss function as follows: ,in Multimodal assembly features for this sample, is the corresponding action category. In the standard sample set In the sample weight Take a larger value (such as 1.0); for the standard sample set The error sample in Take a smaller value (such as 0.1~0.5), for the virtual sample set {A3}, the sample weight A moderate value (such as 0.6~0.8) is chosen to balance authenticity and diversity. The total loss function of the deep learning model is defined as the weighted average of the losses of each assembly sample:

[0079]

[0080] By setting the weight parameters reasonably, the contribution of standard samples, error samples and virtual samples to the deep learning model can be balanced to a certain extent, ensuring that the model mainly learns the standard action mode from the standard samples, while enhancing the environmental adaptability through virtual samples and improving the error recognition ability through error samples; at different stages of model training, the weights Dynamically adjusts to optimize training results.

[0081] Specifically, the network structure and key modules of AssemNet are described in detail below;

[0082] Preferably, the input data of AssemNet comes from the original multimodal data described in step S1. The data of the input samples can be expressed as ;in 、 、 They correspond to the three-dimensional bone data, assembly tool trajectory and mechanical signals obtained by the RGB-D camera, respectively. Indicates the The action category to which the first input sample belongs when it is a training sample, or The input sample is the action category to be predicted when the test sample is is the total number of input samples.

[0083] Since the raw data of different modalities have significant differences in spatiotemporal resolution, numerical range, etc., it is not appropriate to directly splice them as the input features of the network. In order to achieve the effective fusion of high-level semantic information, the input module of AssemNet processes the input samples. Normalization is performed; 3D bone data For example, let joints in The three-dimensional coordinates of the moment are , then the normalized coordinates The calculation is as follows:

[0084]

[0085] in is the number of skeletal joints, is the length of the time series, and Respectively The three-dimensional coordinates of all joints in the sample The mean and standard deviation of the input samples are used to convert the total number of frames of coordinate features to the origin and normalize the dimensions to the unit scale; for three-dimensional bone data The three-dimensional coordinates of all joints in the above normalization are normalized to obtain the normalized bone sequence Similarly, for the assembly tool trajectory and mechanical signals , respectively calculate the normalized results, namely the normalized tool trajectory sequence Normalized mechanical signal .

[0086] Preferably, the feature extraction module of AssemNet contains three parallel sub-networks, which are responsible for high-level semantic representation learning, i.e., feature extraction, of skeletal motion features, trajectory evolution features, and mechanical signal features respectively; the design of each sub-network takes into account the intrinsic structural characteristics of different modal data.

[0087] Specifically, for the extraction of skeleton action features, AssemNet adopts a sequence learning paradigm based on graph convolutional neural network as the skeleton subnetwork, which is based on the normalized skeleton sequence. As input, construct an undirected bone topology graph , where the node set correspond Skeletal joints, edge sets Represents the physical connection relationship between bones, that is, the bone topology. Based on the undirected bone topology graph , using multi-level GCN to learn the motion correlation between different skeletal joints and model the temporal evolution of skeletal motion, extracting the skeletal motion features ,in is the learned skeleton feature dimension, is the time step; the skeletal motion feature integrates the motion correlation information between skeletal joints, such as the collaborative relationship between joints, and the dynamic change characteristics of skeletal motion in the time dimension, such as the duration and rhythm of the motion.

[0088] Specifically, a long short-term memory network is used as the trajectory sub-network to normalize the tool trajectory sequence. Modeling the spatiotemporal dependencies of 、 Feeding into a multi-layer stacked bidirectional long short-term memory network, the temporal semantic information is adaptively aggregated through a gating mechanism to extract the trajectory evolution features of the deep representation ,in is the trajectory feature dimension. This sub-network can effectively capture the key action segments and semantic transformations in assembly tool operations.

[0089] Specifically, based on the normalized mechanical signal , using a one-dimensional convolutional neural network as a mechanical sub-network to extract mechanical features, including: based on the translation invariance of the one-dimensional convolutional neural network in the time dimension, scanning the input normalized mechanical signal through a multi-scale convolution kernel , automatically mine mechanical change patterns at different time scales and map them into compact mechanical signal features ,in is the mechanical characteristic dimension.

[0090] Furthermore, the AssemNet includes an attention fusion module, which is used to adaptively aggregate features from three channels. 、 、 The semantic information of different modalities is used to obtain multimodal fusion features ;

[0091] Specifically, let , respectively represent the query matrix, key matrix and value matrix that map skeletal motion features, trajectory evolution features and mechanical signal features to the attention calculation space, where Is the corresponding feature transformation function, then the multimodal fusion feature The calculation formula is:

[0092]

[0093]

[0094] in is the scaling factor, Represents the tensor splicing operation; after attention fusion, , contains cross-modal interaction information that is crucial for understanding assembly actions, where is the multimodal fusion feature dimension.

[0095] Furthermore, AssemNet includes a judgment output module; based on the multimodal fusion feature ,Through the judgment output module, multiple downstream tasks such as assembly action recognition, anomaly detection, keyframe positioning, etc.

[0096] Specifically, for assembly action recognition, the global average pooling layer is used to transform Aggregated into semantic vectors in the time dimension , and then pass the fully connected layer to Mapping to action category probability distribution ,in The number of preset assembly action categories:

[0097]

[0098] in, are the weight and bias parameters of the fully connected layer respectively;

[0099] By minimizing and the true category label Cross entropy loss function between Optimizing action classification performance: .

[0100] Furthermore, the anomaly detection task is based on , a binary classification fully connected layer is used to determine whether the current action segment contains an incorrect operation;

[0101] Specifically, the two-class fully connected layer will Mapping to abnormal probability , and combined with the binary cross entropy loss of standard samples and error samples To train:

[0102]

[0103]

[0104] in is the parameter of the binary classification fully connected layer, is the actual abnormal label.

[0105] Furthermore, the keyframe positioning is to accurately locate the key moments in the assembly action, including the tool alignment moment and the tightening completion moment, through the keyframe regression branch of AssemNet;

[0106] Specifically, the key frame regression branch is As input, directly predict through one-dimensional convolution and full connection operation The relative position of each key moment in the action sequence , and combined with the squared error loss of relative position To optimize:

[0107]

[0108] in For the The relative position of a real critical moment.

[0109] Furthermore, the model-based processing results of the three tasks of assembly action recognition, anomaly detection and key frame positioning together constitute the assembly action judgment result; that is, the assembly action judgment result includes: action category recognition result, abnormal behavior detection result and key moment positioning result.

[0110] During model training, the performance of the three AssemNet tasks is improved simultaneously by optimizing the following multi-task loss function:

[0111]

[0112] in represents the action classification loss, represents the anomaly detection loss, represents the keyframe positioning loss, 、 is a balancing factor. It should be noted that AssemNet's multi-task learning paradigm helps improve the model's generalization capabilities, enabling it to adapt to the requirements of different assembly processes and action templates. By sharing underlying feature representations and focusing on different task objectives, the model can learn richer representations of assembly actions, thereby improving overall performance.

[0113] Preferably, in step S5, based on the assembly action determination result, an assembly quality evaluation mechanism is established, assembly quality evaluation indicators are defined and calculated, and the assembly operation quality of the assembly operator is evaluated through an assembly quality scoring function and improvement feedback is given.

[0114] Specifically, the assembly quality evaluation indicators are defined and calculated. The assembly quality evaluation indicators include trajectory consistency, posture offset, and mechanical anomaly index, as follows:

[0115] (1) The trajectory consistency is used to measure the degree of deviation between the current assembly worker's tool operation trajectory and the standard trajectory;

[0116] Calculate trajectory consistency based on the trajectory sequence of the current input sample and the average trajectory of the standard sample as follows:

[0117]

[0118] in is the total number of frames of the current input sample trajectory, Respectively represent Tool position and quaternion pose for the frame; , The standard sample trajectory is Mean statistics of the frame; Represents searching for the best time alignment between two trajectories To offset the difference in the start and end times of the action, is the weight balance coefficient of position and posture.

[0119] (2) The posture offset is used to measure the degree of standardization of human posture;

[0120] Based on the skeleton motion features, calculate the posture offset as follows:

[0121]

[0122] in The current input sample is Skeletal motion features of the frame, The standard sample set is The mean of the skeleton motion features at each frame.

[0123] (3) the mechanical anomaly index, which is used to measure the degree of anomaly of the force / torque signal during the assembly process;

[0124] Based on multimodal fusion features , calculate the Mahalanobis distance between the current input sample and the standard sample set, and obtain the mechanical anomaly index :

[0125]

[0126] in Represents the multimodal fusion features of the current input sample in the i-th frame, Represents the standard sample set Mean and covariance matrices of mechanical-related components in frame multimodal fusion features.

[0127] Based on the above assembly quality evaluation indicators, the assembly quality scoring function is defined ,as follows:

[0128]

[0129] in are the weight coefficients of the three assembly quality evaluation indicators, , is the abnormal penalty coefficient; function If the input condition is true, it takes 1, otherwise it takes 0; Indicates that AssemNet detected an error in the current action, and the overall score will be penalized.

[0130] Furthermore, based on the comprehensive quality score The numerical range of the value generates a multi-level feedback plan to guide assembly workers to standardize operations and improve quality;

[0131] Specifically, the comprehensive quality score The numerical range is divided into four levels, and corresponding feedback schemes are given:

[0132] (1) : Indicates that the assembly quality is unqualified, and the system generates a warning prompt, requiring immediate cessation of assembly operations and retraining of standard assembly actions;

[0133] (2) : This range indicates that the assembly quality is barely acceptable. The system will highlight the lowest-scoring of the three assembly quality evaluation indicators and guide workers to practice the corresponding operation points more.

[0134] (3) : This range indicates good assembly quality, and the system provides positive feedback while also suggesting improvements for some details that still have room for improvement;

[0135] (4) : This interval indicates that the assembly quality is excellent. The system gives high praise and summarizes the operational highlights of the input sample and incorporates it into the standard assembly database.

[0136] It's important to note that this multi-level feedback scheme provides targeted guidance for different quality levels, helping to motivate workers to learn and gradually improve overall assembly performance. The feedback should be approachable, concise, and motivating, encouraging assembly workers to repeat their training in a humane manner.

[0137] In summary, the assembly quality assessment mechanism constructed in this embodiment quantitatively characterizes the standardization of the assembly process from multiple dimensions and provides targeted, hierarchical feedback based on the comprehensive quality score, forming a dynamic optimization closed loop of assessment, prompting, improvement, and reassessment. This closed loop ensures continuous improvement in quality evaluation and behavioral reinforcement, ultimately achieving lean management and control.

[0138] In this example, a method for industrial assembly training that integrates multimodal perception, virtual sample augmentation, and deep learning-based reasoning is constructed. First, a perception system consisting of an RGB-D camera, an inertial measurement unit, and a force / torque sensor is deployed, combined with a unified clock synchronization mechanism to collect and synchronize multi-source heterogeneous data, enabling dynamic modeling of the entire assembly process for operators. Second, a virtual assembly environment is constructed based on a three-dimensional simulation platform. A perturbed parameterized digital human model is introduced, random perturbations are applied to skeletal and tool parameters, and environmental interference is simulated to generate a rich set of virtual assembly samples. Third, a deep learning model, AssemNet, is designed to extract skeletal motion features, trajectory evolution features, and mechanical signal features. Multimodal fusion is achieved through an attention mechanism to complete motion recognition, anomaly detection, and keyframe location. Finally, an assembly quality scoring function and a four-level feedback mechanism are constructed based on trajectory consistency, posture offset, and mechanical anomaly index to enable targeted error correction and retraining. By combining multidimensional perception with intelligent evaluation, the accuracy, intelligence level, and human-computer interaction experience of industrial assembly training are significantly improved, demonstrating significant practical value.

[0139] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present invention are indicated by the claims. It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A deep learning-based industrial assembly action recognition and error correction training method, characterized in that: include: Step S1: deploying a multimodal sensing and acquisition system, setting up a central processing host and a simulation feedback terminal, and coordinating the multimodal sensing and acquisition system to collect raw multimodal data during the manual assembly process through a unified clock synchronization module to ensure time synchronization and coordinate transformation of multi-source heterogeneous data; Step S2: The assembly operation data of senior assembly experts are recorded through a multimodal sensing and acquisition system, a standard assembly database is constructed, and a sample classification strategy is implemented. The standard assembly database is divided into two categories based on the quality of assembly actions: standard samples and error samples, corresponding to standard assembly operation data and error assembly operation data respectively; Step S3: Using virtual motion sample synthesis and data augmentation technology, a virtual assembly scene is built in a 3D simulation platform. A parametric digital human body model is introduced, random perturbation functions are applied to skeletal motion parameters and tool operation parameters, and environmental interference factors in the industrial field are simulated to generate large-scale virtual samples. Step S4: Based on the standard assembly database and the virtual sample, a deep learning model is constructed and trained. After normalizing the original multimodal data using the deep learning model, skeletal motion features, trajectory evolution features, and mechanical signal features are extracted and fused into multimodal fusion features. Based on the multimodal fusion features, assembly action recognition, anomaly detection, and keyframe positioning are performed to obtain an assembly action determination result. Step S5: Based on the assembly action judgment results, an assembly quality evaluation mechanism is established, assembly quality evaluation indicators are defined and calculated, and an assembly quality scoring function is constructed to evaluate the assembly action judgment results, thereby obtaining a comprehensive quality score for the assembly operator when performing the assembly operation, and a feedback plan for improving the assembly quality is obtained based on the comprehensive quality score.

2. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 1 is characterized in that: The parameterized digital human body model is introduced, and a random perturbation function is applied to the skeletal motion parameters and tool operation parameters, specifically: A parametrically customized digital human model is introduced into the virtual assembly scene as the main body of the assembly operation. The digital human model should meet the following basic requirements: the skeletal hierarchy is the same as that of a real person, the joint degrees of freedom match, the kinematic / dynamic characteristics are consistent with the human body mechanism, and the appearance and clothing can be customized; The typical assembly movements are collected in three dimensions and processed into a data format suitable for the virtual environment, and the collected typical assembly movement data are combined with the digital human body model in the virtual environment; Through random perturbation functions, variation samples are synthesized based on existing typical assembly actions.

3. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 2 is characterized in that: The synthetic variation samples are generated by introducing random perturbations to map the original typical assembly actions into a cluster of similar but not identical variation samples, specifically including: Skeletal motion parameters of typical assembly actions of assembly operators and tool operating parameters of assembly tools Apply random perturbation functions separately , get: , ,in , is the covariance matrix of the random perturbation amplitude of the skeletal motion parameters and the tool operation parameters, which is reasonably set by simulating the execution error of the real assembly action. The time of data collection.

4. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 3 is characterized in that: The environmental interference factors of the simulated industrial site are used to generate large-scale virtual samples, specifically: Simulating various environmental interference factors of an industrial site on the change sample to obtain the virtual sample; The environmental interference factors include: partial occlusion, ambient light changes, and camera viewing angle deviation; For local occlusion, we simulate it by adding movable obstacles in the virtual environment, including robotic arms and assembly lines; for changes in ambient lighting, we simulate it by adjusting the point light source or surface light source parameters of the virtual environment; the camera perspective offset is simulated by adjusting the external parameters of each virtual camera. Add noise to simulate based on ,in and For noise.

5. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 1, characterized in that: The normalization process includes: The input module of the deep learning model takes the original multimodal data as input samples Perform normalization to obtain a normalized skeleton sequence , normalized tool trajectory sequence and normalized mechanical signal , the input sample ;in 、 、 They correspond respectively to the three-dimensional bone data, assembly tool trajectory and mechanical signal acquired by the RGB-D camera in the multimodal perception acquisition system, Indicates the The action category to which the first input sample belongs when it is a training sample, or The input sample is the action category to be predicted when the test sample is is the total number of input samples; The normalization of the three-dimensional skeleton data is specifically as follows: joints in The three-dimensional coordinates of the moment are , then the normalized coordinates The calculation is as follows: ,in is the number of skeletal joints, is the length of the time series, and Respectively All joint 3D coordinates, labels, The mean and standard deviation of the input samples are used to convert the total number of frames of coordinate features to the origin and normalize the dimensions to the unit scale; for three-dimensional bone data The three-dimensional coordinates of all joints in the above normalization are normalized to obtain the normalized bone sequence .

6. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 1, characterized in that: The deep learning feature extraction module contains three parallel sub-networks: the skeleton sub-network, the trajectory sub-network, and the mechanical sub-network, which are responsible for extracting skeleton motion features, trajectory evolution features, and mechanical signal features respectively; For the extraction of the skeleton action features, a sequence learning paradigm based on graph convolutional neural network is used as the skeleton sub-network to normalize the skeleton sequence As input, construct an undirected bone topology graph , where the node set correspond Skeletal joints, edge sets Represents the physical connection relationship between bones; based on undirected bone topology graph , using multi-level GCN to learn the motion correlation between different skeletal joints and model the temporal evolution of skeletal motion, extracting the skeletal motion features ,in is the learned skeleton feature dimension, is the time step.

7. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 5, characterized in that: The long short-term memory network is used as the trajectory sub-network to normalize the tool trajectory sequence Modeling the spatiotemporal dependencies of 、 Feeding it into a multi-layer stacked bidirectional long short-term memory network, the gating mechanism adaptively aggregates temporal semantic information and extracts trajectory evolution features of deep representation. ,in As the trajectory feature dimension, this sub-network can effectively capture the key action segments and semantic transformations in assembly tool operations.

8. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 5, characterized in that: Based on the normalized mechanical signal , using a one-dimensional convolutional neural network as a mechanical sub-network to extract mechanical features, including: based on the translation invariance of the one-dimensional convolutional neural network in the time dimension, scanning the input normalized mechanical signal through a multi-scale convolution kernel , automatically mine mechanical change patterns at different time scales and map them into compact mechanical signal features ,in is the mechanical characteristic dimension.

9. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 6, characterized in that: The assembly quality evaluation indicators include: trajectory consistency, posture offset, and mechanical anomaly index; The trajectory consistency is used to measure the deviation between the current assembly operator's tool operation trajectory and the standard trajectory. The trajectory consistency is calculated based on the trajectory sequence of the current input sample and the average trajectory of the standard sample. as follows: ,in is the total number of frames of the current input sample trajectory, Respectively represent Tool position and quaternion pose for the frame; , The standard sample trajectory is Mean statistics of the frame; Represents searching for the best time alignment between two trajectories To offset the difference in the start and end times of the action, is the weight balance coefficient of position and posture.

10. The method for industrial assembly action recognition and error correction training based on deep learning according to claim 9, characterized in that: The assembly quality scoring function is as follows: ,in is the posture offset, which is used to measure the degree of standardization of human posture; It is the mechanical anomaly index, which is used to measure the degree of abnormality of the force / torque signal during the assembly process; are the weight coefficients of the three assembly quality evaluation indicators, , is the abnormal penalty coefficient; function If the input condition is true, it takes 1, otherwise it takes 0; Indicates that AssemNet detects an error in the current action, and the overall quality score will be penalized.

Images