A method for recognizing car assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model.

Abstract

This invention belongs to the field of automotive assembly time analysis technology, specifically involving a method for recognizing automotive assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model. The method categorizes automotive assembly actions into upper body actions and lower body actions. It collects sensor data on different actions performed by workers and builds a multi-channel 1DCNN recognition model. The upper body assembly action recognition model and the lower body assembly action recognition model are trained separately. Worker's normal working sensor data is acquired and input into the upper and lower body assembly action recognition models. The first and last columns are extracted from the predicted sequence data output by the upper and lower body assembly action recognition models, and the working time is calculated to complete the recognition of automotive assembly actions. This invention is applied to automotive assembly scenarios, achieving intelligent action segmentation and recognition, simplifying the time verification process, and improving the efficiency and automation of verification analysis.

Description

Technical Field

[0001] This invention belongs to the field of automotive assembly time analysis technology, specifically relating to a method for recognizing automotive assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model. Background Technology

[0002] In the current vehicle manufacturing industry, production operation time measurement generally employs one of three methods: direct observation, stopwatch, or standard analysis. The first two methods are typically used in scenarios where precise time measurement is less critical, while the standard analysis method is generally used for scenarios requiring highly accurate results. The standard analysis method involves segmenting and identifying all actions of a production operator according to a given standard, and then assigning time values ​​to these segmented actions according to the standard's defined time values. While the time results obtained using the standard analysis method are relatively accurate, it suffers from drawbacks such as slow analysis efficiency, difficulty in mastering the method, and inaccuracies in manually segmented actions. Summary of the Invention

[0003] To overcome the above problems, this invention provides a motion recognition and segmentation model and system based on motion capture equipment; applied to automobile assembly scenarios, it realizes intelligent motion segmentation and recognition, simplifies the time verification process, and improves the efficiency and automation of verification analysis.

[0004] A method for recognizing car assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model, specifically including the following:

[0005] Step 1: Divide the complete car assembly process into N types of actions, and then divide the N types of actions into two main categories: upper body actions and lower body actions.

[0006] Step 2: Wear sensors on different parts of the worker's body to collect data from each sensor when the worker performs upper body and lower body movements. The worker must repeat each movement M times, and the data collected by each sensor must be collected M times to obtain the raw data.

[0007] Step 3: Build a multi-channel 1D Convolutional Neural Network (1DCNN) recognition model; input the data collected during upper body movements from the raw data obtained in Step 2 into the multi-channel 1DCNN model for training to obtain an upper body assembly movement recognition model; input the data collected during lower body movements from the raw data obtained in Step 2 into the multi-channel 1DCNN model for training to obtain a lower body assembly movement recognition model; combine the upper body recognition model and the lower body recognition model into the final prediction model;

[0008] Step 4: Wear sensors on various parts of the worker's body to obtain data collected by the sensors during normal work.

[0009] Step 5: Input the sensor data collected in Step 4 into the upper body assembly action recognition model and the lower body assembly action recognition model trained in Step 3 respectively. The upper body assembly action recognition model and the lower body assembly action recognition model predict the action category of the input sensor data respectively.

[0010] Step 6: Extract the first and last columns, namely the frame number and action category, from the predicted sequence data output by the upper body assembly action recognition model and the lower body assembly action recognition model; then convert the extracted frame number into the working time of the corresponding action category; complete the recognition of car assembly actions.

[0011] The sensors worn by the workshop workers in steps two and four are...

[0012] PerceptionNeuronStudio sensor; the sensor collects human coordinate data in BVH format.

[0013] In step one, the complete car assembly actions are divided into the following 24 action categories: electric tapping, hammering, embedding, connecting, torque, temporary fixing, painting, twisting, pressing, adjusting, pasting, pushing, pulling, picking up, placing, single torque, rotating arm, kneeling, getting up, squatting, standing up, turning around, stepping, striding, and long-distance walking.

[0014] In step one, the 24 types of movements are divided into upper body movements and lower body movements. Among them, electric shock, knocking, embedding, connecting, torque, temporary fixing, painting, twisting, pressing, adjusting, pasting, pushing, pulling, pressing, picking up, placing, single torque, and rotating the arm are upper body movements, while kneeling, getting up, squatting, standing up, turning around, stepping, striding, and long-distance walking are lower body movements.

[0015] In steps two and four, the sensors are worn on 59 body parts of the workshop workers, namely: buttocks, right thigh, right calf, right foot, left thigh, left calf, left foot, lower spine, middle spine, upper spine, lower neck, upper neck, head, right shoulder, right upper arm, right forearm, right hand, right thumb base, right thumb middle finger, right thumb tip, right index finger metacarpal bone, right index finger base, right index finger middle finger, right index finger tip, right middle finger metacarpal bone, right middle finger base, right middle finger middle finger, right middle finger tip, right ring finger palm. Bone, base of right ring finger, middle finger of right ring finger, tip of right ring finger, metacarpal bone of right little finger, base of right little finger, middle finger of right little finger, tip of right little finger, left shoulder, left upper arm, left forearm, left hand, base of left thumb, middle finger of left thumb, tip of left thumb, metacarpal bone of left index finger, base of left index finger, middle finger of left index finger, tip of left index finger, metacarpal bone of left middle finger, base of left middle finger, middle finger of left middle finger, tip of left middle finger, metacarpal bone of left ring finger, base of left ring finger, middle finger of left ring finger, tip of left ring finger, metacarpal bone of left little finger, base of left little finger, middle finger of left little finger, tip of left little finger.

[0016] The structure of the multi-channel one-dimensional convolutional neural network (1DCNN) model constructed in step three is as follows:

[0017] The multi-channel one-dimensional convolutional neural network (1DCNN) model uses a multi-head model. Each frame of input data first passes through a convolutional layer with 64 filters, using kernels of sizes 3, 5, and 7 respectively. After convolution, the data is activated by ReLU and then output. The output of the convolutional layer passes through a dropout layer with a dropout rate of 50%. The output of the dropout layer passes through a max pooling layer with a pooling window size of 2. The output of the pooling layer enters a flattening layer, flattening it into a one-dimensional vector. The outputs of the three heads of the flattening layer are merged into a single vector by a merging layer. The merged vector is then fed into a fully connected layer with 24 neurons, activated by ReLU, and then output. The output of the fully connected layer is fed into an output layer with n_outputs neurons, corresponding to the 24 action categories in the classification task, activated by softmax, and output as a 24-element vector containing the probability of belonging to each of the 24 action categories. Finally, the probability of each action category is voted on, and the category with the most votes is the final output.

[0018] The training process for the upper body assembly action recognition model and the lower body assembly action recognition model is as follows:

[0019] Step 1: The PerceptionNeuronStudio sensor will record the following data: sensor pose quaternion (i.e., sensor rotation values ​​in four dimensions), sensor angular velocity in three directions, sensor velocity in three directions, joint velocity in three directions, joint position value in three directions, bone pose quaternion (i.e., bone rotation values ​​in four dimensions), and packet loss rate, for a total of 21 sets of data.

[0020] Each sensor will repeatedly collect the above 21 sets of data. Each set of data is used as a quantity. All sensors will collect a total of 1239 feature quantities in the i-th frame, which will be used as the original dataset.

[0021] Step 2: Label the original data by manually adding a one-dimensional feature to the original dataset. This feature is the label of the action category, thus obtaining the labeled training set.

[0022] Step 3: Input the data labeled as upper body movement categories in the labeled training set into the multi-channel one-dimensional convolutional neural network (1DCNN) model for training until the multi-channel one-dimensional convolutional neural network (1DCNN) model achieves a classification accuracy of over 90% for the data labeled as upper body movement categories. This will result in a well-trained upper body assembly movement recognition model.

[0023] Step four: Input the data labeled as lower body movement categories in the labeled training set into the multi-channel one-dimensional convolutional neural network (1DCNN) model for training until the multi-channel one-dimensional convolutional neural network (1DCNN) model achieves a classification accuracy of over 90% for the data labeled as lower body movement categories. This will result in a well-trained lower body assembly movement recognition model.

[0024] The classification accuracy during the training process of the upper body assembly action recognition model refers to the number of correctly predicted frames in the prediction results output by the multi-channel one-dimensional convolutional neural network (1DCNN) model divided by the total number of frames in the input data multiplied by 100%.

[0025] The classification accuracy during the training process of the lower body assembly action recognition model refers to the number of correctly predicted frames in the prediction results output by the multi-channel one-dimensional convolutional neural network (1DCNN) model divided by the total number of frames in the input data multiplied by 100%.

[0026] The beneficial effects of this invention are:

[0027] This invention is applied to the automotive assembly time analysis process. It uses motion capture equipment to collect motion data, constructs a motion capture and motion segmentation system, accurately obtains the actual motion of the motion capture object, and realizes intelligent motion segmentation and recognition through models. This improves the efficiency of motion segmentation, compensates for the influence of different engineers' skills and experience on the motion segmentation results, and improves accuracy, thus achieving intelligent motion segmentation and recognition.

[0028] Compared with existing technologies, this invention, through specific hyperparameter settings, enables the original unit action recognition model for automobile assembly operations to better extract features, learn complex patterns, and improve the model's classification accuracy in action recognition tasks. Attached Figure Description

[0029] Figure 1 This is an overall flowchart of the motion recognition and segmentation model and system based on motion capture equipment of the present invention;

[0030] Figure 2 The PerceptionNeuronStudio sensor is the motion capture device of this invention;

[0031] Figure 3 This is a schematic diagram of the human body coordinate system obtained by the sensor in this invention;

[0032] Figure 4 This is a structural diagram of the multi-channel one-dimensional convolutional neural network (1DCNN) model of the present invention; Detailed Implementation

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings, not all of the structures.

[0034] In the description of this invention, unless otherwise explicitly specified and limited, the terms "connected," "linked," and "fixed" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0035] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0036] In the description of this embodiment, the terms "upper," "lower," "left," and "right," etc., refer to the orientation or positional relationship shown in the accompanying drawings. They are used only for ease of description and simplification of operation, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first" and "second" are used only for distinction in description and have no special meaning.

[0037] In this embodiment of the invention, an action recognition model for use in the final assembly workshop of an automobile manufacturing company is provided. The specific model establishment includes the following steps:

[0038] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the scope of protection of the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, any person skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention within the scope of the technology disclosed in the present invention. These simple modifications are all within the scope of protection of the present invention.

[0039] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

[0040] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

[0041] Example 1

[0042] A method for recognizing car assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model, including the following:

[0043] Step 1: Divide the complete car assembly process into N types of actions, and then divide the N types of actions into two main categories: upper body actions and lower body actions.

[0044] Step 2: Sensors are attached to different parts of the worker's body to collect data from each sensor when the worker performs upper body and lower body movements. The worker must repeat each movement M times, and the data collected by each sensor is collected M times. All data are then merged to obtain the raw data. The raw data corresponding to the movement categories are divided into training samples and recognition samples.

[0045] Step 3: Build a multi-channel 1D Convolutional Neural Network (1DCNN) recognition model; input the data collected during upper body movements from the raw data obtained in Step 2 into the multi-channel 1DCNN model for training to obtain an upper body assembly movement recognition model; input the data collected during lower body movements from the raw data obtained in Step 2 into the multi-channel 1DCNN model for training to obtain a lower body assembly movement recognition model; combine the upper body recognition model and the lower body recognition model into the final prediction model;

[0046] Step 4: Wear sensors on various parts of the worker's body to obtain data collected by the sensors during normal work.

[0047] Step 5: Input the sensor data collected in Step 4 into the upper body assembly action recognition model and the lower body assembly action recognition model trained in Step 3 respectively. The upper body assembly action recognition model and the lower body assembly action recognition model predict the action category of the input sensor data respectively.

[0048] Step six: Extract the first and last columns, namely the frame number and action category, from the predicted sequence data output by the upper body assembly action recognition model and the lower body assembly action recognition model; then convert the extracted frame number into the working time of the corresponding action category; that is, multiply the frame number extracted in step six by 0.01 to obtain the working time of the corresponding action category; thus completing the recognition of car assembly actions.

[0049] The sensors worn by the workshop workers in steps two and four are...

[0050] PerceptionNeuronStudio sensors are worn by factory workers during car assembly, with each movement corresponding to a specific body part to collect motion data. The data collected by the sensors is human coordinate data in BVH format. (The human coordinates are shown in the image.) Figure 4 As shown;

[0051] In step one, the complete car assembly actions are divided into the following 24 action categories: electric tapping, hammering, embedding, connecting, torque, temporary fixing, painting, twisting, pressing, adjusting, pasting, pushing, pulling, picking up, placing, single torque, rotating arm, kneeling, getting up, squatting, standing up, turning around, stepping, striding, and long-distance walking.

[0052] In step one, the 24 types of movements are divided into upper body movements and lower body movements. Among them, electric shock, knocking, embedding, connecting, torque, temporary fixing, painting, twisting, pressing, adjusting, pasting, pushing, pulling, pressing, picking up, placing, single torque, and rotating the arm are upper body movements, while kneeling, getting up, squatting, standing up, turning around, stepping, striding, and long-distance walking are lower body movements.

[0053] In steps two and four, the sensors are worn on 59 body parts of the workshop workers, namely: buttocks, right thigh, right calf, right foot, left thigh, left calf, left foot, lower spine, middle spine, upper spine, lower neck, upper neck, head, right shoulder, right upper arm, right forearm, right hand, right thumb base, right thumb middle finger, right thumb tip, right index finger metacarpal bone, right index finger base, right index finger middle finger, right index finger tip, right middle finger metacarpal bone, right middle finger base, right middle finger middle finger, right middle finger tip, right ring finger palm. Bone, base of right ring finger, middle finger of right ring finger, tip of right ring finger, metacarpal bone of right little finger, base of right little finger, middle finger of right little finger, tip of right little finger, left shoulder, left upper arm, left forearm, left hand, base of left thumb, middle finger of left thumb, tip of left thumb, metacarpal bone of left index finger, base of left index finger, middle finger of left index finger, tip of left index finger, metacarpal bone of left middle finger, base of left middle finger, middle finger of left middle finger, tip of left middle finger, metacarpal bone of left ring finger, base of left ring finger, middle finger of left ring finger, tip of left ring finger, metacarpal bone of left little finger, base of left little finger, middle finger of left little finger, tip of left little finger.

[0054] The structure of the multi-channel one-dimensional convolutional neural network (1DCNN) model constructed in step three is as follows:

[0055] The multi-channel one-dimensional convolutional neural network (1DCNN) model uses a multi-head model, including three heads, a merging layer, a fully connected layer, and an output layer. Each head contains a convolutional layer, a dropout layer, a pooling layer, and a flattening layer. Each frame of input data first passes through a one-dimensional convolutional layer with 64 filters, using kernels of sizes 3, 5, and 7. After convolution, the data is activated by a ReLU function and then output. The output of the convolutional layer passes through a dropout layer with a dropout rate of 50%. The output of the dropout layer passes through a max pooling layer with a pooling window size of 2. The output of the pooling layer then enters the flattening layer and is flattened... The flattened layer is a one-dimensional vector; the outputs of the three heads are merged into a single vector through a merging layer; the merged vector is fed into a fully connected layer with 24 neurons, and outputs after passing through the ReLU activation function; the output of the fully connected layer is fed into the output layer with n_outputs neurons, corresponding to the number of categories in the classification task, i.e., 24 action categories, and outputs after passing through the softmax activation function. The output of this model will be a 24-element vector containing the probability that a given window belongs to each of the 24 action categories; finally, the probability of each action category is voted on, and the category with the most votes is taken as the final output result.

[0056] The training process for the upper body assembly action recognition model and the lower body assembly action recognition model is as follows:

[0057] Step 1: Use AxisStudio software (existing) to export the BVH format data collected by the PerceptionNeuronStudio sensor into the required CSV format data as the raw data. The CSV format data contains 1240 quantities. The PerceptionNeuronStudio sensor will record the following data: sensor posture quaternion, i.e., sensor rotation values ​​in four dimensions, sensor angular velocity in three directions, sensor velocity in three directions, joint velocity in three directions, joint position values ​​in three directions, skeleton posture quaternion, i.e., skeleton rotation values ​​in four dimensions, and packet loss rate, for a total of 21 sets of data.

[0058] Each sensor will repeatedly collect different sets of the above 21 data. Each set of data is a quantity, plus the number of frames selected when selecting sensor data. All sensors will collect a total of 21*59+1=1240 quantities. These 1240 quantities are recorded as the 1239th feature quantity of the i-th frame, which is used as the original dataset.

[0059] Step 2: Label the original data. Add a one-dimensional feature to the original dataset, namely the 1241st feature. This feature is the label of the action category. In other words, label each set of sensor data in the original data with the corresponding action category label to obtain the labeled training set.

[0060] Step 3: Input the data labeled as upper body movement categories in the labeled training set into the multi-channel one-dimensional convolutional neural network (1DCNN) model for training until the multi-channel one-dimensional convolutional neural network (1DCNN) model achieves a classification accuracy of over 90% for the data labeled as upper body movement categories. This will result in a well-trained upper body assembly movement recognition model.

[0061] Step four: Input the data labeled as lower body movement categories in the labeled training set into the multi-channel one-dimensional convolutional neural network (1DCNN) model for training until the multi-channel one-dimensional convolutional neural network (1DCNN) model achieves a classification accuracy of over 90% for the data labeled as lower body movement categories. This will result in a well-trained lower body assembly movement recognition model.

[0062] The classification accuracy during the training process of the upper body assembly action recognition model refers to the number of correctly predicted frames in the prediction results output by the multi-channel one-dimensional convolutional neural network (1DCNN) model divided by the total number of frames in the input data multiplied by 100%.

[0063] The classification accuracy during the training process of the lower body assembly action recognition model refers to the number of correctly predicted frames in the prediction results output by the multi-channel one-dimensional convolutional neural network (1DCNN) model divided by the total number of frames in the input data multiplied by 100%.

[0064] Example 2

[0065] Step 1: Divide the complete car assembly process into N types of actions, and then divide the N types of actions into two main categories: upper body actions and lower body actions.

[0066] Step 2: Sensors are attached to different parts of the worker's body to collect data from each sensor when the worker performs upper body and lower body movements. The worker must repeat each movement M times, and the data collected by each sensor is collected M times. All data are then merged to obtain the raw data. The raw data corresponding to the movement categories are divided into training samples and recognition samples.

[0067] Step 3: Build a multi-channel 1D Convolutional Neural Network (1DCNN) recognition model; input the data collected during upper body movements from the raw data obtained in Step 2 into the multi-channel 1DCNN model for training to obtain an upper body assembly movement recognition model; input the data collected during lower body movements from the raw data obtained in Step 2 into the multi-channel 1DCNN model for training to obtain a lower body assembly movement recognition model.

[0068] Step 4: Wear sensors on various parts of the worker's body to obtain data collected by the sensors during normal work.

[0069] Step 5: Input the data collected in Step 4 into the upper body assembly action recognition model and the lower body assembly action recognition model trained in Step 3 respectively. The upper body assembly action recognition model and the lower body assembly action recognition model will output the action categories of the upper body actions and lower body actions performed by the workshop workers during normal work, as well as the working time of each action.

[0070] The sensors worn by the workshop workers in steps two and four are...

[0071] PerceptionNeuronStudio sensors are worn by factory workers during car assembly, with each movement corresponding to a specific body part to collect motion data. The data collected by the sensors is human coordinate data in BVH format. (The human coordinates are shown in the image.) Figure 4 As shown;

[0072] In step one, the complete car assembly actions are divided into the following 24 action categories: electric tapping, hammering, embedding, connecting, torque, temporary fixing, painting, twisting, pressing, adjusting, pasting, pushing, pulling, picking up, placing, single torque, rotating arm, kneeling, getting up, squatting, standing up, turning around, stepping, striding, and long-distance walking.

[0073] In step one, the 24 types of movements are divided into upper body movements and lower body movements. Among them, electric shock, knocking, embedding, connecting, torque, temporary fixing, painting, twisting, pressing, adjusting, pasting, pushing, pulling, pressing, picking up, placing, single torque, and rotating the arm are upper body movements, while kneeling, getting up, squatting, standing up, turning around, stepping, striding, and long-distance walking are lower body movements.

[0074] In steps two and four, the sensors are worn on 59 body parts of the workshop workers, namely: buttocks, right thigh, right calf, right foot, left thigh, left calf, left foot, lower spine, middle spine, upper spine, lower neck, upper neck, head, right shoulder, right upper arm, right forearm, right hand, right thumb base, right thumb middle finger, right thumb tip, right index finger metacarpal bone, right index finger base, right index finger middle finger, right index finger tip, right middle finger metacarpal bone, right middle finger base, right middle finger middle finger, right middle finger tip, right ring finger palm. Bone, base of right ring finger, middle finger of right ring finger, tip of right ring finger, metacarpal bone of right little finger, base of right little finger, middle finger of right little finger, tip of right little finger, left shoulder, left upper arm, left forearm, left hand, base of left thumb, middle finger of left thumb, tip of left thumb, metacarpal bone of left index finger, base of left index finger, middle finger of left index finger, tip of left index finger, metacarpal bone of left middle finger, base of left middle finger, middle finger of left middle finger, tip of left middle finger, metacarpal bone of left ring finger, base of left ring finger, middle finger of left ring finger, tip of left ring finger, metacarpal bone of left little finger, base of left little finger, middle finger of left little finger, tip of left little finger.

[0075] The structure of the multi-channel one-dimensional convolutional neural network (1DCNN) model constructed in step three is as follows: The multi-channel one-dimensional convolutional neural network (1DCNN) model uses a multi-head model, wherein the model...

[0076] Each head can use a different kernel size to read the input time step. For example, a three-head model can have three different kernel sizes of 3, 5, and 11, allowing the model to read and interpret sequence data at three different resolutions. Each head contains convolutional layers, dropout layers, pooling layers, and flattening layers.

[0077] Input layer: Data first enters three head layers, each receiving an input tensor of shape (n_timesteps, n_features), where n_timesteps is the time step and n_features is the number of features. In this example, n_timesteps and n_features are automatically inferred from the shape of the input data.

[0078] Convolutional layer: Each frame of input data first passes through a one-dimensional convolutional layer with 64 filters, using convolutional kernels of sizes 3, 5, and 7 respectively; after the convolution operation, the data passes through the ReLU activation function; Dropout layer: The output of each convolutional layer passes through a Dropout layer with a dropout rate of 50%;

[0079] Pooling layer: The output of each Dropout layer goes through a max pooling layer with a pooling window size of 2;

[0080] Flattening layer: The output of each pooling layer is flattened into a one-dimensional vector;

[0081] Merging layer: The outputs of the three flattening layers are merged into a single vector;

[0082] Fully connected layer: The merged vectors are fed into a fully connected layer with 24 neurons and activated by the ReLU function;

[0083] Output layer: The output of the fully connected layer is fed into an output layer with n_outputs neurons, corresponding to the number of categories in the classification task, and activated by the softmax function. The output of the model will be a 24-element vector containing the probability that a given window belongs to each of the 24 activity types.

[0084] The training process for the upper body assembly action recognition model and the lower body assembly action recognition model is as follows:

[0085] Step 1: Using AxisStudio software (available), export the BVH format data acquired by the PerceptionNeuronStudio sensor into the required CSV format as the raw data. The CSV data contains 1240 quantities. The PerceptionNeuronStudio sensor records the sensor pose quaternion Sensor-Quati (representing the rotation of the sensor and skeleton in standard quaternion format, i = 1, 2, 3, 4) and sensor angular velocity Sensor-Gyroi (in radians, i.e., rad / s, i = 5, 6, 7) for each corresponding body part.

[0086] Sensor speed (Sensor-Accei, international standard unit: m / s) 2 The data includes: joint velocity (i = 8, 9, 10), joint position (i = 11, 12, 13), joint position (i = 4, 15, 16), bone pose quaternion (i = 17, 18, 19, 20), and packet loss rate (i = 21). Finally, the sensor will repeatedly collect 21 sets of the above data for each body part, plus the common frame number (1 quantity), for a total of 21 * 59 + 1 = 1240 quantities. These 1240 quantities are the 1239 feature quantities of the i-th frame.

[0087] Step 2: Label the original data and convert the original data from Step 1 into training data. Since the 1DCNN recognition model needs to know the target category for each frame, a one-dimensional feature (the 1241st feature) is manually added to the original dataset. This feature is the action category, where the target category for the upper body training data is T1i (refer to Table 1 for upper body action categories, i = 1, 2, 3, ..., 16).

[0088] The target category for lower body training data is T2i (refer to Table 1 for lower body movement types, i = 1, 2, 3, ..., 8).

[0089] The complete training data is the action category of the 1239th feature in the i-th frame (1241 dimensions in total, with the frame number and the target category each occupying 1 dimension).

[0090] Step 3: Input the data labeled as upper body movements from the labeled training set into a multi-channel 1D Convolutional Neural Network (1DCNN) model for training. This model is suitable for a fixed number of epochs, in this case 100. The batch size is 32; 32 data windows are input into the model before updating the model weights. After the model is fitted, it is evaluated on the test dataset, and the accuracy of the fitted model on the test dataset is returned. Input the data labeled as lower body movements from the labeled training set into the multi-channel 1D Convolutional Neural Network (1DCNN) model for training to obtain a trained lower body assembly movement recognition model.

[0091] The upper body recognition model and the lower body recognition model are combined into the final prediction model. The input is sequence data containing 1240 features (without a target category). After processing by the upper body recognition model and the lower body recognition model, the returned sequence data will have 1241 features. The new column is the action category, which is the 1240 features of frame i plus the action category. The first and last columns of the sequence data with 1241 features returned by the two models, namely the frame number and the action category, are extracted and combined to form the final output result, as shown in Table 2.

[0092] Table 1 – Original Units for 24 Assembly Action Categories;

[0093]

[0094]

[0095] Table 2 – Output Results of Multi-channel One-dimensional Convolutional Neural Network (1DCNN) Model

[0096]

[0097] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the scope of protection of the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, any person skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention within the scope of the technology disclosed in the present invention. These simple modifications are all within the scope of protection of the present invention.

[0098] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

[0099] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

Claims

1. A method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model, characterized in that, Specifically, it includes the following: Step 1: Divide the complete car assembly process into N types of actions, and then divide the N types of actions into two main categories: upper body actions and lower body actions. Step 2: Wear sensors on different parts of the worker's body to collect data from each sensor when the worker performs upper body and lower body movements. The worker must repeat each movement M times, and the data collected by each sensor must be collected M times to obtain the raw data. Step 3: Build a multi-channel 1D Convolutional Neural Network (1DCNN) recognition model; input the data collected during upper body movements from the raw data obtained in Step 2 into the multi-channel 1DCNN recognition model for training to obtain an upper body assembly movement recognition model; input the data collected during lower body movements from the raw data obtained in Step 2 into the multi-channel 1DCNN recognition model for training to obtain a lower body assembly movement recognition model; combine the upper body recognition model and the lower body recognition model into the final prediction model; Step 4: Wear sensors on various parts of the worker's body to obtain data collected by the sensors during normal work. Step 5: Input the sensor data collected in Step 4 into the upper body assembly action recognition model and the lower body assembly action recognition model trained in Step 3 respectively. The upper body assembly action recognition model and the lower body assembly action recognition model predict the action category of the input sensor data respectively. Step 6: Extract the first and last columns, namely the frame number and action category, from the predicted sequence data output by the upper body assembly action recognition model and the lower body assembly action recognition model; then convert the extracted frame number into the working time of the corresponding action category; complete the recognition of car assembly actions.

2. The method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model according to claim 1, characterized in that, The sensors worn by the workshop workers in steps two and four are PerceptionNeuronStudio sensors; the data collected by the sensors is human body coordinate data in bvh format.

3. The method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model according to claim 1, characterized in that, In step one, the complete car assembly actions are divided into the following 24 action categories: electric tapping, hammering, embedding, connecting, torque, temporary fixing, painting, twisting, pressing, adjusting, pasting, pushing, pulling, picking up, placing, single torque, rotating arm, kneeling, getting up, squatting, standing up, turning around, stepping, striding, and long-distance walking.

4. The method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model according to claim 3, characterized in that, In step one, the 24 types of movements are divided into upper body movements and lower body movements. Among them, electric shock, knocking, embedding, connecting, torque, temporary fixing, painting, twisting, pressing, adjusting, pasting, pushing, pulling, pressing, picking up, placing, single torque, and rotating the arm are upper body movements, while kneeling, getting up, squatting, standing up, turning around, stepping, striding, and long-distance walking are lower body movements.

5. The method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model according to claim 1, characterized in that, In steps two and four, the sensors are worn on 59 body parts of the workshop workers, namely: buttocks, right thigh, right calf, right foot, left thigh, left calf, left foot, lower spine, middle spine, upper spine, lower neck, upper neck, head, right shoulder, right upper arm, right forearm, right hand, right thumb base, right thumb middle finger, right thumb tip, right index finger metacarpal bone, right index finger base, right index finger middle finger, right index finger tip, right middle finger metacarpal bone, right middle finger base, right middle finger middle finger, right middle finger tip, right ring finger palm. Bone, base of right ring finger, middle finger of right ring finger, tip of right ring finger, metacarpal bone of right little finger, base of right little finger, middle finger of right little finger, tip of right little finger, left shoulder, left upper arm, left forearm, left hand, base of left thumb, middle finger of left thumb, tip of left thumb, metacarpal bone of left index finger, base of left index finger, middle finger of left index finger, tip of left index finger, metacarpal bone of left middle finger, base of left middle finger, middle finger of left middle finger, tip of left middle finger, metacarpal bone of left ring finger, base of left ring finger, middle finger of left ring finger, tip of left ring finger, metacarpal bone of left little finger, base of left little finger, middle finger of left little finger, tip of left little finger.

6. The method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model according to claim 1, characterized in that, The structure of the multi-channel one-dimensional convolutional neural network (1DCNN) model constructed in step three is as follows: The multi-channel one-dimensional convolutional neural network (1DCNN) model uses a multi-head model. Each frame of input data first passes through a convolutional layer with 64 filters, using kernels of sizes 3, 5, and 7 respectively. After convolution, the data is activated by ReLU and then output. The output of the convolutional layer passes through a dropout layer with a dropout rate of 50%. The output of the dropout layer passes through a max pooling layer with a pooling window size of 2. The output of the pooling layer enters a flattening layer, flattening it into a one-dimensional vector. The outputs of the three heads of the flattening layer are merged into a single vector by a merging layer. The merged vector is then fed into a fully connected layer with 24 neurons, activated by ReLU, and then output. The output of the fully connected layer is fed into an output layer with n_outputs neurons, corresponding to the 24 action categories in the classification task, activated by softmax, and output as a 24-element vector containing the probability of belonging to each of the 24 action categories. Finally, the probability of each action category is voted on, and the category with the most votes is the final output.

7. The method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model according to claim 6, characterized in that, The training process for the upper body assembly action recognition model and the lower body assembly action recognition model is as follows: Step 1: The PerceptionNeuronStudio sensor will record the following data: sensor pose quaternion (i.e., sensor rotation values ​​in four dimensions), sensor angular velocity in three directions, sensor velocity in three directions, joint velocity in three directions, joint position value in three directions, bone pose quaternion (i.e., bone rotation values ​​in four dimensions), and packet loss rate, for a total of 21 sets of data. Each sensor will repeatedly collect the above 21 sets of data. Each set of data is used as a quantity. All sensors will collect a total of 1239 feature quantities in the i-th frame, which will be used as the original dataset. Step 2: Label the original data by manually adding a one-dimensional feature to the original dataset. This feature is the label of the action category, thus obtaining the labeled training set. Step 3: Input the data labeled as upper body movement categories in the labeled training set into the multi-channel one-dimensional convolutional neural network (1DCNN) model for training until the multi-channel one-dimensional convolutional neural network (1DCNN) model achieves a classification accuracy of over 90% for the data labeled as upper body movement categories. This will result in a well-trained upper body assembly movement recognition model. Step four: Input the data labeled as lower body movement categories in the labeled training set into the multi-channel one-dimensional convolutional neural network (1DCNN) model for training until the multi-channel one-dimensional convolutional neural network (1DCNN) model achieves a classification accuracy of over 90% for the data labeled as lower body movement categories. This will result in a well-trained lower body assembly movement recognition model.

8. The method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model according to claim 7, characterized in that, In the training process of the upper body assembly action recognition model, the classification accuracy is the number of correctly predicted frames in the prediction results output by the multi-channel one-dimensional convolutional neural network (1DCNN) model divided by the total number of frames in the input data multiplied by 100%.

9. The method for recognizing automobile assembly actions using a multi-channel one-dimensional convolutional neural network (1DCNN) model according to claim 7, characterized in that, In the training process of the lower body assembly action recognition model, the classification accuracy is the number of correctly predicted frames in the prediction results output by the multi-channel one-dimensional convolutional neural network (1DCNN) model divided by the total number of frames in the input data multiplied by 100%.

Images