Abnormity recognition method, device and equipment for double-arm robot and medium

Abstract

The embodiment of the invention provides an anomaly recognition method, device and equipment for a double-arm robot and a medium. The method comprises the steps of obtaining a three-dimensional image and time sequence data of an environment where the double-arm robot is located; performing feature extraction and data fusion on the three-dimensional image and the time series data to obtain fused feature data; the fused feature data are input into a preset anomaly detection model, errors between the fused feature data and reconstructed feature data are obtained, the anomaly detection model comprises an encoder, a decoder and a calculation module, the encoder is used for mapping the input data to a low-dimensional implicit space, and the decoder is used for decoding the input data to obtain the reconstructed feature data; the decoder is used for reconstructing input data from a low-dimensional implicit space to obtain reconstructed feature data, and the calculation module is used for calculating an error between the input data and the reconstructed feature data; and according to the error and a preset error threshold value, whether the two-arm robot is abnormal or not is determined. The method is used for achieving the effect of improving the generalization ability of performing anomaly recognition on the double-arm robot.

Description

Technical Field

[0001] The present application relates to the technical field of robot control, and particularly to an abnormal recognition method, device, equipment and medium for a dual-arm robot. Background Art

[0002] Dual-arm robots are an important branch of robot technology. In recent years, dual-arm robots have been widely used in many fields such as industrial manufacturing, medical surgery, and agricultural harvesting. Their core advantages lie in a larger operating space and stronger flexibility, making them particularly suitable for complex operation scenarios such as disassembly, assembly, cleaning, and material handling tasks.

[0003] Since dual-arm robots will face various abnormalities when performing complex tasks, such as collisions, sensor failures, system malfunctions, etc., therefore, identifying the self-abnormalities of dual-arm robots is a key measure to ensure the safe, efficient, and stable operation of the robots.

[0004] However, the abnormal recognition methods in the prior art usually need to rely on a large number of abnormal data samples in specific scenarios to train the abnormal detection model, which makes this method lack the generalization ability for unknown abnormalities in new environments, resulting in difficulty in effectively applying it in dynamic environments or complex tasks. Summary of the Invention

[0005] Embodiments of the present application provide an abnormal recognition method, device, equipment and medium for a dual-arm robot, so as to achieve the effect of improving the generalization ability of abnormal recognition.

[0006] In a first aspect, embodiments of the present application provide an abnormal recognition method for a dual-arm robot, including:

[0007] Obtain a three-dimensional image and time-series data of the environment where the dual-arm robot is located, where the time-series data includes the pose information and joint information of the dual-arm robot;

[0008] Perform feature extraction and data fusion on the three-dimensional image and the time-series data to obtain fusion feature data;

[0009] Input the fusion feature data into a preset abnormal detection model to obtain the error between the fusion feature data and the reconstructed feature data, where the abnormal detection model includes an encoder, a decoder, and a calculation module. The encoder is used to map the input data to a low-dimensional latent space, the decoder is used to reconstruct the input data from the low-dimensional latent space to obtain the reconstructed feature data, and the calculation module is used to calculate the error between the input data and the reconstructed feature data;

[0010] Determine whether the dual-arm robot has an abnormality according to the error and a preset error threshold.

[0011] In a possible implementation, if it is determined that the dual-arm robot has an abnormality, the method further includes:

[0012] Determine the target abnormality level of the dual-arm robot according to the error and the error ranges corresponding to a plurality of preset abnormality levels;

[0013] Determine the target abnormality level of the dual-arm robot according to the error and the error ranges corresponding to a plurality of preset abnormality levels.

[0014] In a possible implementation, the determining the target abnormality level of the dual-arm robot according to the error and the error ranges corresponding to a plurality of preset abnormality levels includes:

[0015] If the target abnormality level is a minor abnormality, adjust the control parameters according to a preset parameter adjustment method to adjust the joint torque or the driving path of the dual-arm robot;

[0016] If the target abnormality level is a moderate abnormality, the task currently being executed by the dual-arm robot;

[0017] If the target abnormality level is a serious abnormality, stop the task currently being executed by the dual-arm robot, disconnect the power output of all joint motors of the dual-arm robot, and push an alarm message.

[0018] In a possible implementation, the extracting features and fusing data from the three-dimensional image and the time-series data to obtain fused feature data includes:

[0019] Perform feature extraction processing on the three-dimensional image by using a convolutional neural network to obtain visual feature data;

[0020] Extract features from the time-series data to obtain time-series feature data;

[0021] Stitch the visual feature data and the time-series feature data, and perform dimensionality reduction processing on the stitched data to obtain fused feature data.

[0022] In a possible implementation, the method further includes:

[0023] Obtain new fused feature data and abnormal conditions generated during the operation of the dual-arm robot;

[0024] Optimize and train the abnormality detection model according to the new fused feature data and the abnormal conditions to obtain an optimized abnormality detection model.

[0025] In a possible implementation, the pose information of the dual-arm robot includes: the posture and motion state of the dual-arm robot;

[0026] The joint information of the dual-arm robot includes: force feedback, angle, speed, and position of the joints of the dual-arm robot.

[0027] In a possible implementation, before performing feature extraction and data fusion on the 3D image and the time-series data, the method further includes:

[0028] Performing data stream synchronization and denoising processing on the 3D image and the time-series data to obtain a processed 3D image and processed time-series data;

[0029] Performing timestamp alignment processing on the data collected by different sensors in the processed time-series data to obtain new time-series data;

[0030] Correspondingly, performing feature extraction and data fusion on the 3D image and the time-series data includes:

[0031] Performing feature extraction and data fusion on the processed 3D image and the new time-series data.

[0032] In a second aspect, an embodiment of the present application provides an abnormal recognition device for a dual-arm robot, including:

[0033] A first acquisition unit, configured to acquire a 3D image of the environment where the dual-arm robot is located and time-series data, where the time-series data includes the pose information and joint information of the dual-arm robot;

[0034] A fusion unit, configured to perform feature extraction and data fusion on the 3D image and the time-series data to obtain fusion feature data;

[0035] A detection unit, configured to input the fusion feature data into a preset abnormal detection model to obtain an error between the fusion feature data and the reconstructed feature data, where the abnormal detection model includes an encoder, a decoder, and a calculation module, the encoder is configured to map the input data to a low-dimensional latent space, the decoder is configured to reconstruct the input data from the low-dimensional latent space to obtain the reconstructed feature data, and the calculation module is configured to calculate the error between the input data and the reconstructed feature data;

[0036] A first determination unit, configured to determine whether the dual-arm robot has an abnormality according to the error and a preset error threshold.

[0037] In a possible implementation, if it is determined that the dual-arm robot has an abnormality, the abnormal recognition device of the dual-arm robot further includes:

[0038] A second determination unit, configured to determine a target anomaly level of the dual-arm robot according to the error and error ranges corresponding to a plurality of preset anomaly levels;

[0039] An execution unit, configured to control the dual-arm robot to execute corresponding anomaly handling measures according to the target anomaly level

[0040] In a possible implementation manner, the execution unit is specifically configured to:

[0041] If the target anomaly level is a minor anomaly, control parameter adjustment is performed according to a preset parameter adjustment method to adjust the joint torque or driving path of the dual-arm robot;

[0042] If the target anomaly level is a moderate anomaly, the task currently being executed by the dual-arm robot is paused;

[0043] If the target anomaly level is a severe anomaly, the task currently being executed by the dual-arm robot is stopped, the power output of all joint motors of the dual-arm robot is disconnected, and an alarm message is pushed.

[0044] In a possible implementation manner, the fusion unit specifically includes:

[0045] Performing feature extraction processing on the three-dimensional image by using a convolutional neural network to obtain visual feature data;

[0046] Performing feature extraction on the time series data to obtain time series feature data;

[0047] Splicing the visual feature data and the time series feature data, and performing dimensionality reduction processing on the spliced data to obtain fusion feature data.

[0048] In a possible implementation manner, the anomaly recognition device of the dual-arm robot further includes:

[0049] A second acquisition unit, configured to acquire new fusion feature data and anomaly situations generated during the operation of the dual-arm robot;

[0050] An optimization unit, configured to optimize and train the anomaly detection model according to the new fusion feature data and the anomaly situations to obtain an optimized anomaly detection model.

[0051] In a possible implementation manner, the pose information of the dual-arm robot in the first acquisition unit includes: the attitude and motion state of the dual-arm robot;

[0052] The joint information of the dual-arm robot includes: the force feedback, angle, speed, and position of the joints of the dual-arm robot.

[0053] In a possible implementation, the anomaly recognition device of the dual-arm robot further includes:

[0054] A first processing unit, configured to perform data stream synchronization and denoising processing on the three-dimensional image and the time series data to obtain a processed three-dimensional image and processed time series data;

[0055] A second processing unit, configured to perform timestamp alignment processing on the data collected by different sensors in the processed time series data to obtain new time series data;

[0056] Correspondingly, the fusion unit is specifically configured to:

[0057] Perform feature extraction and data fusion on the processed three-dimensional image and the new time series data.

[0058] In a third aspect, an embodiment of the present application provides a computer device, including: a memory and a processor;

[0059] The memory stores computer-executable instructions;

[0060] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect as described above.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the first aspect and / or various possible implementation manners of the first aspect as described above.

[0062] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the first aspect and / or various possible implementation manners of the first aspect as described above.

[0063] The abnormal recognition method, device, equipment and medium of the dual-arm robot provided by the embodiments of the present application first obtain the three-dimensional image and time-series data of the environment where the dual-arm robot is located, then perform feature extraction and data fusion on the three-dimensional image and time-series data to obtain fusion feature data, and then input the fusion feature data into a preset abnormal detection model to obtain the error between the fusion feature data and the reconstructed normal feature data. Finally, according to the error and the preset error threshold, it is determined whether the dual-arm robot has an abnormality. Among them, the abnormal detection model in this method only needs to be trained with multi-modal data under normal operations collected during the daily operation of the dual-arm robot, which reduces the dependence of the abnormal detection model on a large number of abnormal data samples in a specific scenario, achieves the effect of improving the generalization ability of the model, and enables the abnormal detection model to be effectively applied in a dynamic environment or complex tasks. In addition, this method can also achieve a rapid response to the faults of the robot and effectively ensure the stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0065] Figure 1 It is a schematic flowchart of the abnormal recognition method of the dual-arm robot provided by Embodiment 1 of the present application;

[0066] Figure 2 It is a schematic structural diagram of the abnormal recognition device of the dual-arm robot provided by Embodiment 5 of the present application;

[0067] Figure 3 It is a schematic structural diagram of the abnormal recognition device of the dual-arm robot provided by Embodiment 6 of the present application;

[0068] Figure 4 It is a schematic structural diagram of the computer equipment provided by the present application.

[0069] Through the above-mentioned accompanying drawings, the clear embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and the textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of the devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0071] To clearly understand the technical solution of this application, the following details the solutions of the prior art.

[0072] Dual-arm robots are an important branch of robotics technology. In recent years, dual-arm robots have been widely used in various fields such as industrial manufacturing, medical surgery, and agricultural harvesting. Their core advantages lie in a larger operating space and stronger flexibility, making them particularly suitable for complex operation scenarios such as disassembly, assembly, cleaning, and material handling tasks. Currently, the development of dual-arm robots is moving towards higher automation and more flexible operation, which benefits from the combination of various technologies such as visual recognition, path planning, and force control.

[0073] Dual-arm robots will face various anomalies when performing complex tasks, such as collisions, sensor failures, system malfunctions, etc. In the prior art, since a major challenge in anomaly detection is that it is difficult to obtain a large number of labeled anomaly datasets in the real production line, especially in industrial applications, it is difficult to simulate real fault scenarios and perform labeling. Many anomaly detection methods based on machine learning in the prior art rely on a large number of anomaly data samples in specific scenarios to train the anomaly detection model, which makes the methods in the prior art generally lack the generalization ability for unknown anomalies in new environments, resulting in difficulty in effectively applying them in dynamic environments or complex tasks.

[0074] In addition, in applications such as assisting robot operations, due to the diverse types of data generated by sensors, how to fuse this high-dimensional and heterogeneous data to improve the accuracy of anomaly detection remains a difficult problem; furthermore, although there are already some algorithms that can detect anomalies, the delay in detection and response during task execution is also an urgent problem to be solved. Especially in complex environments, their real-time performance and response speed may not be sufficient to prevent faults from occurring.

[0075] In view of the technical problems in the above-mentioned background art, when studying the abnormal recognition method of a dual-arm robot, the inventor found that by first constructing an abnormal recognition model including an encoder, a decoder, and a calculation module, and using multi-modal data under normal operations collected during the daily operation of the dual-arm robot to train the decoder in the abnormal detection model. Based on this, when recognizing the abnormality of the dual-arm robot based on this abnormal detection model, the input data can be first mapped to a low-dimensional latent space through the encoder, and then the decoder can reconstruct the input data from the low-dimensional latent space to obtain the reconstructed feature data. Finally, the calculation module calculates the error between the input data and the reconstructed feature data. In this way, it is not necessary to use a large number of abnormal data samples to train the abnormal detection model, which can effectively improve the generalization ability of the abnormal detection model, and further enable the abnormal detection model to be applied in dynamic environments or complex tasks. In addition, this method can also improve the detection speed and response speed of abnormal recognition to a certain extent.

[0076] The following uses specific embodiments to elaborate in detail on the technical solutions of the present application and how the technical solutions of the present application solve the above-mentioned technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0077] Figure 1 It is a schematic flowchart of the abnormal recognition method for a dual-arm robot provided in Embodiment 1 of the present application. As Figure 1 shown, the method includes:

[0078] S101. Obtain a three-dimensional image and time-series data of the environment where the dual-arm robot is located, where the time-series data includes the pose information and joint information of the dual-arm robot.

[0079] In this step, in order to realize the abnormal state of the dual-arm robot, it is necessary to obtain multi-modal data of the dual-arm robot according to multiple sensors installed on the dual-arm robot. Among them, the multi-modal data includes a three-dimensional image of the environment where the dual-arm robot is located and time-series data, and the time-series data includes the time-series data of the pose information and joint information of the dual-arm robot.

[0080] In a specific implementation manner, the pose information of the dual-arm robot includes the posture and motion state of the dual-arm robot; the joint information of the dual-arm robot includes the force feedback, angle, speed, and position of the joints of the dual-arm robot.

[0081] Specifically, a three-dimensional image of the environment where the dual-arm robot is located can be obtained according to the RGB-D camera in the vision sensor installed on the end effector of the dual-arm robot; the interaction force between joints can be monitored according to the force / torque sensor installed at the joints of the dual-arm robot to obtain the force feedback of the joints of the dual-arm robot; the angles, speeds, and positions of the joints can be obtained according to the position sensors installed at the joints of the dual-arm robot; and the posture and motion state of the robot can be obtained according to the Inertial Measurement Unit (IMU) installed in the dual-arm robot.

[0082] S102. Extract features and perform data fusion on the three-dimensional image and the time-series data to obtain fused feature data.

[0083] In this step, it is necessary to perform feature extraction on the obtained three-dimensional image and time-series data based on a deep learning algorithm, respectively obtaining visual feature data and time-series feature data; and by performing data fusion on the visual feature data and the time-series feature data, fused feature data is obtained. Among them, the visual feature data extracted from the three-dimensional image includes the shape, position, and obstacle information of the objects in the environment where the dual-arm robot is located.

[0084] In a specific implementation manner, before performing feature extraction and data fusion on the three-dimensional image and the time-series data, the abnormal recognition method of the dual-arm robot provided in this application further includes the following preprocessing processes in steps 1 - 2:

[0085] Step 1. Perform data stream synchronization and denoising processing on the three-dimensional image and the time-series data to obtain the processed three-dimensional image and the processed time-series data;

[0086] In this step, first, it is necessary to perform data stream synchronization processing on the three-dimensional image and the time-series data to unify the sampling frequencies of each sensor; then, perform denoising processing on the three-dimensional image and the time-series data that have completed the synchronization processing.

[0087] Optionally, the data points of the data collected by the sensor with a small sampling frequency can be increased by an interpolation method (such as linear interpolation), or the data points of the data collected by the sensor with a large sampling frequency can be reduced by a downsampling method, so that the data of different sensors can be synchronously processed within the same time interval.

[0088] Among them, when using linear interpolation to synchronize data with different sampling frequencies, the following formula can be used for calculation:

[0089]

[0090] Among them, , , Represents the data points of the sensor at two moments, Represents the data points newly added to the data stream.

[0091] Optionally, a Kalman filter can be used to denoise the synchronized 3D image and time-series data to reduce errors introduced by sensor jitter or environmental interference.

[0092] Step 2: Align the timestamps of the data collected by different sensors in the processed time-series data to obtain new time-series data.

[0093] In this step, to ensure that the data collected by different sensors at the same time point can be corresponding, it is necessary to align the timestamps of the data collected by different sensors in the processed time-series data to obtain new time-series data.

[0094] Optionally, a unified timestamp can be determined according to the timestamp of each sensor and the number of sensors, and the data can be aligned according to the unified timestamp. The following formula can be used to determine the unified timestamp:

[0095]

[0096] Where, Represents the unified timestamp, Represents the number of sensors, Represents the th timestamp of the sensor.

[0097] Correspondingly, after obtaining the processed 3D image and new time-series data by using the preprocessing process in Steps 1 - 2, the method in Step S102 corresponds to: extracting features and fusing data for the processed 3D image and new time-series data.

[0098] Optionally, after obtaining the processed 3D image and new time-series data by using the preprocessing process in Steps 1 - 2, the new time-series data can also be normalized to obtain normalized time-series data, so that the data from different sensors can perform feature extraction and fusion at the same scale. Correspondingly, the method in Step S102 corresponds to: extracting features and fusing data for the processed 3D image and new time-series data.

[0099] In a specific implementation manner, the following methods in Steps S1021 - S1022 can be used to extract features and fuse data for the 3D image and time-series data to obtain fused feature data:

[0100] S1021: Use a convolutional neural network to perform feature extraction processing on the 3D image to obtain visual feature data.

[0101] In this step, a Convolutional Neural Network (CNN) is used to extract high-level visual feature data from the three-dimensional image through multi-layer convolution and pooling operations.

[0102] Specifically, the convolution operation formula of the CNN is:

[0103]

[0104] where, represents the convolutional kernel weight; represents the pixel value of the input image; represents the pixel value in the output feature map, and each pixel value in the output feature map represents the feature of the image at ; represents the size of the convolutional kernel.

[0105] S1022. Extract features from the time-series data to obtain time-series feature data.

[0106] In this step, a deep learning algorithm capable of extracting features from one-dimensional time-series data can be used to process one-dimensional time-series data such as joint force feedback, angle, and speed to obtain time-series feature data.

[0107] Exemplarily, a Long Short Term Memory (LSTM) network can be used to extract features from the time-series data to obtain time-series feature data.

[0108] S1023. Concatenate the visual feature data and the time-series feature data, and perform dimensionality reduction on the concatenated data to obtain fused feature data.

[0109] In this step, in order to comprehensively utilize the feature data extracted from multi-modal data, it is necessary to concatenate the visual feature data and the time-series feature data, and perform dimensionality reduction on the concatenated data to obtain fused feature data for anomaly detection of the dual-arm robot.

[0110] Exemplarily, first, the extracted visual feature data and time-series feature data can be concatenated to obtain the concatenated data. It should be understood that the concatenated data is a combined feature vector. Specifically, it can be expressed by the following formula:

[0111]

[0112] In the above formula, represents the visual feature data extracted from the three-dimensional image, represents the time-series feature data extracted from the time-series data, Represents the concatenated data.

[0113] Then, a fully connected layer can be used to further fuse and reduce the dimensionality of the concatenated data, which can be expressed by the following formula:

[0114]

[0115] In the above formula, Represents the weight matrix of the fully connected layer; Represents the bias, Represents the fused feature data.

[0116] In this implementation, visual feature data is extracted from 3D images using CNN, and temporal feature data is obtained by combining LSTM to process temporal data, enabling the system to process spatial and temporal information simultaneously; moreover, through the fusion of multi-modal data, the efficient perception of complex robot behaviors and dynamic changes is ensured, achieving the effect of improving the accuracy and robustness of anomaly detection.

[0117] S103. Input the fused feature data into a preset anomaly detection model to obtain the error between the fused feature data and the reconstructed feature data, where the anomaly detection model includes an encoder, a decoder, and a calculation module.

[0118] In this step, it is necessary to input the fused feature data that has fused multi-modal data features into a preset anomaly detection model to calculate the error for anomaly detection. First, the input fused feature data is mapped to a low-dimensional latent space by the encoder, then the decoder reconstructs the input data from the low-dimensional latent space to obtain the reconstructed feature data (which can also be called the reconstructed data), and finally, the calculation module calculates the error (which can also be called the reconstruction error) between the input data and the reconstructed feature data.

[0119] It should be understood that the decoder reconstructing the input data means reconstructing the input data into normal feature data.

[0120] Specifically, ① The encoder maps the input fused feature data to a low-dimensional latent space, which can be expressed by the following formula:

[0121]

[0122] In the above formula, Represents the weight matrix of the encoder; Represents the bias of the encoder; Represents the activation function, Represents the input fused feature data, Represents the latent representation obtained by compressing the input fused feature data.

[0123] ②The decoder reconstructs the input data from the low-dimensional latent space, which can be expressed by the following formula:

[0124]

[0125] Among them, is the weight matrix of the decoder; is the bias of the decoder, represents the reconstructed data.

[0126] ③The mean square error between the input data and the reconstructed data can be used as the output of the calculation module, which can be expressed by the following formula:

[0127]

[0128] In the above formula, represents the reconstruction error.

[0129] In a specific implementation, the mean square error function can be used as the loss function for training the autoencoder. Among them, the autoencoder refers to the encoder and decoder of the anomaly detection model. The sample data used for training the autoencoder is usually the multi-modal data of the dual-arm robot under normal operation. By continuously adjusting the and in the autoencoder, the autoencoder can reconstruct the input data into normal feature data. The loss function can be expressed by the following formula:

[0130]

[0131] Among them, represents the number of data samples, represents the th input data, represents the th reconstructed data.

[0132] S104. Determine whether the dual-arm robot has an anomaly according to the error and a preset error threshold.

[0133] In this step, it is necessary to compare the error between the obtained input data and the reconstructed feature data with the preset error threshold, and determine whether the dual-arm robot has an anomaly according to the comparison result.

[0134] Specifically, when the reconstruction error exceeds the preset error threshold , it is determined that the dual-arm robot has an anomaly, which can be expressed by the following formula:

[0135]

[0136] Among them, the preset error threshold can be determined according to the actual application situation, and this application does not limit it. Exemplarily, the error threshold can be selected according to the distribution of the reconstruction error corresponding to the multi-modal data (i.e., normal data) of the dual-arm robot under normal operation. For example, the error threshold is set to the mean plus a certain multiple of the standard deviation:

[0137]

[0138] In the above formula, represents the mean in the distribution of the reconstruction error corresponding to the normal data, represents the standard deviation in the distribution of the reconstruction error corresponding to the normal data, represents the adjustment coefficient.

[0139] The abnormal recognition method of the dual-arm robot provided in this embodiment first obtains the three-dimensional image and time-series data of the environment where the dual-arm robot is located, then extracts features and fuses the data of the three-dimensional image and time-series data to obtain fused feature data, and then inputs the fused feature data into a preset abnormal detection model to obtain the error between the fused feature data and the reconstructed normal feature data. Finally, according to the error and the preset error threshold, it is determined whether the dual-arm robot has an abnormality. Among them, the abnormal detection model only needs to be trained by using the multi-modal data under normal operation collected during the daily operation of the dual-arm robot. This method can not only reduce the dependence of the abnormal detection model on a large number of abnormal data samples, improve the generalization ability of the model, so that the model can be effectively applied in dynamic environments or complex tasks, but also can achieve a rapid response to the faults of the robot and effectively ensure the stable operation of the system. In addition, the visual feature data is extracted from the three-dimensional image by using CNN, and the time-series feature data is obtained by combining LSTM to process the time-series data, enabling the system to process spatial and temporal information simultaneously; and through the fusion of multi-modal data, the efficient perception of the complex behaviors and dynamic changes of the robot is ensured, achieving the effect of improving the accuracy and robustness of abnormal detection.

[0140] Further, Embodiment 2 of this application provides an abnormal recognition method for a dual-arm robot. To ensure the real-time performance and accuracy of abnormal recognition, on the basis of the above embodiment, this embodiment provides a specific implementation method, including: real-time monitoring of the multi-modal data of the robot's operation through a sliding window mechanism to obtain the fused feature data of the time series, and based on the fused feature data within each window, determining whether the dual-arm robot has an abnormality at the current time point.

[0141] Specifically, assuming that the size of the sliding window is , then the sliding window data at time point is:

[0142]

[0143] In the above formula, represents the input data at the time point , where , , …, .

[0144] Within the sliding window, it is necessary to reconstruct the data for each time point in the window and calculate the reconstruction error :

[0145]

[0146] Then, calculate the average reconstruction error within the entire window :

[0147]

[0148] Finally, based on the average reconstruction error corresponding to the current time point and a preset error threshold, it can be determined whether there is an abnormality in the dual-arm robot at the current time point.

[0149] In addition, every time a time step passes, the window slides forward by one time step. Exemplarily, the data in the sliding window at the time point is updated as:

[0150]

[0151] For the sliding window corresponding to the time point , it is necessary to reconstruct the data for each time point in the window again and calculate the average reconstruction error corresponding to the time point ; furthermore, based on the average reconstruction error corresponding to the time point , determine whether there is an abnormality in the dual-arm robot at the time point .

[0152] In this embodiment, by using the sliding window mechanism, real-time monitoring of the multi-modal data of the robot's operation is performed to obtain the fused feature data of the time series, and based on the fused feature data within each window, the means of determining whether there is an abnormality in the dual-arm robot at the current time point ensures the real-time and accuracy of abnormality recognition, improves the response speed of abnormality recognition, and effectively prevents the occurrence of serious failures of the dual-arm robot.

[0153] Further, Embodiment 3 of the present application provides an abnormality recognition method for a dual-arm robot. On the basis of the above embodiment, if it is determined that there is an abnormality in the dual-arm robot, the method provided in this embodiment further includes:

[0154] Step 1: Determine the target anomaly level of the dual-arm robot according to the error and the error ranges corresponding to a plurality of preset anomaly levels.

[0155] In this step, the target anomaly level of the dual-arm robot can be determined based on the calculated error and the error ranges corresponding to the preset anomaly levels, which is used to determine the anomaly handling measures for the dual-arm robot. Among them, the error can be the reconstruction error obtained by calculation in step 103 of Embodiment 1. It can also be the average reconstruction error of the entire sliding window corresponding to each time step obtained by calculation in Embodiment 2. .

[0156] In a specific implementation manner, the plurality of anomaly levels include: minor anomaly, moderate anomaly, and severe anomaly. Exemplarily, if the error is greater than the first threshold and less than or equal to the second threshold, it is determined that the target anomaly level of the dual-arm robot is a minor anomaly; if the error is greater than the second threshold and less than or equal to the third threshold, it is determined that the target anomaly level of the dual-arm robot is a moderate anomaly; if the error is greater than the third threshold, it is determined that the target anomaly level of the dual-arm robot is a severe anomaly.

[0157] Step 2: Control the dual-arm robot to execute corresponding anomaly handling measures according to the target anomaly level.

[0158] In a specific implementation manner, the method for determining the anomaly handling measures of the dual-arm robot according to the target anomaly level of the dual-arm robot includes:

[0159] Case 1: If the target anomaly level is a minor anomaly, control parameter adjustment is performed according to the preset parameter adjustment method to adjust the joint torque or driving path of the dual-arm robot.

[0160] In this case, exemplarily, the following formula can be used to adjust the torque of the dual-arm robot:

[0161]

[0162] In the above formula, represents the original torque, represents the adjustment value, represents the adjusted torque.

[0163] Case 2: If the target anomaly level is a moderate anomaly, pause the task currently executed by the dual-arm robot.

[0164] In this case, it is necessary to pause the task currently executed by the dual-arm robot to facilitate recalibration or adjustment of the dual-arm robot.

[0165] Case 3: If the target anomaly level is a severe anomaly, stop the task currently being executed by the dual-arm robot, disconnect the power output of all joint motors of all dual-arm robots, and push an alarm message.

[0166] In this case, to avoid damage to the robot body or the surrounding environment, it is necessary to immediately stop the operation of the dual-arm robot. Therefore, an emergency stop needs to be triggered to disconnect the power output of all joint motors and send an alarm signal. Among them, the formula for the emergency stop is:

[0167]

[0168] Among them, represents disconnecting the power output of all joint motors.

[0169] In the third embodiment of the present application, by determining the target anomaly level of the dual-arm robot according to the calculated error and the error ranges corresponding to multiple preset anomaly levels, and based on the target anomaly level of the arm robot, the means for abnormal handling measures of the dual-arm robot are used to perform fine-grained control on the dual-arm robot, ensuring the safety and stability of the operation of the dual-arm robot.

[0170] Furthermore, the fourth embodiment of the present application provides an abnormal recognition method for a dual-arm robot. On the basis of the above embodiments, the method provided in this embodiment further includes:

[0171] Step 1: Obtain new fused feature data and abnormal situations generated during the operation of the dual-arm robot;

[0172] Step 2: Optimize and train the anomaly detection model according to the new fused feature data and abnormal situations to obtain an optimized anomaly detection model.

[0173] In this embodiment, during the operation of the dual-arm robot, every time a batch of new fused feature data is obtained, an incremental learning mechanism is triggered to perform an online update on the anomaly detection model. It should be understood that the model does not need to be trained from scratch, but only needs to be fine-tuned on the basis of the existing parameters.

[0174] Specifically, the online Stochastic Gradient Descent (SGD) algorithm can be used to perform small-step online updates on the parameters of the model. It can be specifically represented by the following formula:

[0175]

[0176] Among them, represents the parameters of the current model; represents the learning rate; represents the gradient of the loss function of the current batch of data, Represents the new model parameters. Optionally, the loss function used in the optimization training phase can be the same as the loss function used in the initial training phase. For example, the mean squared error function can be selected as the loss function for optimizing the training of the autoencoder.

[0177] Optionally, in the incremental learning mechanism, it is necessary to prevent the anomaly detection model from overfitting to new data and forgetting the previous data features. Regularization terms or methods based on experience replay can be used to achieve this. Exemplarily, the loss function after introducing the regularization term is:

[0178]

[0179] Where Represents the regularization coefficient, which is used to ensure that the model does not overly bias towards new data, Represents the newly input fused feature data, Represents the reconstructed data corresponding to the newly fused feature data.

[0180] The anomaly recognition method for the dual-arm robot provided in this embodiment uses an incremental learning mechanism to obtain the newly fused feature data and anomaly situations generated during the operation of the dual-arm robot. According to the newly fused feature data and anomaly situations, the anomaly detection model is optimized and trained through the SGD algorithm to obtain an optimized anomaly detection model, thereby online updating the anomaly detection model, improving the adaptive ability and generalization ability of the model, ensuring that the system can adapt to new tasks and environments, and avoiding model obsolescence or performance degradation.

[0181] Furthermore, this application also provides a shared deep neural network architecture, where the underlying network shares the weights for feature extraction and is used for processing multi-modal data, while the high-level network can be fine-tuned for different tasks.

[0182] The shared network part extracts the common features of various tasks:

[0183]

[0184] Where Is the weight of the shared layer, Is the bias term, Is the multi-modal input data, Represents the extracted common features.

[0185] Task-specific output layer: For each task (such as different working modes of the dual-arm robot: assembly, handling, etc.), a dedicated output layer is used to handle the requirements of specific tasks:

[0186]

[0187] Where, and are the exclusive weights and biases for each task.

[0188] Multi-task loss function: The total loss function for multi-task learning is the weighted sum of the losses of multiple tasks:

[0189]

[0190] where, represents the number of tasks; represents the loss function of the -th task; represents the weight of the -th task, which is used to balance the importance of different tasks; represents the total loss function of the deep neural network.

[0191] Furthermore, Embodiment 4 of the present application provides an abnormal recognition method for a dual-arm robot. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0192] In this embodiment, an experiment is designed to verify the beneficial effects of the abnormal recognition method for the dual-arm robot provided by the present application. The experiment mainly collects multi-modal data such as vision, force feedback, and position sensors during the operation of the dual-arm robot, and performs feature extraction, data fusion, abnormal detection, and incremental learning optimization. The purpose of the experiment is to evaluate the abnormal recognition ability of the dual-arm robot under different load and task scenarios, and to verify the advantages of multi-modal data fusion and incremental learning of the invention.

[0193] Experimental object: A dual-arm robot equipped with the following sensors: force / torque sensors installed at each joint, an RGB-D vision sensor on the end effector, joint position sensors, and an IMU for monitoring the robot's posture. Each sensor collects the running state data of the robot in real time, including position, angular velocity, posture, load information, force feedback data, etc.

[0194] Experimental preparation: Sensor configuration: Six-axis force / torque sensors are installed on the joints of the dual-arm robot to monitor the interaction forces between joints. A vision sensor (RGB-D camera) is installed on the end effector of the robot to collect three-dimensional visual information of the environment and objects. The position sensors obtain the angle, position, and speed information of each joint in real time, and the IMU is used to record the posture and motion state of the robot during task execution.

[0195] During the experiment, each sensor continuously collected data. The experiment was run 50 times in different task scenarios, covering tasks such as material handling, assembly, and fine operation. The sensor acquisition frequency was 100 Hz. When collecting data, accurate timestamps were attached, and time synchronization was performed through a unified clock source. After preprocessing, the data was denoised and normalized to a unified scale to ensure the consistency of multimodal data in different tasks and scenarios.

[0196] Anomaly detection model: An anomaly detection model was used to detect anomalies in robot operations. By learning the data patterns of normal robot operations, the anomaly detection model can identify abnormal situations that deviate from the normal pattern. In the experiment, the robot performed simulations of normal and abnormal operations. The anomalies included joint failures, load overloading, and path deviation, etc. The experiment updated the anomaly detection model through a continuous data collection mechanism. After preprocessing, the new data was used in the online stochastic gradient descent algorithm for small-step updates of the model to ensure that the model can adapt to new working environments and task changes. The incremental learning effect of the model was verified in different task scenarios in the experiment. Some experimental results are shown in Table 1.

[0197] Table 1 Partial experimental data display table

[0198]

[0199] From the above experimental data, it can be seen that the anomaly detection method for the dual-arm robot adopting the content of the present invention has significant advantages in various complex task scenarios.

[0200] First of all, the performance of the visual reconstruction error in the table in different tasks shows that the reconstruction error of the robot increases significantly in the scenarios of load overloading and joint failure, reaching 4.8% and 5.6% respectively, which is in sharp contrast with 0.9% under normal operation. This result proves that by extracting visual features through the CNN convolutional neural network and performing data fusion, the abnormal behaviors of the dual-arm robot in tasks can be effectively captured.

[0201] At the same time, the data of the position sensor reconstruction error and the IMU angular velocity also provide strong support. In Task 5 (overload test) and Task 6 (joint failure), the position sensor errors reached 2.0 degrees and 2.8 degrees respectively, while it was only 0.2 degrees under normal operation; similar results were also shown in the anomaly detection of the IMU angular velocity.

[0202] According to the above content, through the fusion of multimodal data, the abnormal behaviors in the robot motion can be effectively captured, so as to perform early fault detection.

[0203] In addition, the effect of incremental learning optimization is also verified by the data of task completion time. As the experimental tasks progress, through continuous data updates and online model optimization, the task completion time gradually shortens. For example, in Task 2 and Task 4, the task completion times after model optimization are 180 seconds and 130 seconds respectively, and in complex load and fault scenarios, the task completion time also significantly decreases. Especially in Task 7 (normal operation), the model achieved a fast operation of 110 seconds after incremental learning, demonstrating the significant advantages of the invention in improving task efficiency and adapting to new scenarios.

[0204] Figure 2 FIG. is a schematic structural diagram of an abnormal recognition device for a dual-arm robot provided in Embodiment 5 of the present application, as Figure 2 shown, the abnormal recognition device 20 for a dual-arm robot provided in this embodiment includes:

[0205] A first acquisition unit 201, configured to acquire a three-dimensional image and time-series data of the environment where the dual-arm robot is located, where the time-series data includes pose information and joint information of the dual-arm robot;

[0206] A fusion unit 202, configured to perform feature extraction and data fusion on the three-dimensional image and the time-series data to obtain fusion feature data;

[0207] A detection unit 203, configured to input the fusion feature data into a preset abnormal detection model to obtain an error between the fusion feature data and the reconstructed feature data, where the abnormal detection model includes an encoder, a decoder, and a calculation module, the encoder is configured to map the input data to a low-dimensional latent space, the decoder is configured to reconstruct the input data from the low-dimensional latent space to obtain the reconstructed feature data, and the calculation module is configured to calculate an error between the input data and the reconstructed feature data;

[0208] A first determination unit 204, configured to determine whether the dual-arm robot has an abnormality according to the error and a preset error threshold.

[0209] The abnormal recognition device 20 for a dual-arm robot provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, and will not be elaborated here in this embodiment.

[0210] Figure 3 FIG. is a schematic structural diagram of an abnormal recognition device for a dual-arm robot provided in Embodiment 6 of the present application, as Figure 3 shown, on the basis of the above embodiment, the abnormal recognition device 20 for a dual-arm robot provided in this embodiment further includes:

[0211] The second determination unit 205 is configured to determine the target anomaly level of the dual-arm robot according to the error and the error ranges corresponding to a plurality of preset anomaly levels.

[0212] The execution unit 206 is configured to determine the target anomaly level of the dual-arm robot according to the error and the error ranges corresponding to a plurality of preset anomaly levels.

[0213] The second acquisition unit 207 is configured to acquire new fusion feature data and anomaly situations generated during the operation of the dual-arm robot.

[0214] The optimization unit 208 is configured to optimize and train the anomaly detection model according to the new fusion feature data and the anomaly situations to obtain an optimized anomaly detection model.

[0215] The first processing unit 209 is configured to perform data stream synchronization and denoising processing on the 3D image and the time series data to obtain a processed 3D image and processed time series data.

[0216] The second processing unit 210 is configured to perform timestamp alignment processing on the data collected by different sensors in the processed time series data to obtain new time series data.

[0217] Correspondingly, the fusion unit 202 is specifically configured to:

[0218] Extract features and perform data fusion on the processed 3D image and the new time series data.

[0219] In a possible implementation manner, the execution unit 206 is specifically configured to: if the target anomaly level is a minor anomaly, adjust the control parameters according to a preset parameter adjustment method to adjust the joint torque or the driving path of the dual-arm robot; if the target anomaly level is a moderate anomaly, pause the task currently executed by the dual-arm robot; if the target anomaly level is a severe anomaly, stop the task currently executed by the dual-arm robot, disconnect the power output of all joint motors of the dual-arm robot, and push an alarm message. In a possible implementation manner, the fusion unit 202 is specifically configured to include:

[0220] Perform feature extraction processing on the 3D image by using a convolutional neural network to obtain visual feature data.

[0221] Extract features from the time series data to obtain time series feature data.

[0222] Stitch the visual feature data and the time series feature data, and perform dimensionality reduction processing on the stitched data to obtain fusion feature data.

[0223] In a possible implementation, the pose information of the dual-arm robot in the first acquisition unit 201 includes: the posture and motion state of the dual-arm robot; the joint information of the dual-arm robot includes: the force feedback, angle, speed, and position of the joints of the dual-arm robot.

[0224] The abnormal recognition device 20 of the dual-arm robot provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0225] Figure 4 It is a schematic structural diagram of the computer device provided in this application. As Figure 4 shown, the computer device 30 provided in this embodiment includes: at least one processor 301 and a memory 302. Optionally, the device 30 further includes a communication component 303. Among them, the processor 301, the memory 302, and the communication component 303 are connected through a bus 304.

[0226] In the specific implementation process, at least one processor 301 executes the computer execution instructions stored in the memory 302, so that at least one processor 301 executes the above method.

[0227] The specific implementation process of the processor 301 can refer to the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0228] This application also provides a computer-readable storage medium, in which computer execution instructions are stored. When the processor executes the computer execution instructions, the above method is implemented.

[0229] This application also provides a computer program product, including a computer program, which implements the above method when executed by the processor.

[0230] Finally, it should be noted that: those skilled in the art will easily think of other implementation schemes of the present invention after considering the specification and practicing the invention disclosed here. The present invention aims to cover any variations, uses, or adaptive changes of the present invention. These variations, uses, or adaptive changes follow the general principles of the present invention and include the common general knowledge or conventional technical means in the technical field not disclosed in the present invention. It is not limited to the exact structure described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A method for identifying anomalies of a dual-arm robot, characterized in that: include: Acquire a three-dimensional image and time series data of the environment in which the dual-arm robot is located, wherein the time series data includes posture information and joint information of the dual-arm robot; Performing feature extraction and data fusion on the three-dimensional image and the time series data to obtain fused feature data; Input the fused feature data into a preset anomaly detection model to obtain an error between the fused feature data and the reconstructed feature data, wherein the anomaly detection model includes an encoder, a decoder and a calculation module, the encoder is used to map the input data to a low-dimensional implicit space, the decoder is used to reconstruct the input data from the low-dimensional implicit space to obtain the reconstructed feature data, and the calculation module is used to calculate the error between the input data and the reconstructed feature data; According to the error and a preset error threshold, it is determined whether the dual-arm robot has an abnormality.

2. The method according to claim 1, characterized in that If it is determined that the dual-arm robot is abnormal, the method further includes: Determine the target abnormality level of the dual-arm robot according to the error and the error ranges corresponding to the preset multiple abnormality levels;.

3. The method according to claim 2, characterized in that The controlling the dual-arm robot to execute corresponding abnormality handling measures according to the target abnormality level includes: Controlling the dual-arm robot to execute corresponding abnormality handling measures according to the target abnormality level; if the target abnormality level is a slight abnormality, adjusting the control parameters according to a preset parameter adjustment method to adjust the joint torque or driving path of the dual-arm robot; If the target abnormality level is a moderate abnormality, the task currently being performed by the dual-arm robot is suspended; If the target abnormality level is a serious abnormality, the task currently executed by the dual-arm robot is stopped, the power output of all joint motors of the dual-arm robot is disconnected, and an alarm message is pushed.

4. The method according to any one of claims 1 to 3, characterized in that: The step of extracting features and fusing data on the three-dimensional image and the time series data to obtain fused feature data includes: Using a convolutional neural network to perform feature extraction processing on the three-dimensional image to obtain visual feature data; Extracting features from the time series data to obtain time series feature data; The visual feature data and the time series feature data are spliced, and the spliced ​​data are subjected to dimensionality reduction processing to obtain fused feature data.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire new fusion feature data and abnormal conditions generated during the operation of the dual-arm robot; The anomaly detection model is optimized and trained according to the new fused feature data and the anomaly situation to obtain an optimized anomaly detection model.

6. The method according to any one of claims 1 to 3, characterized in that: The position information of the dual-arm robot includes: the posture and motion state of the dual-arm robot; The joint information of the dual-arm robot includes: force feedback, angle, speed and position of the dual-arm robot joints.

7. The method according to any one of claims 1 to 3, characterized in that: Before extracting features and fusing data on the three-dimensional image and the time series data, the method further includes: Performing data stream synchronization and denoising processing on the three-dimensional image and the time series data to obtain a processed three-dimensional image and processed time series data; Performing time stamp alignment processing on the data collected by different sensors in the processed time series data to obtain new time series data; Accordingly, the feature extraction and data fusion of the three-dimensional image and the time series data includes: Feature extraction and data fusion are performed on the processed three-dimensional image and the new time series data.

8. An abnormality recognition device for a dual-arm robot, characterized in that: include: An acquisition unit, used to acquire a three-dimensional image and time series data of the environment in which the dual-arm robot is located, wherein the time series data includes position information and joint motion information of the dual-arm robot; A fusion unit, used for performing feature extraction and data fusion on the three-dimensional image and the time series data to obtain fused feature data; A detection unit, used for inputting the fused feature data into a preset anomaly detection model to obtain an error between the fused feature data and the reconstructed feature data, wherein the anomaly detection model includes an encoder, a decoder and a calculation module, the encoder is used for mapping the input data to a low-dimensional implicit space, the decoder is used for reconstructing the input data from the low-dimensional implicit space to obtain the reconstructed feature data, and the calculation module is used for calculating the error between the input data and the reconstructed feature data; A determination unit is used to determine whether the dual-arm robot has an abnormality based on the error and a preset error threshold.

9. A computer device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images