Industrial robot and environment contact event identification method based on cross-modal learning

Abstract

The invention discloses an industrial robot and environment contact event identification method based on cross-modal learning, and the method comprises the steps: collecting a visual image and a force sense time sequence of the tail end of a robot, and extracting visual features and force sense features through convolution operation and continuous wavelet transformation; a force sense feature driving offset generation network is utilized to calculate a time offset scalar, time dimension deformable convolution is executed on visual features, and time-space accurate alignment of visual and tactile features is achieved; according to kinematics parameters of the robot, kinetic theoretical stress is calculated, a physical residual vector is obtained through difference between the kinetic theoretical stress and actually-measured force, and inertia force interference generated by motion of the robot is eliminated; mapping the physical residual error, fusing the mapped physical residual error with the aligned visual tactile features, and inputting the fused physical residual error into a neural network to identify a contact type; according to the method, a physical consistency constraint and anti-occlusion reconstruction mechanism is introduced in a model training stage, so that the problems that multi-modal data are asynchronous, and the model lacks physical interpretability and visual occlusion are effectively solved, and the accuracy and robustness of event recognition are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial robot perception and artificial intelligence, and particularly relates to an industrial robot and environment contact event recognition method based on cross-modal learning. BACKGROUND

[0002] With the in-depth development of intelligent manufacturing and Industry 4.0, industrial robots are increasingly popular in unstructured scenarios such as precision assembly, surface polishing, and human-robot collaboration. In these complex tasks, accurately identifying contact events between the robot and the environment (such as collision, friction, and flexible contact) is a key prerequisite for achieving flexible control and safe operation. Traditional contact detection mainly relies on single force sensor to set static threshold, which is difficult to cope with complex working conditions. In recent years, cross-modal perception technology based on deep learning has gradually become a research hotspot. This technology aims to fuse the environmental semantic information of visual sensors and the contact physical information of force sensors to build a more comprehensive environmental cognition model. The current mainstream method usually uses convolutional neural networks (CNN) or long short-term memory networks (LSTM) to extract visual and force features respectively, and performs cascaded fusion at the feature layer to improve recognition accuracy.

[0003] However, existing cross-modal contact recognition technology still faces severe challenges in practical applications. First, visual data (low frequency, high latency) and force data (high frequency, real-time) are inherently heterogeneous. The existing simple interpolation or fixed time window alignment method cannot solve the non-synchronization problem in the time dimension of the two, making it difficult to accurately match the transient impact features in high-frequency force with visual frames. Second, most deep learning models are pure data-driven "black box" models, lacking physical interpretability, and often cannot distinguish between inertial forces generated by high-speed robot motion and real external contact forces, leading the model to violate Newton-Euler dynamics laws. In addition, the robot's end often blocks the camera's line of sight in industrial sites, but existing methods lack a mechanism to reconstruct features in the visual missing area using force signals, resulting in a significant decrease in the system's robustness under visual occlusion. SUMMARY

[0004] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, abstract and title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0005] In view of the above existing problems, the present application is proposed. Therefore, the present application provides an industrial robot and environment contact event recognition method based on cross-modal learning to solve the problems proposed in the background art.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a method for recognizing industrial robot-environment contact events based on cross-modal learning, comprising: Visual image sequences from the end-effector camera of an industrial robot are acquired, and multi-dimensional force sensing time-series data from a torque sensor are acquired simultaneously. Convolution operations are performed on the visual image sequences to extract visual feature tensors, and continuous wavelet transforms are performed on the multi-dimensional force sensing time-series data to generate force sensing feature tensors containing time-frequency domain information. The force feature tensor is input into the offset generation network to calculate the time offset scalar corresponding to the visual sampling time. Based on the time offset scalar, a deformable convolution operation in the time dimension is performed on the visual feature tensor to obtain the spatiotemporally aligned visual features. The dynamic theoretical force is calculated based on the kinematic parameters of the industrial robot. The difference between the theoretical force and the force measured by the sensor is calculated to obtain the physical residual vector. The physical residual vector is mapped to the feature space and then spliced ​​and fused with the force perception feature tensor and the spatiotemporally aligned visual features to generate multimodal fusion features. The multimodal fusion features are input into a classification neural network to output the current contact event category between the industrial robot and the environment.

[0007] As a preferred embodiment of the industrial robot-environment contact event recognition method based on cross-modal learning described in this invention, the step of performing continuous wavelet transform on the multidimensional force perception time series data includes: For each dimension of the multidimensional force perception time series data, determine the mother wavelet function and the corresponding scaling factor sequence and translation factor sequence at different frequencies; The wavelet coefficients are calculated by performing an integral transform on the force perception time series data using the mother wavelet function. Calculate the square of the modulus of the wavelet coefficients to generate a time-frequency spectrum; A downsampling convolution operation is performed on the time-frequency spectrum to adjust its spatial dimension to match the spatial dimension of the visual feature tensor, thereby obtaining the force feature tensor.

[0008] As a preferred embodiment of the industrial robot-environment contact event recognition method based on cross-modal learning described in this invention, the step of calculating the time offset scalar corresponding to the visual sampling time includes: The force perception feature tensor is input into a one-dimensional convolutional network; Extract gradient change features from force sensory signals; The regression layer outputs a time offset scalar corresponding to each visual feature sampling point, the time offset scalar being a non-integer value representing the displacement of the visual sampling point on the time axis.

[0009] As a preferred solution of the industrial robot and environment contact event recognition method based on cross-modal learning, wherein: characterized in that, the step of performing a deformable convolution operation in the time dimension on the visual feature tensor based on the time offset scalar comprises: Based on the visual feature sampling position at the current time, superimpose the time offset scalar to determine the target sampling coordinates; Based on the feature values of the adjacent frames of the target sampling coordinates, perform a bilinear interpolation operation to calculate the interpolated visual features at the target sampling coordinates; The interpolated visual features are weighted and summed using the convolution kernel weights to output the visual features corrected on the time axis.

[0010] As a preferred solution of the industrial robot and environment contact event recognition method based on cross-modal learning, wherein: the step of calculating the physical residual vector by calculating the difference between the theoretical force and the measured force of the sensor based on the kinematic parameters of the industrial robot comprises: Obtain the real-time joint angle, joint angular velocity and joint angular acceleration of each joint of the robot; Substitute the joint angle, joint angular velocity and joint angular acceleration into the rigid body dynamics equation, combine the inertia matrix parameters, Coriolis force parameters and gravity parameters of the robot, and calculate the theoretical dynamics force of the robot end effector; Perform subtraction operation to calculate the difference between the measured force collected by the torque sensor and the theoretical dynamics force, and generate a physical residual vector.

[0011] As a preferred solution of the industrial robot and environment contact event recognition method based on cross-modal learning, wherein: the classification neural network is obtained by the following steps: Construct a physical force decoder, and input the multi-modal fusion features into the physical force decoder; Output a predicted contact force vector through the physical force decoder; Calculate the Euclidean distance between the predicted contact force vector and the physical residual vector as a physical consistency loss value; Minimize the physical consistency loss value by using a back propagation algorithm to adjust the weight parameters of the classification neural network.

[0012] As a preferred scheme of the industrial robot and environment contact event recognition method based on cross-modal learning provided in the present application, wherein: the training process of the classification neural network further comprises an anti-occlusion reconstruction training step, which comprises: Determine the region corresponding to the robot end effector in the visual image sequence, and set the pixel value of the region to zero to generate a masked visual feature; Input the force sensation feature tensor and the masked visual feature into the cross-modal reconstruction decoder; Output the pixel prediction value of the region set to zero by the cross-modal reconstruction decoder; Calculate the difference between the pixel prediction value and the pixel value of the corresponding region in the original visual image sequence, and adjust the network parameters.

[0013] As a preferred scheme of the industrial robot and environment contact event recognition method based on cross-modal learning provided in the present application, wherein: the step of outputting the pixel prediction value of the region set to zero by the cross-modal reconstruction decoder comprises: Map the force sensation feature tensor to a query vector; Map the masked visual feature to a key vector and a value vector; Calculate the attention weight matrix between the query vector and the key vector; Based on the attention weight matrix, weight aggregate the value vector to generate a reconstructed visual feature representation, and map it back to the pixel space to obtain the pixel prediction value.

[0014] As a preferred scheme of the industrial robot and environment contact event recognition method based on cross-modal learning provided in the present application, wherein: the step of outputting the current contact event class of the industrial robot and the environment further comprises: Calculate the deviation value of the predicted contact force vector and the physical residual vector, or calculate the reconstruction error value of the pixel prediction value and the original pixel value; Compare the deviation value or the reconstruction error value with a preset threshold value; If the deviation value or the reconstruction error value is greater than the preset threshold value, output a low confidence identifier while outputting the contact event class.

[0015] As a preferred scheme of the industrial robot and environment contact event recognition method based on cross-modal learning provided in the present application, wherein: the training process of the classification neural network comprises calculating a joint loss function, and the step of calculating the joint loss function comprises: Calculate the cross-entropy loss value between the output contact event class and the true class label; Calculate the regression loss value between the predicted force decoded from the multi-modal fusion feature and the physical residual vector; calculating a pixel difference loss value between the reconstructed image of the masked region and the original image; performing weighted summation on the cross-entropy loss value, the regression loss value and the pixel difference loss value to obtain a total loss value.

[0016] Compared with the prior art, the scheme has the beneficial effects that: 1、The scheme discards the traditional simple interpolation or fixed window alignment method, introduces an offset generation network and a time dimension deformable convolution, adaptively calculates a time offset scalar of a visual sampling point by mining the transient mutation characteristics (such as the collision moment) in the high-frequency force signal, and accordingly performs non-rigid time axis correction on the visual features, effectively overcoming the heterogeneous contradiction between the low frame rate and high delay of the visual sensor and the high frequency and real-time of the force sensor, and improving the feature matching accuracy in the fast dynamic interaction process. 2、In view of the defect that a pure data-driven model cannot distinguish between "inertial force" and "contact force", the scheme combines a robot rigid body dynamics equation to calculate a physical residual vector, subtracts the measured force from the theoretical dynamics force to strip off the interference components generated by the acceleration and deceleration of the industrial robot, gravity and Coriolis force, and extracts pure external contact signals, thereby improving the recognition accuracy of the model in the high-speed motion scene, and making the neural network have the explainability conforming to Newtonian mechanics through physical consistency constraint training. 3、In addition, in view of the problem of occlusion of the end effector from the camera view in the industrial field, the scheme designs an anti-occlusion reconstruction training step, uses the force sensation features as "indexes", reconstructs the masked (occluded) visual region in the latent space through the attention mechanism, so that the model can "imagine" the contact state according to the subtle changes of the force sensation signals when the visual information is missing, thereby greatly enhancing the reliability of the system in the complex unstructured environment. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor. Among them: Figure 1 The general flowchart of the industrial robot and environment contact event recognition method based on cross-modal learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present application.

[0019] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. The present application, however, can be practiced in a variety of ways beyond the specific embodiments described herein without departing from the scope of the present application. It can be appreciated by those skilled in the art that the present application can be implemented in other different ways, and therefore the present application is not limited to the specific embodiments disclosed below.

[0020] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an embodiment that is independent of or selected from other embodiments.

[0021] The present application is described in detail in conjunction with the schematic drawings. In the detailed description of the embodiments of the present application, the cross-sectional view of the device structure is partially enlarged without the general proportion for the convenience of description, and the schematic drawings are only examples, which should not limit the scope of protection of the present application. In addition, three-dimensional spatial dimensions including length, width and depth should be included in actual manufacturing.

[0022] Meanwhile, in the description of the present application, it should be noted that the terms "upper, lower, inner and outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first, second or third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0023] Unless otherwise specifically defined and limited, the terms "mounting, connecting, connection" in the present application should be understood broadly, for example: it can be fixed connection, detachable connection or integral connection; it can also be mechanical connection, electrical connection or direct connection, it can also be indirectly connected through intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0024] Example 1 Reference Figure 1For the first embodiment of the present application, the embodiment provides an industrial robot and environment contact event recognition method based on cross-modal learning, comprising: S1, collecting a visual image sequence of an industrial robot end camera, and synchronously collecting a multi-dimensional force time series data of a torque sensor, performing convolution operation on the visual image sequence to extract a visual feature tensor, and simultaneously performing continuous wavelet transform on the multi-dimensional force time series data to generate a force feature tensor containing time-frequency domain information; Further, a visual sensor (such as an RGB industrial camera) and a six-dimensional force / torque sensor installed on an end effector of an industrial robot are started to collect a video stream of the industrial robot during operation at a fixed frame rate (for example, 30 fps or 60 fps), and the collected visual image sequence is defined as , wherein, is the total number of frames in the time window, and each frame image corresponds to the height, width and RGB color channel of the image respectively; it should be noted that since the contact force signal often contains high-frequency transient characteristics, the sampling frequency of the force sensor is usually much higher than that of the visual sensor (for example, 1000 Hz), so the multi-dimensional force time series data needs to be synchronously collected, and the multi-dimensional force time series data is defined as , wherein, 6 represents the force along the axis and the torque around the axis, is the total number of high-frequency sampling points in the time window; Further, in order to obtain the spatial semantic information of the environment, the visual image sequence is input into a convolutional neural network (CNN) for feature encoding, and in the embodiment, ResNet-50 or VGG-16 without full connection layer can be selected as the backbone network; Specifically, the convolutional neural network performs layer-by-layer convolution and pooling operation on each frame image to extract deep spatial features, and the finally output visual feature tensor is denoted as , wherein, is the number of feature channels, and are the feature map height and width after downsampling; It should be noted that the visual feature tensor retains the spatial structure information of the industrial robot in the operation scene; Further, in order to capture the vibration signal (such as scratch, micro-collision) that the robot may generate in the operation scene, which is extremely weak and short at the moment of contact, in the embodiment, the Fourier transform which loses time information is not used, and the one-dimensional force signal is mapped into a two-dimensional time-frequency spectrum by using continuous wavelet transform. Specifically, for each dimension of the multi-dimensional force sense time series data , the following sub-steps are performed: S101, determine the mother wavelet and the transform parameter: Preferably, the Morlet wavelet with Gaussian envelope characteristics is used as the mother wavelet function , because the Morlet wavelet with Gaussian envelope characteristics has good localization characteristics in time domain and frequency domain, and is suitable for analyzing non-stationary impact signals; by setting a series of scale factor sequences corresponding to different frequencies and a series of translation factor sequences corresponding to different time positions ; S102, perform integral transform: Convolution integral of the force sense signal is performed by using the mother wavelet function to calculate the wavelet coefficient , and the calculation formula is as follows: Wherein, represents the wavelet coefficient at the th force sense dimension, the scale is , and the position is ; is the scale factor, which is inversely proportional to the frequency ( ), and is used to control the stretching and contraction of the wavelet, and a smaller corresponds to a high-frequency component, and a larger corresponds to a low-frequency component; is the translation factor, which is used to control the sliding position of the wavelet on the time axis to realize time domain positioning; is the original force sense time series signal of the th dimension; is the complex conjugate function of the mother wavelet function ; is the normalization coefficient, which is used to ensure the energy conservation of the transform under different scales; S103, generate time-frequency spectrum: Considering that the wavelet coefficient may be a complex number, in order to obtain the energy distribution, the square of the modulus of the wavelet coefficient needs to be calculated, and the time-frequency spectrum is generated: ​​It should be noted that at this time, for the 6-dimensional force signal, 6 two-dimensional images reflecting the "time-frequency-energy" distribution are generated; S104, downsampling and dimension matching: Since the generated time-frequency spectrum usually has high resolution and the aspect ratio is inconsistent with the visual features, in order to subsequent cross-modal fusion, the above-mentioned 6-dimensional time-frequency spectrum is stacked in the channel dimension to form an initial 6-channel feature map, and is processed through a downsampling module composed of convolution layers; Specifically, a convolution kernel with a step greater than 1 is used to perform convolution operation on the time-frequency spectrum, and the spatial dimension thereof is adjusted so that the final output dimension matches the spatial dimension of the visual feature tensor ), and finally a force sensation feature tensor containing rich time-frequency domain information is obtained, denoted as , wherein is the number of channels of the force sensation feature, which is 6 in this embodiment; S2, inputting the force sensation feature tensor into the offset generation network to calculate the time offset scalar corresponding to the visual sampling time, and performing deformable convolution operation on the visual feature tensor in the time dimension based on the time offset scalar to obtain the spatio-temporally aligned visual feature; It should be noted that since the acquisition of the visual signal usually has a delay (such as exposure time, transmission delay) and a low sampling rate, and the response of the force sensation signal to the contact event is almost instantaneous and has a very high sampling rate, directly fusing the two at the same timestamp will cause feature misalignment, therefore, this step adaptively finds the best matching point of the visual signal on the time axis by learning the abrupt feature in the force sensation signal; Further, in order to determine how much distance each frame of visual image should move on the time axis to match the current force sensation state, the present scheme constructs an offset generation network based on one-dimensional convolution (1D-CNN), which includes multiple one-dimensional convolution layers, activation functions and a fully connected regression layer. The convolution layer and the activation function therein are used to extract high-frequency gradient change features (such as force abruptness at the contact moment and oscillation generated by friction) in the force sensation signal, and the fully connected regression layer is used to output a continuous real value; For example, in this embodiment, the structure of the offset generation network can be designed as follows: the input is a force sensation feature sequence with a dimension of after global average pooling (wherein is the batch size, is the time step, is the number of feature channels), the feature sequence is first passed through three consecutive one-dimensional convolution layers, and the specific parameters can be set as follows: ​First layer: 64 convolutional kernels, kernel size 3, stride 1, padding 1, followed by ReLU activation function; The second layer consists of 128 convolutional kernels, with a kernel size of 3, a stride of 1, padding of 1, followed by a ReLU activation function. The third layer consists of 256 convolutional kernels, with a kernel size of 3, a stride of 1, padding of 1, followed by a ReLU activation function. The convolutional features are then subjected to global temporal average pooling to compress the temporal dimension. Finally, they are passed through a regression head consisting of two fully connected layers. The first fully connected layer maps the feature dimension to 64, and the second fully connected layer outputs a scalar value, namely the time offset scalar. The number of layers, the number of convolutional kernels, and their sizes mentioned above are just examples and can be adjusted according to specific application scenarios. It should be noted that the high-frequency gradient change characteristics imply the precise moment when the contact event occurs; Specifically, the force feature tensor generated in step S1... As input to this offset generation network, in order to focus on changes in the time dimension, the first step is to... Perform global average pooling along the spatial dimension to compress the dimension to... This yields a sequence of force feature vectors containing only time-series information. Then, for the first... At each sampling time, the corresponding time offset scalar is calculated using the offset generation network. Its computational model is expressed as: in, Corresponding to the The time offset scalar of the frame visual features is a non-integer value (e.g., 0.5, -1.2), representing the number of frames the visual sampling point should move forward or backward on the time axis to match the current force perception state. This represents the mapping function of the offset generation network, which learns through backpropagation to minimize cross-modal alignment error; Indicates the first A localized force sensory feature segment centered on a specific moment; Furthermore, due to the calculated time offset scalar The values ​​are usually non-integers, which means that the target sampling position does not fall on the actual collected discrete video frames (i.e., it is located between two frames), so the feature value cannot be read directly. To solve this problem, the present invention uses a differentiable bilinear interpolation algorithm to calculate the features of the virtual sampling point. Specifically, firstly, based on the current visual feature sampling time... Using this as a baseline, and superimposing offsets, the target sampling coordinates on the time axis are determined. : Then, based on the target sampling coordinates Adjacent previous frame index and the next frame index Calculate the interpolated visual features at this location. : in, Indicates position on a continuous time axis Reconstructed visual features at the location; The original visual feature tensor extracted in step S1 is in the... The feature value of a frame (integer index); The interpolation kernel function (such as a linear kernel) is defined as follows: This is used to assign different weights to adjacent frames based on their distance. Furthermore, after obtaining the interpolated visual features at precise moments, a deformable convolution operation in the temporal dimension is performed to aggregate visual information within the local temporal window and complete the final feature alignment. Specifically, define a temporal convolution kernel with weights as follows: The interpolated visual features are weighted and summed to output the spatiotemporally aligned visual features. : in, It is a visual feature that has been corrected for time axis distortion and synchronized with force signals at the microsecond level; The size of the temporal convolution kernel determines the size of the receptive field; The first in the convolution kernel Learnable weight parameters for each position; This means that each sampling point covered by the convolution kernel has an independent dynamic offset, thereby achieving non-rigid time axis deformation, which can flexibly adapt to non-linear time misalignment caused by the variable speed motion of industrial robots. S3. Calculate the dynamic theoretical force based on the kinematic parameters of the industrial robot, calculate the difference between the theoretical force and the actual force measured by the sensor to obtain the physical residual vector, map the physical residual vector to the feature space, and stitch and fuse it with the force perception feature tensor and the spatiotemporally aligned visual features to generate multimodal fusion features. It should be noted that existing pure data-driven models often have difficulty distinguishing whether changes in sensor values ​​are caused by inertial forces resulting from the robot's own high-speed acceleration and deceleration, or by actual contact forces caused by contact with the external environment. In order to overcome the shortcomings of this "black box" model, this step introduces prior physical knowledge to construct physical residual branches. Furthermore, the dynamic parameters of the industrial robot are obtained in advance; Specifically, a parameterized dynamic model of the robot can be established using the Newton-Euler method, and the robot can be controlled to run a preset excitation trajectory (such as a Fourier series trajectory). Joint position, velocity and torque data can be collected, and the inertial matrix parameters, Coriolis force parameters and gravity parameters can be identified offline using the least squares method or genetic algorithm. The identified parameters can then be stored in the controller or edge computing unit. Furthermore, through the industrial robot's controller communication interface (such as EtherCAT or Profinet), the encoder feedback data of each joint of the robot can be read in real time, defining the time... The joint state vector of the robot; Specifically, the joint state vector includes: real-time joint angles. Joint angular velocity and joint angular acceleration ,in, For the number of degrees of freedom of an industrial robot (e.g., a 6-axis robotic arm), then ); Furthermore, based on the robot's rigid body dynamics model, the torque sensor should theoretically measure the force value (i.e. the resultant force generated by the end-effector's load gravity, inertial force, and Coriolis force) under the current motion state, assuming that the end-effector has no external contact. Specifically, the dynamic equations are established using the Lagrange method or the Newton-Euler method, and then mapped to the Cartesian space of the end effector to calculate the theoretical dynamic forces acting on the end effector. : Alternatively, the calculation can be performed by combining the joint space dynamics equations with the Jacobian matrix: in, For a moment The theoretical dynamic force vector (6-dimensional, containing 3 forces and 3 moments) characterizes the dynamics caused solely by its own motion; These are joint angle, angular velocity, and angular acceleration, respectively. The parameters of the robot's inertia matrix describe the influence of the mass distribution of each link in the robotic arm on the motion. The parameters for the Coriolis force and centrifugal force matrix describe the nonlinear forces generated by multi-joint coupled motion; These are gravity vector parameters, describing the effect of gravity on each joint; The theoretical joint torque required to maintain the current motion; This is the transpose of the robot's Jacobian matrix; This represents a pseudo-inverse operation used to map the torque in the joint space to the end-effector Cartesian space. Furthermore, a subtraction operation is performed to convert the actual measured force collected by the torque sensor into a value. With the calculated theoretical dynamic forces Perform difference analysis to generate the physical residual vector. : in, This is the physical residual vector, indicating that the vector eliminates spurious contact signals generated by the robot's own motion (such as sudden stops or rapid turns). Furthermore, theoretically, when there is no external contact, When a collision or contact occurs, It can accurately reflect the real external forces; Furthermore, due to the physical residual vector It is a low-dimensional vector ( The visual feature tensors generated by steps S1 and S2 and force feature tensor It is a high-dimensional tensor, therefore dimensional projection and spatial alignment are required; Specifically, a feature projection layer composed of a multilayer perceptron (MLP) is constructed to map the 6-dimensional physical residual vector to a high-dimensional feature space, thereby obtaining the physical feature vector. ,in, This represents the number of channels after projection; simultaneously, in order to align with the spatial dimension of visual features ( Compatible with physical feature vectors The physical feature map is generated by copying and expanding it in the spatial dimension. Then, in the channel dimension, the visual features after spatiotemporal alignment are... Force feature tensor and physical feature map The features are then stitched together to generate multimodal fusion features. : in, The final generated multimodal fusion feature has the following dimensions: ; It should be noted that this multimodal fusion feature simultaneously integrates visual semantic information, temporal and frequency details of force perception, and dynamic residual information that conforms to the laws of physics. S4. Input the multimodal fusion features into the classification neural network and output the current contact event category between the industrial robot and the environment; It should be noted that, in order to improve the robustness of the model under complex conditions (such as visual occlusion, lighting changes, and sensor noise), this step not only includes the classification reasoning process, but also integrates a training mechanism based on physical constraints and image reconstruction. Furthermore, the dimension is The multimodal fusion features are input into a classification neural network, which typically consists of a global average pooling (GAP) layer and a fully connected layer. It should be noted that during the actual recognition process, the classification neural network will be in inference mode, and the network weights are fixed. Specifically, firstly, a global average pooling layer is used to compress the spatial dimension of the feature tensor to obtain a global feature vector. Then, a fully connected layer is used to map the features to the class space, and the Softmax function is used to calculate the probability distribution of each contact event category (e.g., free motion, hard collision, flexible contact, grinding friction, assembly jamming, etc.). : in, This represents the output probability distribution vector, the length of which is equal to the total number of preset contact event categories; and These are the weight matrix and bias term of the classification layer, respectively; This is a global average pooling operation; Finally, the category with the highest probability value is selected as the category of the contact event between the industrial robot and the environment at the current moment; Furthermore, in order to prevent the model from giving incorrect "high confidence" predictions when the data is abnormal, the present invention introduces a self-evaluation mechanism when outputting the category. Specifically, the contact force vector predicted by the computational model. The physical residual vector calculated in step S3 Deviation between Alternatively, calculate the reconstruction error between the visually reconstructed pixel values ​​and the original pixel values. And set a preset threshold. Execute the following judgment logic: It should be noted that if the deviation exceeds the preset threshold, it indicates that the current model prediction violates the laws of physics and dynamics (for example, the model predicts no contact, but the physical residual shows a huge force), or that a serious anomaly has occurred in the visual scene. In this case, the output category will be adjusted accordingly. At the same time, it will be marked as "low confidence". This prompts the host computer to perform a shutdown check or require manual intervention. Furthermore, in order to enable the neural network to not only fit data but also learn physical laws that conform to Newtonian mechanics, the present invention constructs a physical force decoder during the model training phase. Specifically, firstly, multimodal features are fused. Input to the decoder, output the predicted contact force vector Then, the predicted value is calculated and compared with the physical residual vector calculated based on the dynamic equation. The Euclidean distance between (i.e., the actual external contact forces) serves as the loss of physical consistency. : It should be noted that by minimizing this loss, the features extracted by the neural network are forced to contain information that can explain the physical forces, thereby solving the problem of the "black box" model being uninterpretable. Furthermore, to address the common visual occlusion problem in industrial settings, a cross-modal reconstruction decoder is introduced to "fill in" the occluded visual area using force signals. Specifically, during model training, the pixel values ​​of the regions in the visual image corresponding to the robot's end effector are manually set to zero to generate masked visual features. Then, an attention mechanism is used for feature reconstruction, and the steps of feature reconstruction are as follows: Map the force feature tensor to a query vector ; Mapping mask visual features to key vectors Sum value vector ; Calculate the attention weight matrix and aggregate features: in, This represents the reconstructed visual features. This represents the scaling factor, typically the square root of the key vector dimension, used to prevent gradient vanishing. It is important to emphasize that the determination of the aforementioned end effector region depends on the industrial robot's hand-eye calibration matrix and the robot's real-time forward kinematics. Specifically, the three-dimensional coordinates of the robot's end effector are projected onto the camera's two-dimensional imaging plane, and the bounding box of the end effector in the image is calculated. The region within this bounding box is defined as the mask region. This process is as follows: First, the robot controller obtains the current angles of each joint and uses the forward kinematics model to calculate the three-dimensional pose of the end effector in the robot's base coordinate system. Then, using a pre-calibrated hand-eye relationship matrix, the three-dimensional pose is transformed into the camera coordinate system. Finally, using the camera intrinsic parameter matrix, the three-dimensional points in the camera coordinate system are projected onto the two-dimensional pixel plane of the image, thereby accurately locating the pixel area covered by the end effector in each frame of the image. It should be noted that this formula aims to use force signals as an index to find correlations in unoccluded visual information and infer the content of the occluded area. Specifically, will The pixel prediction value can be obtained by mapping back to pixel space. And based on the pixel prediction value Calculate the pixel difference loss between the corresponding region of the original image and the original image. ; The training process of the classification neural network employs a multi-task joint optimization strategy to calculate the total loss value. : in, This is the cross-entropy loss value, used to measure the accuracy of contact event classification; This is the regression loss value, namely the physical consistency loss mentioned above, used to constrain physical laws; This is the pixel difference loss value, i.e., the reconstruction loss mentioned above, used to improve the ability to resist occlusion; These are the weighting coefficients for each loss term, used to balance the learning weight of different tasks, and ; Furthermore, the total loss value is minimized using the backpropagation algorithm. Update the network weights. After the model training is complete, remove the physics decoder and cross-modal reconstruction decoder (or bypass them), and keep only the classification backbone network for model inference. It should be noted that, through the above steps, the present invention can generate a contact event recognition model that combines high accuracy, physical interpretability, and resistance to environmental interference.

[0025] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0026] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0027] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0028] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0029] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0030] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for recognizing an industrial robot and environment contact event based on cross-modal learning, characterized by, The method comprises the following steps: Collecting a visual image sequence of an industrial robot end camera and synchronously collecting a multi-dimensional force time series data of a torque sensor, performing convolution operation on the visual image sequence to extract a visual feature tensor, and simultaneously performing continuous wavelet transform on the multi-dimensional force time series data to generate a force feature tensor containing time-frequency domain information; Inputting the force feature tensor into an offset generation network to calculate a time offset scalar corresponding to a visual sampling time, and performing a deformable convolution operation on the time dimension of the visual feature tensor based on the time offset scalar to obtain a spatio-temporally aligned visual feature; Calculating a theoretical force based on the kinematic parameters of the industrial robot, calculating a physical residual vector by calculating the difference between the theoretical force and the measured force of the sensor, mapping the physical residual vector to a feature space, and fusing the physical residual vector with the force feature tensor and the spatio-temporally aligned visual feature to generate a multi-modal fusion feature; Inputting the multi-modal fusion feature into a classification neural network to output a current contact event category of the industrial robot and the environment. 2.The industrial robot and environment contact event recognition method based on cross-modal learning of claim 1, wherein, The step of performing continuous wavelet transform on the multi-dimensional force time series data comprises: For each dimension of the multi-dimensional force time series data, determining a mother wavelet function, a scale factor sequence and a translation factor sequence corresponding to different frequencies; Performing integral transform on the force time series data using the mother wavelet function to calculate wavelet coefficients; Calculating the square of the modulus of the wavelet coefficients to generate a time-frequency spectrum; Performing down-sampling convolution operation on the time-frequency spectrum to adjust the spatial dimension of the time-frequency spectrum to match the spatial dimension of the visual feature tensor to obtain the force feature tensor. 3.The industrial robot and environment contact event recognition method based on cross-modal learning of claim 1, wherein, The step of calculating a time offset scalar corresponding to a visual sampling time comprises: Inputting the force feature tensor into a one-dimensional convolution network; Extracting gradient change features in the force signal; Outputting a time offset scalar corresponding to each visual feature sampling point through a regression layer, the time offset scalar being a non-integer value, which represents the displacement of the visual sampling point on the time axis.

4. The industrial robot and environment contact event recognition method based on cross-modal learning of claim 3, wherein, The step of performing a deformable convolution operation on the time dimension of the visual feature tensor based on the time offset scalar comprises: Taking the visual feature sampling position at the current time as a reference, superimposing the time offset scalar to determine a target sampling coordinate; Performing bilinear interpolation operation based on the feature values of the adjacent frames of the target sampling coordinate to calculate the interpolated visual feature at the target sampling coordinate; Performing weighted summation on the interpolated visual feature using convolution kernel weights to output the visual feature corrected on the time axis. 5.The industrial robot and environment contact event recognition method based on cross-modal learning of claim 1, wherein, The step of calculating a theoretical force based on the kinematic parameters of the industrial robot, calculating a physical residual vector by calculating the difference between the theoretical force and the measured force of the sensor, mapping the physical residual vector to a feature space, and fusing the physical residual vector with the force feature tensor and the spatio-temporally aligned visual feature to generate a multi-modal fusion feature comprises: Obtaining real-time joint angles, joint angular velocities and joint angular accelerations of each joint of the robot; Substituting the joint angles, joint angular velocities and joint angular accelerations into the rigid body dynamics equation, combining the inertia matrix parameters, Coriolis force parameters and gravity parameters of the robot to calculate the theoretical dynamic force of the robot end effector; Performing subtraction operation to calculate the difference between the measured force collected by the torque sensor and the theoretical dynamic force, and generate a physical residual vector. 6.The industrial robot and environment contact event recognition method based on cross-modal learning of claim 1, wherein, The classification neural network is pre-trained by the following steps: Building a physical force decoder, and inputting the multi-modal fusion feature into the physical force decoder; Outputting a predicted contact force vector through the physical force decoder; Calculating the Euclidean distance between the predicted contact force vector and the physical residual vector as a physical consistency loss value; Adjusting the weight parameters of the classification neural network by minimizing the physical consistency loss value through a back propagation algorithm. 7.The industrial robot and environment contact event recognition method based on cross-modal learning of claim 1, wherein, The training process of the classification neural network further includes an anti-occlusion reconstruction training step, which includes: Determining the region corresponding to the robot end effector in the visual image sequence, and setting the pixel values of the region to zero to generate a masked visual feature; Inputting the force sensation feature tensor and the masked visual feature into a cross-modal reconstruction decoder; Outputting pixel prediction values for the region set to zero through the cross-modal reconstruction decoder; Calculating the difference between the pixel prediction values and the pixel values of the corresponding region in the original visual image sequence to adjust the network parameters.

8. The industrial robot and environment contact event recognition method based on cross-modal learning of claim 7, wherein, The step of outputting pixel prediction values for the region set to zero through the cross-modal reconstruction decoder includes: Mapping the force sensation feature tensor into a query vector; Mapping the masked visual feature into a key vector and a value vector; Calculating an attention weight matrix between the query vector and the key vector; Based on the attention weight matrix, weighting and aggregating the value vector to generate a reconstructed visual feature representation, and mapping it back to the pixel space to obtain the pixel prediction values.

9. The cross-modal learning based industrial robot and environment contact event recognition method of claim 6 or 8, wherein, The step of outputting the current contact event category of the output industrial robot and the environment further includes: Calculating the deviation value of the predicted contact force vector and the physical residual vector, or calculating the reconstruction error value of the pixel prediction value and the original pixel value; Comparing the deviation value or the reconstruction error value with a preset threshold value; If the deviation value or the reconstruction error value is greater than the preset threshold value, outputting a low confidence identifier while outputting the contact event category. 10.The industrial robot and environment contact event recognition method based on cross-modal learning of claim 1, wherein, The training process of the classification neural network includes calculating a joint loss function, and the step of calculating the joint loss function includes: Calculating a cross-entropy loss value between the output contact event category and the true category label; Calculating a regression loss value between the predicted force decoded from the multi-modal fusion feature and the physical residual vector; Calculating a pixel difference loss value between the reconstructed image of the masked region and the original image; Weighted sum of the cross-entropy loss value, the regression loss value and the pixel difference loss value to obtain a total loss value.

Images