A levit-based industrial robot abnormal joint positioning system and method

Abstract

This invention provides a LeViT-based system and method for locating abnormal joints in industrial robots, comprising a Stem module, a LeViT Stage module, and a classification head module connected in sequence. The Stem module extracts a first feature map from the power spectrum; the LeViT Stage module extracts features from the first feature map, downsampling to reduce the feature map size during extraction to obtain a second feature map; the classification head module performs abnormal pattern classification on the second feature map to obtain the abnormal joint location. This invention can extract and effectively classify abnormal joint features, providing accurate abnormal joint location and reducing industrial robot downtime and maintenance costs. This invention solves the problems of complex model building and poor model portability in existing expert system methods, and addresses the issues of poor accuracy and versatility in current pre-set threshold judgment methods due to the complexity of industrial robot joints and electromechanical coupling.

Description

Technical Field

[0001] This invention relates to the field of industrial robots, and more specifically, to an abnormal joint localization system and method for industrial robots based on LeViT. Background Technology

[0002] Industrial robots are widely used in intelligent manufacturing of key equipment such as new energy vehicles, battery manufacturing, and semiconductor production. However, industrial robots that operate continuously on factory production lines are prone to malfunctions or even breakdowns, leading to economic losses and safety hazards such as decreased production quality and even shutdowns. Due to the complex electromechanical coupling relationships in the joints of industrial robots, locating and alarming abnormal joints to assist in maintenance during anomaly detection has become a major challenge.

[0003] Currently, methods for anomaly localization and fault diagnosis of industrial robots are mainly divided into two categories.

[0004] The first type of method is the knowledge-driven expert system method, such as the fault diagnosis method and system for a six-axis industrial robot based on the AR model, which is disclosed in patent publication number CN106599794A. This method uses an expert system to diagnose faults by adding vibration acceleration sensors to the joints and processing the signals.

[0005] The second category of methods consists of data-driven machine learning methods. For example, patent publication number CN115935232A describes a CNN-LSTM-based industrial robot fault detection method. This method uses a CNN-LSTM model to detect faults in the joints of the industrial robot based on acceleration signals, with the detection result being either 0 (no fault) or 1 (fault present). Another example is patent publication number CN114897102A, which describes a CNN-LSTM-based industrial robot fault detection method that calculates feature similarity using a target Siamese neural network and then compares thresholds to determine the fault type.

[0006] However, the above method has the following limitations:

[0007] 1. There are complex electromechanical coupling relationships between the joints of industrial robots. An abnormality in a single joint can cause signal abnormalities in other joints due to the coupling relationship. Furthermore, due to the internal coupling relationship of the joints, the degree to which the signals of different joints respond to abnormalities is also different. Simple threshold settings cannot reflect this coupling relationship. Therefore, it is difficult to accurately locate abnormal joints through a uniform preset fault threshold. A set of fault thresholds set for a single robot cannot be extended to other robots.

[0008] 2. Knowledge-based expert system methods also suffer from poor versatility and portability. Accurate abnormal joint localization requires a high-quality, complex, and large expert knowledge base. However, it is very difficult to perform decoupling mechanism analysis on the complex electromechanical coupling relationship of industrial robot joints from the perspective of mechanism. Furthermore, due to the differences in event association logic, the knowledge of a single robot is difficult to reuse simply. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the purpose of this invention is to provide an abnormal joint localization system and method for industrial robots based on LeViT.

[0010] According to one aspect of the present invention, an abnormal joint localization system for an industrial robot based on LeViT is provided, comprising a Stem module, a LeViT Stage module, and a classification head module connected in sequence, wherein:

[0011] The Stem module extracts preliminary features from the input power spectrum as the first feature map;

[0012] The LeViT Stage module continues to extract features from the first feature map, and performs downsampling to reduce the size of the feature map during the feature extraction process to obtain the second feature map;

[0013] The classification head module performs abnormal pattern classification on the second feature map and obtains the abnormal joint location.

[0014] Preferably, the Stem module includes 4 convolutional layers;

[0015] Each of the convolutional layers comprises, in sequence, a Conv2d convolutional layer, a BatchNorm layer, and an activation layer;

[0016] The Conv2d convolutional layer has a stride of 2, which extracts the feature map of the input power spectrum and reduces the size of the feature map by 1 / 2.

[0017] The operation of the BatchNorm layer is as follows:

[0018]

[0019] Where, x (k) It is the value of the k-th feature of the feature map output by the Conv2d convolutional layer; μ B It is x in the current batch (k) The mean; It is x in the current batch (k) The variance; ∈ is a constant used to avoid the denominator being zero; γ (k) and β (k)These are the learnable scaling and translation parameters;

[0020] The activation function of the activation layer is Hardswish:

[0021]

[0022] Here, ReLU6(x) is the truncated ReLU function, defined as ReLU6(x) = min(max(0,x),6), where x represents the input feature map.

[0023] Preferably, the LeViT Stage module includes three sequentially connected sub-modules;

[0024] The first sub-module includes at least one first Transformer Block, which includes a first Self-Attention module and a first MLP module. The first Self-Attention module is connected to the first MLP module through a residual network, and the first MLP module is also connected to subsequent modules through a residual network. The first MLP module includes two linear normalization layers, which are connected to each other through an activation layer with Hardswish activation function and a Dropout layer.

[0025] The second sub-module includes at least one second downsample module and multiple second Transformer Blocks; the second downsample module includes an average pooling layer, a second self-attention module, and a second MLP module; the second MLP module is connected to the second Transformer Block through a residual network; the second downsample module reduces the feature map size; the second Transformer Block includes a third self-attention module and a third MLP module; the third self-attention module is connected to the third MLP module through a residual network, and the third MLP module is also connected to subsequent modules through a residual network;

[0026] The third sub-module contains at least one third downsample module and multiple third Transformer Blocks; the third downsample module includes an average pooling layer, a fourth self-attention module, and a fourth MLP module, the fourth MLP module being connected to the third Transformer Block through a residual network; the third downsample module reduces the feature map size; the third Transformer Block includes a fifth self-attention module and a fifth MLP module; the fifth self-attention module is connected to the fifth MLP module through a residual network, and the fifth MLP module is also connected to subsequent modules through a residual network;

[0027] The fourth third Transformer Block connects the feature map to the classification head module through an average pooling layer.

[0028] Preferably, the number of first Transformer Blocks in the first sub-module is two; the first Self-Attention module has 4 attention heads and each head has a dimension of 32; the weight matrix of the first linear normalization layer in the first MLP module is [256, 128], and the weight matrix of the second linear normalization layer is [128, 256].

[0029] The number of the second Downsample module and the number of the second Transformer Block in the second sub-module are 1 and 3, respectively;

[0030] In the second downsample module, the second self-attention module has 8 attention heads, each with a dimension of 32; the weight matrices of the second MLP module are [384,256] and [256,384], respectively; the second downsample module reduces the feature map size from 14×14 to 7×7;

[0031] In the second Transformer Block, the third Self-Attention module has 6 attention heads, each with a dimension of 32; the weight matrices of the third MLP module are [512,256] and [256,512], respectively.

[0032] The number of the third Downsample module and the third Transformer Block in the third sub-module are 1 and 4 respectively;

[0033] In the third Downsample module, the fourth Self-Attention module has 16 attention heads, each with a dimension of 32; the weight matrices of the fourth MLP module are [768,384] and [384,768], respectively; the third Downsample module reduces the feature map size from 7×7 to 4×4;

[0034] In the third TransformerBlock, the fifth Self-Attention module has 8 attention heads, each with a dimension of 32, and the weight matrices of the fifth MLP module are [768,384] and [384,768].

[0035] The preferred multi-head self-attention module is as follows:

[0036] MultiHead(X)=Concat(head1,...,head h W O

[0037] Where X is the input tensor; head i For the i-th attention head, specifically:

[0038] head i =Attention(XQ) i XK i XV i ), i = 1, ..., h

[0039]

[0040] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively. i =XW i Q ,K i =XW i K V i =XW i V W Q W K W V W O These are the learned projection matrices, d k For the dimension of Q;

[0041] Each of the three sub-modules of the LeViT Stage module uses a residual network connection, and its output is represented as...

[0042] Output = LayerNorm(x+y)

[0043] Where x and y represent the input and output of the Self-Attention module and the MLP module, respectively; LayerNorm represents the normalization of the feature dimensions of each input sample, specifically:

[0044]

[0045] Where x is the eigenvector; μ is the mean of the eigenvectors; σ 2 is the variance of the feature vector; ∈ is a constant value used to avoid zero in the denominator; γ and β are the learnable scaling and translation parameters, respectively.

[0046] Preferably, the classification head module includes two parallel classification heads; wherein:

[0047] One is a standard classification head, which contains a BatchNorm1d, a Dropout layer and a Linear layer, with N+1 output channels, used to classify different anomaly types, where N is the number of joints in the industrial robot;

[0048] Another classification head is the distillation classification head, which contains a BatchNorm1d and a Dropout layer, with N+1 output channels, used for the output of the distillation strategy;

[0049] Specifically, BatchNorm1d normalizes the output of the fourth third Transformer Block of the LeViT Stage module after average pooling;

[0050] The Dropout function takes the output of BatchNorm1d as its input, and randomly discards a portion of its input values ​​with probability p before outputting the result to prevent overfitting. Specifically:

[0051]

[0052] The input to the Linear layer is the output of the Dropout, and the Linear layer is mapped to an output of size N+1 through a weight matrix of [N+1,384].

[0053] The classification head module takes the average of the standard classification head and the distillation classification head as the final output.

[0054] According to a second aspect of the present invention, a method for locating abnormal joints in an industrial robot based on LeViT is provided, comprising:

[0055] Obtain historical datasets of the industrial robot under test, and construct training, validation, and test sets;

[0056] Based on the training set, validation set, and test set, a LeViT-based industrial robot joint anomaly localization system is trained. This system includes:

[0057] The Stem module extracts preliminary features from the input power spectrum to obtain the first feature map;

[0058] The LeViT Stage module further extracts features from the first feature map and downsamples to reduce the size of the feature map during the feature extraction process to obtain the second feature map.

[0059] The classification head module performs anomaly pattern classification on the second feature map and obtains anomaly localization;

[0060] The test data is input into a trained LeViT-based industrial robot joint anomaly localization system to obtain anomaly localization results.

[0061] Preferably, the step involves acquiring the historical dataset of the industrial robot under test and constructing a training set, a validation set, and a test set.

[0062] An experiment was conducted to inject joint anomalies by modifying the joint servo parameters of an industrial robot using a host computer, and current and speed signals of multiple joints of the industrial robot were collected.

[0063] Based on the fact that the sum of the absolute values ​​of the velocity signals of each joint is zero, the interval time index between industrial robot cycles is determined, and the current signal data of multiple joints in each cycle are uniformly divided into cycles using the interval time index.

[0064] The current signal data of each joint after periodic segmentation are filtered to remove high-frequency noise and DC bias;

[0065] The current signal data of each joint after denoising is completed are spliced ​​into a time series;

[0066] The power spectrum of the time series was plotted using continuous wavelet transform, and the time-frequency characteristic map of the joint current signal of the industrial robot was extracted.

[0067] After labeling the time-frequency feature maps with corresponding tags, they are classified, and training, validation, and test sets are constructed in a set ratio.

[0068] Preferably, during training, the training data is randomly shuffled; and cross-entropy loss is used as the loss function, specifically:

[0069] p = softmax(output)

[0070]

[0071] Where output is the model output; p is the value of the model output after SoftMax processing; onehot encoding... yi It is the one-hot encoding of the true value, where i is the i-th true target value and N is the total number of true target values; the number of true target values ​​is the amount of data in the training set.

[0072] Preferably, the step of plotting the power spectrum using continuous wavelet transform to extract the time-frequency feature map of the industrial robot joint current signal includes:

[0073]

[0074] Where W(a,b) are wavelet coefficients, representing the signal features at scale a and position b, which are then subjected to continuous wavelet transform to plot the time-frequency graph; x(t) is the input signal, i.e., the time series after sequential concatenation; ψ * (t) is the complex conjugate of the mother wavelet; a is the scaling parameter, which controls the expansion or contraction of the wavelet and reflects the frequency information of the signal; b is the displacement parameter, which controls the translation of the wavelet on the signal and represents its position in time.

[0075] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0076] The LeViT-based industrial robot abnormal joint localization system and method in this invention can extract abnormal joint features and perform effective classification, providing accurate abnormal joint localization and reducing industrial robot downtime and maintenance costs.

[0077] The LeViT-based industrial robot abnormal joint localization system and method in this embodiment of the invention combines the ability of continuous wavelet transform to plot wavelet power spectrum to reflect the time-frequency characteristics of joint current signals with the high-accuracy classification performance of the LeViT model on feature images.

[0078] The LeViT-based industrial robot abnormal joint localization system and method in this invention can be quickly and easily ported to different industrial robots simply by changing the dataset and retraining the LeViT model, without requiring remodeling of the complex mechanisms of the industrial robot itself. This method solves the problems of poor accuracy and versatility of current data-based neural network methods that rely on pre-set thresholds for judgment, which are affected by the complex electromechanical coupling of industrial robot joints. It effectively and accurately locates abnormal joints in industrial robots without needing to set a specific threshold for a single robot based on a large amount of experimental data. This method effectively improves the versatility and accuracy of abnormal joint localization methods for industrial robots. Attached Figure Description

[0079] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0080] Figure 1 This is a schematic diagram of the structure of an abnormal joint localization system for an industrial robot based on LeViT according to an embodiment of the present invention;

[0081] Figure 2 This is the classification result of the LeViT neural network validation set in a specific embodiment of the present invention;

[0082] Figure 3 This is the chaotic matrix of the LeViT neural network test set in a specific embodiment of the present invention;

[0083] Figure 4 This is the t-SNE result of the LeViT neural network training set in a specific embodiment of the present invention;

[0084] Figure 5 This is a flowchart of an abnormal joint localization method for industrial robots based on LeViT, according to a specific embodiment of the present invention. Detailed Implementation

[0085] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0086] Existing methods suffer from difficulties in accurately locating abnormal joints, reliance on preset expert knowledge or fault thresholds, and poor versatility and portability. Therefore, how to accurately locate abnormal joints in industrial robots without relying on preset thresholds is a pressing problem. The core issue in locating abnormal joints in industrial robots lies in extracting features reflecting different joint abnormalities and making high-precision distinctions between them. Therefore, in one embodiment of the present invention, an industrial robot abnormal joint localization system based on LeViT is provided, such as… Figure 1 As shown, it includes the Stem module, the LeViT Stage module, and the classification header module connected in sequence, wherein:

[0087] The Stem module extracts preliminary features from the input power spectrum as the first feature map;

[0088] The LeViT Stage module continues to extract features from the first feature map and downsamples to reduce the size of the feature map during the feature extraction process to obtain the second feature map;

[0089] The classification head module performs abnormal pattern classification on the second feature map and obtains the abnormal joint localization.

[0090] The Stem module, LeViT Stage module, and classification head module in the above embodiments form a neural network called the LeViT neural network. This network is an optimized neural network structure based on the traditional Transformer encoder structure. The function of the LeViT neural network is to extract features from the wavelet power spectrum image containing the time-frequency characteristics of the joint current signals of the industrial robot obtained by continuous wavelet transform (CWT), and then classify the image based on the extracted features to obtain the abnormal joint localization results. In other words, it accurately classifies the wavelet power spectrum image into N abnormal joints or N+1 normal joints.

[0091] To better extract preliminary features, a preferred embodiment of the present invention provides a preferred structure for the Stem module. The input to the Stem module is the power spectrum of the joint current signal after continuous wavelet transform, which is used as input to a 3-channel RGB feature image.

[0092] The Stem module consists of four convolutional layers; each convolutional layer contains a Conv2d convolutional layer with a stride of 2, a BatchNorm layer, and an activation layer with the Hardswish activation function.

[0093] The operations of the BatchNorm layer are as follows:

[0094]

[0095] Where, x (k) It is the value of the k-th feature of the feature map output by the Conv2d convolutional layer; μ B It is x in the current batch (k) The mean; It is x in the current batch (k) The variance; ∈ is a constant used to avoid the denominator being zero; γ (k) and β (k) These are the learnable scaling and translation parameters. BatchNorm normalizes the batch, ensuring that the outputs of each batch have a similar distribution, thereby accelerating training and improving stability. `batch` refers to the number of images input into the neural network model for training in each batch. For example, a batch size of 16 means that 16 images are input into the model at once for training. Each image undergoes Conv2d to obtain a feature map; the mean and variance here are calculated for the feature map matrix of this single batch.

[0096] The activation function for the activation layer is Hardswish, and the operation is as follows:

[0097]

[0098] Here, ReLU6(x) is the truncated ReLU function, defined as ReLU6(x) = min(max(0,x),6), where x represents the input feature map.

[0099] In the above embodiments, convolutional layers are introduced into the Transformer architecture. Compared with the Transformer, convolutional layers can perform inductive bias with spatial locality and translation invariance when extracting features from images. This can maximize the learning and preservation of original image information. Each Conv2d convolutional layer with a stride of 2 reduces the feature map size by 1 / 2. The image input size is 3×224×224. The output size after the first convolutional layer is 16×122×122, the output size after the second convolutional layer is 32×56×56, the output size after the third convolutional layer is 64×28×28, and the output size after the fourth convolutional layer is 128×14×14, which is used as the output of the Stem layer.

[0100] By using four convolutional layers to quickly reduce the size of the feature map while preserving the original image features, and then inputting it into the LeViT Stage module based on the Transformer architecture, the number of computations of the Transformer neural network is quadratically linearly related to the feature map size, which can significantly reduce the computational cost of the model.

[0101] To further extract more complex features, a preferred embodiment of the present invention provides a preferred structure for the LeViT Stage module, such as... Figure 1 As shown, the LeViT Stage module contains three sub-modules, each used to extract more complex features layer by layer.

[0102] The first sub-module includes two first Transformer Blocks. Each first Transformer Block comprises a Self-Attention module with four attention heads, each with a dimension of 32, and a first MLP module. The first MLP module includes two linear normalization layers: the weight matrix of the first linear normalization layer is [256, 128], and the weight matrix of the second linear normalization layer is [128, 256]. The two linear normalization layers are connected by an activation layer with a Hardswish activation function and a Dropout layer. The Self-Attention module is connected to the MLP module via a residual network, and the MLP module is also connected to subsequent modules via a residual network.

[0103] The second submodule comprises one second downsample module and three second transformer blocks. The second downsample module includes an average pooling layer, a second self-attention module with eight attention heads, each with a dimension of 32, and a second MLP module with weight matrices of [384, 256] and [256, 384], respectively. The second MLP module is connected to the second transformer blocks via a residual network. The second downsample module reduces the feature map size from 14×14 to 7×7. The second transformer block includes a third self-attention module with six attention heads, each with a dimension of 32, and a third MLP module with weight matrices of [512, 256] and [256, 512], respectively. The third self-attention module is connected to the third MLP module via a residual network, and the third MLP module is also connected to subsequent modules via a residual network.

[0104] The third sub-module comprises one third downsample module and four third transformer blocks. The third downsample module includes an average pooling layer, a fourth self-attention module with 16 attention heads, each with a dimension of 32, and a fourth MLP module with weight matrices of [768, 384] and [384, 768], respectively. The fourth MLP module is connected to the third transformer block via a residual network. The third downsample module reduces the feature map size from 7×7 to 4×4. The third transformer block includes a fifth self-attention module with 8 attention heads, each with a dimension of 32, and a fifth MLP module with weight matrices of [768, 384] and [384, 768], respectively. The fifth self-attention module is connected to the fifth MLP module via a residual network, and the fifth MLP module is also connected to subsequent modules via a residual network. The fourth third transformer block connects the feature map to the classification head module via an average pooling layer.

[0105] The second and third downsample modules both use a self-attention module with twice the attention head to ensure the preservation and transmission of feature information and prevent the loss of feature information during downsampling when the feature map downsamples to half its size.

[0106] The 128×14×14 feature map output from the Stem module is input into the three sub-modules of the above embodiment, and the feature map sizes obtained are 128×14×14, 256×7×7, and 384×4×4 respectively. After passing through the average pooling layer, the feature map size is 384.

[0107] The above embodiments all mention a multi-head self-attention module. In a preferred embodiment, the calculation process of the multi-head self-attention module is as follows:

[0108] MultiHead(X)=Concat(head1,...,head h W O

[0109] Where X is the input tensor; head i For the i-th attention head, the calculation method is as follows:

[0110] head i =Attention(XQ) i XK i XV i ), i = 1, ..., h

[0111]

[0112] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively. i =XW i Q ,K i =XW i K V i =XW i V W Q W K W V W O These are the learned projection matrices, d k For the dimension of Q.

[0113] In another preferred embodiment, each of the three sub-modules of the LeViT Stage module uses a residual network connection, and its output is represented as...

[0114] Output = LayerNorm(x+y)

[0115] Where x and y represent the input and output of the Self-Attention module and the MLP module, respectively; LayerNorm represents the normalization of the feature dimensions of each input sample, specifically:

[0116]

[0117] Where x is the eigenvector; μ is the mean of the eigenvectors; σ 2 is the variance of the feature vector; ∈ is a constant value used to avoid zero denominators; γ and β are learnable scaling and translation parameters, respectively. LayerNorm normalizes the feature dimensions of each input sample, avoiding the dependency of BatchNorm on batch size.

[0118] Following the above embodiments, after extracting complex joint anomaly features, classification is required. To further improve classification accuracy, in one embodiment of the present invention, the classification head module includes two parallel classification heads for anomaly pattern classification output, thereby achieving anomaly localization. One is a standard classification head, containing a BatchNorm1d, a Dropout layer, and a Linear layer, with N+1 output channels, used to classify different anomaly types, where N is the number of joints in the industrial robot; the other classification head is a distillation classification head, containing a BatchNorm1d and a Dropout layer, with N+1 output channels, used for outputting distillation strategies.

[0119] BatchNorm1d normalizes the output of the fourth third TransformerBlock of the LeViT Stage module after average pooling.

[0120] Dropout takes the output of BatchNorm1d as input, and randomly discards a portion of its input with probability p before outputting it to prevent overfitting. Specifically:

[0121]

[0122] The input to the Linear layer is the output of Dropout, and the Linear layer is mapped to an output of size N+1 through a weight matrix of [N+1,384].

[0123] The distillation strategy involves introducing a distilled classification head into the CNN model as a teacher model to guide the LeViT neural network towards rapid convergence, as CNN models converge faster than Transformers when processing image tasks. The entire classification head module performs a CE Loss calculation between the standard classification head and the ground truth, and a CE Loss calculation between the distilled classification head and the CNN output. After training iterations, the average of the two classification heads is taken as the output.

[0124] Compared to other Transformer-based models, the LeViT model (Stem module, LeViT Stage module, and classification head module) in the above embodiments retains the high classification accuracy of other models while introducing CNN convolutions in the image feature extraction stage. This leverages the powerful image feature extraction capabilities of CNN convolutions to efficiently extract initial image features while rapidly reducing the feature map size. This reduces the computational cost and the number of model parameters required for the subsequent Transformer-based SelfAttention module. Furthermore, a Downsample module is added to the second and third sub-modules of the LeViT Stage module. By adding an average pooling layer to the SelfAttention module, the feature map size is reduced by half, and the number of attention heads in the SelfAttention module is doubled. This prevents loss of feature information while reducing the feature map size. The Downsample module further reduces the feature map size, decreasing the computational cost and the number of model parameters required. Finally, in the classification head module, distillation learning is used to introduce CNN convolutions as a teacher model to guide the LeViT model to converge quickly. This allows the LeViT model to maintain high performance while significantly reducing computational cost and the number of model parameters.

[0125] In the above embodiments, without relying on preset thresholds or establishing complex expert systems, abnormal joint features can be extracted and effectively classified, providing accurate abnormal joint localization and reducing industrial robot downtime and maintenance costs.

[0126] In some specific embodiments, the parameters of each module can use the data shown in Table 1.

[0127] Table 1 LeViT Neural Network Model Parameters

[0128]

[0129]

[0130] Based on the same inventive concept, this invention provides a method for locating abnormal joints in industrial robots using LeViT, which can be implemented through the following steps:

[0131] Step 1: Obtain the historical dataset of the industrial robot under test, and construct the training set, validation set, and test set;

[0132] Step 2: Based on the training set, validation set, and test set constructed in Step 1, train the LeViT-based industrial robot abnormal joint localization system. This system includes:

[0133] The Stem module extracts preliminary features from the input power spectrum to obtain the first feature map;

[0134] The LeViT Stage module continues to extract features from the first feature map and downsamples to reduce the size of the feature map during the feature extraction process to obtain the second feature map.

[0135] The classification head module performs anomaly pattern classification on the second feature map and obtains anomaly localization;

[0136] The test data is input into a trained LeViT-based industrial robot joint anomaly localization system to obtain anomaly localization results.

[0137] Step 3: Input the data to be tested into the trained LeViT-based industrial robot abnormal joint localization system to obtain the abnormal localization results.

[0138] The specific implementation technology of each module / unit in the above examples of the present invention can be referred to the module / unit of the abnormal joint localization system of industrial robot based on LeViT in the above embodiments, and will not be repeated here.

[0139] The method described in the above embodiments solves the problems of complex model building and poor model portability of current knowledge-based expert system methods. This method allows for rapid and convenient portability to different industrial robots simply by changing the dataset and retraining the LeViT model, without requiring remodeling of the complex mechanisms of the industrial robot itself. Furthermore, this method addresses the issues of poor accuracy and versatility of current data-based neural network methods that rely on pre-set thresholds for judgment, which are affected by the complex electromechanical coupling of industrial robot joints. It eliminates the need to set a specific threshold for a single robot based on extensive experimental data to effectively and accurately locate abnormal joints in industrial robots. This method effectively improves the versatility and accuracy of abnormal joint localization methods for industrial robots.

[0140] In a preferred embodiment of the present invention, a preferred process for step 1 is provided, specifically:

[0141] S1: The data preparation steps for the LeViT-based industrial robot abnormal joint localization system (also known as a neural network model) include:

[0142] S11: Anomaly injection experiments were conducted on the industrial robot's joints. The joint servo parameters were modified via a host computer to inject anomalies at four different levels into each of the six joints. For each anomaly injection experiment, current signal data from the six joints was collected for 25 complete cycles. Simultaneously, current signal data from the six joints was collected for 25 complete cycles under normal operating conditions on the industrial robot. The experiment used a smart terminal connected to the industrial robot's PLC control cabinet via network cable to read real-time current signal data from each joint, and also recorded joint speed signal data for subsequent analysis. The sampling frequency was 250Hz.

[0143] S12: Determine the interval time index between industrial robot cycles based on the fact that the sum of the absolute values ​​of the six joint speed signals corresponding to the collected industrial robot joint current signals is 0. Use the interval time index to uniformly divide the six joint current signals of each cycle into cycles.

[0144] S13: After periodic segmentation, the current signals of each joint are filtered using a 5th-order Butterworth filter with cutoff frequencies of 0.1Hz and 50Hz to retain the 0.1Hz to 50Hz portion of the signal. The DC bias of the current signal is removed by the 0.1Hz cutoff frequency, and high-frequency noise is removed by the 50Hz cutoff frequency.

[0145] S14: Consolidate the current signal data of 6 joints in the same period into a time series.

[0146] S15, use continuous wavelet transform to plot the power spectrum of the time series to extract the time-frequency features of the industrial robot joint current signal. The specific parameters of the continuous wavelet transform are as follows: time interval. The sampling time interval corresponds to a sampling frequency of 250Hz; N is the total length of the continuous wavelet transform, i.e., the number of data points in the time series after sequential concatenation; the time array t is the corresponding time array generated for this time series of length N, i.e., an equally spaced sequence of [0×dt, N×dt]; the minimum scale s0 = 2×dt; the scale step size in the continuous wavelet transform. Total number of decomposition scales The Morlet wavelet is chosen as the mother wavelet. The mathematical formula for the Morlet wavelet is as follows:

[0147]

[0148] Wherein, ψ(t) is the mother wavelet function; ω0 is the center frequency of the wavelet, which is taken as 6 in this method; It is a complex exponential term, which provides the oscillatory properties of wavelets; It is the Gaussian envelope, which controls the attenuation of the wavelet in the time domain. The formula for the continuous wavelet transform is shown below:

[0149]

[0150] Where W(a,b) are wavelet coefficients, representing the signal features at scale a and position b, which are then subjected to continuous wavelet transform to extract the time-frequency plot; x(t) is the input signal, i.e., the time series after sequential concatenation; ψ * (t) is the complex conjugate of the mother wavelet; a is the scaling parameter, which controls the expansion or contraction of the wavelet and reflects the frequency information of the signal; b is the displacement parameter, which controls the translation of the wavelet on the signal and represents its position in time.

[0151] The continuous wavelet transform in this step, by adjusting the scaling parameter 'a', enables the analysis of signal characteristics at different frequency scales. High frequencies (small scales) correspond to the local details of the signal, while low frequencies (large scales) correspond to the overall outline of the signal. The continuous wavelet transform can simultaneously extract the signal's feature information in both time and frequency.

[0152] Continuous wavelet transform offers good time resolution at high frequencies and high frequency resolution at low frequencies, enabling it to capture transient high-frequency abrupt changes and long-term low-frequency trends. It is suitable for extracting anomalous features from joint current signals of industrial robots that contain complex frequency components and noise. The wavelet power spectrum plotted using continuous wavelet transform reflects the temporal variation of energy distribution among different frequency components in the current signal, while simultaneously extracting the time-frequency characteristics of the joint current signal.

[0153] S16: After labeling 100 time-frequency images from each of the 6 groups of abnormal operating conditions and 1 group of normal operating conditions, classify them and construct training, validation and test sets in an 8:1:1 ratio.

[0154] Using the dataset obtained in the above embodiments, model training is performed. In a preferred embodiment, S2: The training and verification testing steps for the LeViT neural network model used for abnormal joint localization in industrial robots include:

[0155] S21: Before model training, all input images are adjusted to a fixed size (3×224×224) to meet the input requirements of the LeViT neural network model. The images are then normalized, with the pixel values ​​standardized to a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225].

[0156] S22: Use DataLoader to create a data loader to load the training, validation, and test sets classified in S15, and perform the data preprocessing steps in S21. Set the batch size to 16, and set the training data to be randomly shuffled so that the data order is different in each epoch, increasing the model's generalization ability. Use Cross-Entropy Loss (CE Loss) as the loss function, which is suitable for multi-class classification problems. The formula for calculating CE Loss is as follows:

[0157] p = softmax(output)

[0158]

[0159] Where output is the model output; p is the value of the model output after softmax processing; onehot encoding... yi This is the one-hot encoding of the true values. AdamW is used as the optimizer with a learning rate of 0.001. AdamW is an improved Adam optimizer with weight decay properties, which helps prevent overfitting. The number of training epochs is set to 50. GPUs are used for training, validating, and testing the LeViT neural network model to accelerate training.

[0160] S23: The model quantitatively evaluates the abnormal joint localization performance of industrial robots by using the results of the trained LeViT neural network model on the validation set, the chaos matrix on the test set, and the t-SNE visualization classification results on the training set, as follows: Figure 2 , Figure 3 and Figure 4 As shown. From Figure 2 It can be seen that the classification of 6 joint abnormalities and 1 normal condition on the validation set is completely consistent with the actual situation; from Figure 3 The chaotic matrix shows that the classification of all six joint abnormalities and one normal condition on the test set was correct; from Figure 4 The t-SNE visualization classification results show that the model achieved good classification results on the training set. Therefore, this model has a high accuracy in abnormal joint localization of industrial robots, which is a significant improvement over the 78.57% accuracy of the ViT neural network on the same dataset.

[0161] The trained and tested LeViT-based neural network for locating abnormal joints in industrial robots is then applied. In a preferred embodiment of the present invention, S3, the method for locating abnormal joints in industrial robots can employ the following steps, such as... Figure 5 As shown:

[0162] S31: Data Acquisition: Reads real-time joint current and speed data by connecting to the industrial robot PLC via a smart terminal.

[0163] S32: Data Preprocessing: Periodic segmentation is performed based on the fact that the sum of the absolute values ​​of the six joint velocity signals corresponding to the current signal is 0. After filtering the six joint current signals of a single period through the same Butterworth filter as in S13, the six joint current data are sequentially concatenated to obtain a time series of a single current signal.

[0164] S33: Time-frequency feature extraction: Perform the same continuous wavelet transform as in S14 on the time series obtained in S32 to obtain the corresponding power spectrum, which is used to extract the time-frequency features of the current signal time series.

[0165] S34: LeViT Model Joint Anomaly Localization: Input the power spectrum into the LeViT model trained in S2 to perform anomaly joint localization and output the anomaly joint localization results of the industrial robot.

[0166] The LeViT-based method for locating abnormal joints in industrial robots described in the above embodiments combines the ability of continuous wavelet transform to plot wavelet power spectra that reflect the time-frequency characteristics of joint current signals with the high-accuracy classification performance of the LeViT model for feature images, thereby achieving high-accuracy localization of abnormal joints in industrial robots.

[0167] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. An abnormal joint localization system for industrial robots based on LeViT, characterized in that, It includes the Stem module, the LeViT Stage module, and the classification head module, which are connected in sequence, wherein: The Stem module extracts preliminary features from the input power spectrum as the first feature map; The LeViT Stage module continues to extract features from the first feature map, and performs downsampling to reduce the size of the feature map during the feature extraction process to obtain the second feature map; The classification head module performs abnormal pattern classification on the second feature map and obtains the abnormal joint location.

2. The abnormal joint localization system for industrial robots based on LeViT according to claim 1, characterized in that, The Stem module includes four convolutional layers; Each of the convolutional layers comprises, in sequence, a Conv2d convolutional layer, a BatchNorm layer, and an activation layer; The Conv2d convolutional layer has a stride of 2, which extracts the feature map of the input power spectrum and reduces the size of the feature map by 1 / 2. The operation of the BatchNorm layer is as follows: Where, x (k) It is the value of the k-th feature of the feature map output by the Conv2d convolutional layer; μ B It is x in the current batch (k) The mean; It is x in the current batch (k) The variance; ∈ is a constant used to avoid the denominator being zero; γ (k) and β (k) These are the learnable scaling and translation parameters; The activation function of the activation layer is Hardswish: Here, ReLU6(x) is the truncated ReLU function, defined as ReLU6(x) = min(max(0,x),6), where x represents the input feature map.

3. The abnormal joint localization system for industrial robots based on LeViT according to claim 1, characterized in that, The LeViT Stage module includes three sequentially connected sub-modules; The first submodule includes at least one first Transformer Block, which includes a first Self-Attention module and a first MLP module; The first Self-Attention module is connected to the first MLP module through a residual network, and the first MLP module is also connected to subsequent modules through a residual network. The first MLP module includes two linear normalization layers, and the two linear normalization layers are connected through an activation layer with Hardswish activation function and a Dropout layer. The second submodule contains at least one second Downsample module and multiple second Transformer Blocks; The second downsample module includes an average pooling layer, a second self-attention module, and a second MLP module; the second MLP module is connected to the second Transformer Block through a residual network. The second downsample module reduces the feature map size; the second Transformer Block includes a third Self-Attention module and a third MLP module; the third Self-Attention module is connected to the third MLP module through a residual network, and the third MLP module is also connected to subsequent modules through a residual network; The third submodule contains at least one third downsample module and multiple third Transformer Blocks; the third downsample module includes an average pooling layer, a fourth self-attention module and a fourth MLP module, and the fourth MLP module is connected to the third Transformer Block through a residual network; The third Downsample module reduces the feature map size; the third Transformer Block includes a fifth Self-Attention module and a fifth MLP module; the fifth Self-Attention module is connected to the fifth MLP module through a residual network, and the fifth MLP module is also connected to subsequent modules through a residual network; The fourth third Transformer Block connects the feature map to the classification head module through an average pooling layer.

4. The abnormal joint localization system for industrial robots based on LeViT according to claim 3, characterized in that, The number of first Transformer Blocks in the first sub-module is two; the first Self-Attention module has 4 attention heads, each with a dimension of 32; the weight matrix of the first linear normalization layer in the first MLP module is [256, 128], and the weight matrix of the second linear normalization layer is [128, 256]. The number of the second Downsample module and the number of the second Transformer Block in the second sub-module are 1 and 3, respectively; In the second downsample module, the second self-attention module has 8 attention heads, each with a dimension of 32; the weight matrices of the second MLP module are [384,256] and [256,384], respectively; the second downsample module reduces the feature map size from 14×14 to 7×7; In the second Transformer Block, the third Self-Attention module has 6 attention heads, each with a dimension of 32; the weight matrices of the third MLP module are [512,256] and [256,512], respectively. The number of the third Downsample module and the third Transformer Block in the third sub-module are 1 and 4 respectively; In the third Downsample module, the fourth Self-Attention module has 16 attention heads, each with a dimension of 32; the weight matrices of the fourth MLP module are [768,384] and [384,768], respectively; the third Downsample module reduces the feature map size from 7×7 to 4×4; In the third TransformerBlock, the fifth Self-Attention module has 8 attention heads, each with a dimension of 32, and the weight matrices of the fifth MLP module are [768,384] and [384,768].

5. The abnormal joint localization system for industrial robots based on LeViT according to claim 3, characterized in that, The multi-head self-attention module is as follows: MultiHead(X)=Concat(head1,...,head h )W O Where X is the input tensor; head i For the i-th attention head, specifically: head i =Attention(XQ i ,XK i ,XV i ),i=1,...,h Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively. i =XW i Q ,K i =XW i K V i =XW i V W Q W K W V W O These are the learned projection matrices, d k For the dimension of Q; Each of the three sub-modules of the LeViT Stage module uses a residual network connection, and its output is represented as Output = LayerNorm(x + y). Where x and y represent the input and output of the Self-Attention module and the MLP module, respectively; LayerNorm represents the normalization of the feature dimensions of each input sample, specifically: Where x is the eigenvector; μ is the mean of the eigenvectors; σ 2 is the variance of the feature vector; ∈ is a constant value used to avoid zero in the denominator; γ and β are the learnable scaling and translation parameters, respectively.

6. The abnormal joint localization system for industrial robots based on LeViT according to claim 1, characterized in that, The classification head module includes two parallel classification heads; wherein: One is a standard classification head, which contains a BatchNorm1d, a Dropout layer and a Linear layer, with N+1 output channels, used to classify different anomaly types, where N is the number of joints in the industrial robot; Another classification head is the distillation classification head, which contains a BatchNorm1d and a Dropout layer, with N+1 output channels, used for the output of the distillation strategy; Specifically, BatchNorm1d normalizes the output of the fourth third Transformer Block of the LeViT Stage module after average pooling; The Dropout function takes the output of BatchNorm1d as its input, and randomly discards a portion of its input values ​​with probability p before outputting the result to prevent overfitting. Specifically: The input to the Linear layer is the output of the Dropout, and the Linear layer is mapped to an output of size N+1 through a weight matrix of [N+1,384]. The classification head module takes the average of the standard classification head and the distillation classification head as the final output.

7. A method for locating abnormal joints in an industrial robot based on LeViT, characterized in that, include: Obtain historical datasets of the industrial robot under test, and construct training, validation, and test sets; Based on the training set, validation set, and test set, a LeViT-based industrial robot joint anomaly localization system is trained. This system includes: The Stem module extracts preliminary features from the input power spectrum to obtain the first feature map; The LeViT Stage module further extracts features from the first feature map and downsamples to reduce the size of the feature map during the feature extraction process to obtain the second feature map. The classification head module performs anomaly pattern classification on the second feature map and obtains anomaly localization; The test data is input into a trained LeViT-based industrial robot joint anomaly localization system to obtain anomaly localization results.

8. The method for locating abnormal joints of an industrial robot based on LeViT according to claim 7, characterized in that, The process involves acquiring historical datasets of the industrial robot under test and constructing training, validation, and test sets. An experiment was conducted to inject joint anomalies by modifying the joint servo parameters of an industrial robot using a host computer, and current and speed signals of multiple joints of the industrial robot were collected. Based on the fact that the sum of the absolute values ​​of the velocity signals of each joint is zero, the interval time index between industrial robot cycles is determined, and the current signal data of multiple joints in each cycle are uniformly divided into cycles using the interval time index. The current signal data of each joint after periodic segmentation are filtered to remove high-frequency noise and DC bias; The current signal data of each joint after denoising is completed are spliced ​​into a time series; The power spectrum of the time series was plotted using continuous wavelet transform, and the time-frequency characteristic map of the joint current signal of the industrial robot was extracted. After labeling the time-frequency feature maps with corresponding tags, they are classified, and training, validation, and test sets are constructed in a set ratio.

9. A method for locating abnormal joints in an industrial robot based on LeViT according to claim 7, characterized in that, During training, the training data is randomly shuffled; and cross-entropy loss is used as the loss function, specifically: p = softmax(output) Where output is the model output; p is the value of the model output after SoftMax processing; onehot encoding... yi It is the one-hot encoding of the true value, where i is the i-th true target value and N is the total number of true target values; the number of true target values ​​is the amount of data in the training set.

10. A method for locating abnormal joints of an industrial robot based on LeViT according to claim 8, characterized in that, The step of plotting the power spectrum using continuous wavelet transform to extract the time-frequency characteristic map of the industrial robot joint current signal includes: Where W(a,b) are wavelet coefficients, representing the signal features at scale a and position b, which are then subjected to continuous wavelet transform to plot the time-frequency graph; x(t) is the input signal, i.e., the time series after sequential concatenation; ψ * (t) is the complex conjugate of the mother wavelet; a is the scaling parameter, which controls the expansion or contraction of the wavelet and reflects the frequency information of the signal; b is the displacement parameter, which controls the translation of the wavelet on the signal and represents its position in time.

Images