Industrial robot assembly practical training system and method based on virtual simulation

Abstract

The invention discloses an industrial robot assembly training system and method based on virtual simulation. The system comprises the following steps: providing a virtual simulation platform for providing a virtual scene and virtual equipment for industrial robot assembly practical training; the virtual reality equipment is used for interacting with the virtual simulation platform and operating and controlling the virtual scene and the virtual equipment; the data processing module is used for receiving an input signal of the virtual reality equipment, analyzing and processing the input signal, generating a corresponding feedback signal and sending the feedback signal to the virtual reality equipment; and the evaluation module is used for evaluating the assembly operation of the user. In this way, objective evaluation of the assembly operation of the user can be realized.

Description

Technical Field

[0001] The present application relates to the field of industrial robots, and more specifically, to an industrial robot assembly training system and method based on virtual simulation. Background Art

[0002] Industrial robots are mechanical devices that can automatically perform industrial production tasks. They have the advantages of high speed, high precision, high flexibility and high reliability, and are widely used in the fields of automobiles, electronics, machinery, etc. However, the assembly process of industrial robots is a complex technical activity that requires professional knowledge and skills. Traditional training methods have problems such as high cost, low efficiency and poor safety.

[0003] Therefore, an industrial robot assembly training system and method based on virtual simulation is expected. Summary of the invention

[0004] In view of this, the present application proposes an industrial robot assembly training system and method based on virtual simulation, which can evaluate the deviation between the user's assembly operation and the standard operation, and realize objective evaluation of the user's assembly operation.

[0005] According to one aspect of the present application, there is provided an industrial robot assembly training system based on virtual simulation, which comprises:

[0006] Virtual simulation platform, used to provide virtual scenes and virtual equipment for industrial robot assembly training;

[0007] A virtual reality device, used to interact with the virtual simulation platform to operate and control the virtual scene and the virtual device;

[0008] a data processing module, configured to receive an input signal from the virtual reality device, analyze and process the input signal, generate a corresponding feedback signal, and send the feedback signal to the virtual reality device; and

[0009] The evaluation module is used to evaluate the user's assembly operation.

[0010] In the above-mentioned industrial robot assembly training system based on virtual simulation, the evaluation module includes:

[0011] A data acquisition unit, used to acquire the assembly operation video of the user acquired by a camera, and a reference operation video;

[0012] A timing analysis unit, used for performing timing analysis on the assembly operation video and the reference operation video to obtain an assembly operation timing characteristic diagram and a reference operation timing characteristic diagram;

[0013] a semantic difference extraction unit, configured to extract operation semantic difference features between the assembly operation timing feature graph and the reference operation timing feature graph to obtain an operation difference semantic transfer representation feature graph; and

[0014] The deviation analysis unit is used to determine whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold based on the operation difference semantic transfer representation feature map.

[0015] In the above-mentioned industrial robot assembly training system based on virtual simulation, the timing analysis unit includes:

[0016] a preprocessing subunit, configured to preprocess the assembly operation video and the reference operation video to obtain a sequence of assembly operation key frames and a sequence of reference operation key frames; and

[0017] The timing correlation feature extraction subunit is used to obtain the assembly operation timing feature graph and the reference operation timing feature graph by passing the sequence of the assembly operation key frames and the sequence of the reference operation key frames through an operation timing correlation feature extractor based on a three-dimensional convolutional network model.

[0018] In the above-mentioned industrial robot assembly training system based on virtual simulation, the preprocessing subunit is used to:

[0019] The assembly operation video and the reference operation video are sparsely sampled to obtain a sequence of the assembly operation key frames and a sequence of the reference operation key frames.

[0020] In the above-mentioned industrial robot assembly training system based on virtual simulation, the semantic difference extraction unit is used to:

[0021] The transfer matrix between the feature matrices of each group of corresponding channel dimensions of the assembly operation timing feature graph and the reference operation timing feature graph is calculated to obtain the operation difference semantic transfer representation feature graph.

[0022] In the above-mentioned industrial robot assembly training system based on virtual simulation, the deviation analysis unit includes:

[0023] A local operation saliency subunit, configured to pass the operation difference semantic transfer representation feature map through a local operation saliency device based on an adaptive attention layer to obtain a salient operation difference semantic transfer representation feature map; and

[0024] The classification subunit is used to pass the semantic transfer representation feature map of the significant operation difference through a classifier to obtain a classification result, and the classification result is used to indicate whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

[0025] In the above-mentioned industrial robot assembly training system based on virtual simulation, the local operation significant subunit is used to:

[0026] The operation difference semantic transfer representation feature map is processed by the following adaptive attention formula to obtain the salient operation difference semantic transfer representation feature map; wherein the adaptive attention formula is:

[0027] v c =pool(F)

[0028] A=σ(W a *v c +b a )

[0029]

[0030] F′=A′⊙F

[0031] Where F is the semantic transfer representation feature map of the operation difference, pool is the pooling process, and v c is the pooling vector, W a is the weight matrix, b a is the bias vector, σ is the activation process, A is the initial element weight feature vector, A i is the eigenvalue of the i-th position in the initial meta-weight feature vector, A′ is the corrected meta-weight feature vector, F′ is the significant operation difference semantic transfer representation feature map, and ⊙ represents the multiplication of each feature matrix of the operation difference semantic transfer representation feature map along the channel dimension with the eigenvalue in the corrected meta-weight feature vector as the weight.

[0032] In the above-mentioned industrial robot assembly training system based on virtual simulation, the classification subunit includes:

[0033] A feature map optimization secondary subunit, used for optimizing the saliency operation difference semantic transfer representation feature map to obtain an optimized saliency operation difference semantic transfer representation feature map; and

[0034] The deviation classification secondary subunit is used to pass the optimized significant operation difference semantic transfer representation feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

[0035] In the above-mentioned industrial robot assembly training system based on virtual simulation, the deviation classification secondary subunit is used to:

[0036] Expanding the optimized saliency operation difference semantic transfer representation feature map into a classification feature vector according to a row vector or a column vector;

[0037] Performing full connection encoding on the classification feature vector using the fully connected layer of the classifier to obtain an encoded classification feature vector; and

[0038] The encoded classification feature vector is input into the Softmax classification function of the classifier to obtain the classification result.

[0039] According to another aspect of the present application, there is provided an industrial robot assembly training method based on virtual simulation, which comprises:

[0040] Provide virtual scenes and virtual equipment for industrial robot assembly training through virtual simulation platform;

[0041] The virtual scene and the virtual device are operated and controlled by interacting with the virtual simulation platform through a virtual reality device;

[0042] receiving an input signal of the virtual reality device through a data processing module, analyzing and processing the input signal, generating a corresponding feedback signal, and sending the feedback signal to the virtual reality device; and

[0043] The user's assembly operations are evaluated through the evaluation module.

[0044] In the present application, the system includes: a virtual simulation platform for providing a virtual scene and virtual equipment for industrial robot assembly training; a virtual reality device for interacting with the virtual simulation platform to realize the operation and control of the virtual scene and the virtual equipment; a data processing module for receiving an input signal of the virtual reality device, analyzing and processing the input signal, generating a corresponding feedback signal, and sending the feedback signal to the virtual reality device; and an evaluation module for evaluating the user's assembly operation. In this way, an objective evaluation of the user's assembly operation can be achieved.

[0045] Other features and aspects of the present application will become apparent from the following detailed description of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A block diagram of an industrial robot assembly training system based on virtual simulation according to an embodiment of the present application is shown.

[0048] Figure 2 A block diagram of the evaluation module in the virtual simulation-based industrial robot assembly training system according to an embodiment of the present application is shown.

[0049] Figure 3 A flow chart of an industrial robot assembly training method based on virtual simulation according to an embodiment of the present application is shown.

[0050] Figure 4 A schematic diagram of the architecture of sub-step S140 in the industrial robot assembly training method based on virtual simulation according to an embodiment of the present application is shown.

[0051] Figure 5 An application scenario diagram of an industrial robot assembly training system based on virtual simulation according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0053] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0054] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0055] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0056] This application provides an industrial robot assembly training system based on virtual simulation. Figure 1 FIG. 1 is a block diagram of an industrial robot assembly training system based on virtual simulation according to an embodiment of the present application. Figure 1As shown, according to an embodiment of the present application, an industrial robot assembly training system 100 based on virtual simulation includes: a virtual simulation platform 110, which is used to provide virtual scenes and virtual devices for industrial robot assembly training; a virtual reality device 120, which is used to interact with the virtual simulation platform to realize the operation and control of the virtual scenes and the virtual devices; a data processing module 130, which is used to receive input signals from the virtual reality device, analyze and process the input signals, generate corresponding feedback signals, and send the feedback signals to the virtual reality device; and an evaluation module 140, which is used to evaluate the user's assembly operation.

[0057] Among them, evaluating the user's assembly operation can measure his skill level and identify existing problems to help the user improve operational efficiency and accuracy. In addition, by evaluating the user's assembly operation, equipment damage or safety accidents caused by incorrect operation can be prevented. In the prior art, the evaluation of the user's assembly operation is usually performed manually based on the degree to which the user completes the task. For example, based on the set task requirements and time limits, it is evaluated whether the user completes the assembly task as required, and a corresponding score or feedback is given. However, the evaluation results obtained in this way may be affected by personal subjective factors, such as emotional fluctuations and fatigue levels, which may cause deviations in the evaluation results. Therefore, an optimized solution is expected.

[0058] In response to the above technical problems, the technical concept of the present application is: to use a deep algorithm to jointly analyze the user's assembly operation video and the reference operation video, so as to learn and characterize the difference between the two, that is, the difference between the user's assembly operation and the standard operation, so as to evaluate the deviation between the user's assembly operation and the standard operation, and achieve an objective evaluation of the user's assembly operation.

[0059] Based on this, Figure 2 As shown, the evaluation module 140 includes: a data acquisition unit 141, which is used to obtain the assembly operation video of the user collected by a camera, and a reference operation video; a timing analysis unit 142, which is used to perform timing analysis on the assembly operation video and the reference operation video to obtain an assembly operation timing feature diagram and a reference operation timing feature diagram; a semantic difference extraction unit 143, which is used to extract the operation semantic difference feature between the assembly operation timing feature diagram and the reference operation timing feature diagram to obtain an operation difference semantic transfer representation feature diagram; and a deviation analysis unit 144, which is used to determine whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold based on the operation difference semantic transfer representation feature diagram.

[0060] It should be understood that the evaluation module 140 is a part of the industrial robot assembly training system, which is used to evaluate the user's assembly operation. Among them, in the data acquisition unit 141, by collecting videos, the user's actual operation and the reference operation process can be recorded. The timing analysis unit 142 performs timing analysis on the assembly operation video and the reference operation video to obtain the assembly operation timing feature diagram and the reference operation timing feature diagram. The timing feature diagram reflects the time sequence and timing characteristics of the operation, which can be used to compare the difference between the user's operation and the reference operation. The semantic difference extraction unit 143 is used to extract the operation semantic difference features between the assembly operation timing feature diagram and the reference operation timing feature diagram. By extracting the semantic difference features, the difference between the user's operation and the standard operation can be captured. The deviation analysis unit 144 determines whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold based on the operation difference semantic transfer representation feature diagram. By analyzing the deviation, it can be evaluated whether the user's operation meets the standard requirements. Combining the functions of the above units, the evaluation module can conduct a comprehensive evaluation of the user's assembly operation, including timing analysis, semantic difference extraction and deviation analysis, to help users improve their operating skills and assembly quality.

[0061] Specifically, in the technical solution of the present application, the encoding process of the evaluation module includes: first, obtaining the assembly operation video of the user collected by the camera, and the reference operation video. Here, the assembly operation video of the user records the actual operation of the user during the assembly operation. That is, by collecting the assembly operation video of the user through the camera, the user's actions, postures, and operation sequence during the actual operation can be obtained. This video can be used as an important data source for evaluating the user's operation skill level and discovering existing problems. The reference operation video refers to a video of the assembly operation process of the specification or standard recorded in advance. This video shows the correct assembly steps, operation sequence, posture requirements, etc. The reference operation video is used as a comparison and reference to evaluate the difference and deviation between the user's assembly operation and the specification operation. That is, by comparing the user's assembly operation video with the reference operation video, the user's errors, inaccurate actions or violations of the specification in the assembly operation can be determined, and corresponding feedback can be provided.

[0062] It should be understood that the assembly operation video and the reference operation video are usually continuous video streams, including a large number of frame images. In actual assembly operations, key actions and posture changes often occur in certain specific key frames, and other frames may contain repeated or unimportant information. Therefore, in the technical solution of the present application, the assembly operation video and the reference operation video are sparsely sampled to select a part of the key frames to represent the entire video data, thereby obtaining a sequence of assembly operation key frames and a sequence of reference operation key frames.

[0063] Next, the sequence of assembly operation key frames and the sequence of reference operation key frames are passed through an operation timing association feature extractor based on a three-dimensional convolutional network model to obtain an assembly operation timing feature graph and a reference operation timing feature graph. Those of ordinary skill in the art should understand that three-dimensional convolutional network models are widely used in the field of video processing, and they can effectively capture timing information in video sequences. By inputting the assembly operation key frame sequence and the reference operation key frame sequence into an operation timing association feature extractor based on a three-dimensional convolutional network model, they can be subjected to timing feature extraction, thereby learning key features such as action sequence, posture changes, and motion trajectory in the operation sequence. These features can reflect the dynamic changes and laws of assembly operations, and help to more comprehensively understand and characterize the user's assembly operations and reference operations.

[0064] Correspondingly, the timing analysis unit 142 includes: a preprocessing subunit, used to preprocess the assembly operation video and the reference operation video to obtain a sequence of assembly operation key frames and a sequence of reference operation key frames; and a timing association feature extraction subunit, used to pass the sequence of assembly operation key frames and the sequence of reference operation key frames through an operation timing association feature extractor based on a three-dimensional convolutional network model to obtain the assembly operation timing feature graph and the reference operation timing feature graph.

[0065] It is worth mentioning that the 3D convolutional neural network is a deep learning model for processing data with a time dimension. It is extended on the basis of the 2D convolutional neural network and can effectively process data with time series characteristics such as videos and time series. In the time series analysis unit 142, the time series association feature extraction subunit uses the operation time series association feature extractor based on the 3D convolutional network model to extract the assembly operation time series feature map and the reference operation time series feature map. Specifically, the 3D convolutional network can capture the characteristics and patterns of the data in time series by performing convolution operations on the time dimension. The 3D convolutional network model is usually composed of multiple convolutional layers, pooling layers and fully connected layers. It extracts features and represents learning of input data by performing convolution operations on three dimensions (width, height and time). The convolution layer can identify local time series patterns, the pooling layer can reduce the size and number of parameters of the feature map, and the fully connected layer can map the extracted features to specific categories or feature spaces. The 3D convolutional network has a wide range of applications in tasks such as video analysis, action recognition, and behavior recognition. It can extract temporal features from video sequences, capture the evolution of actions and temporal patterns, and plays an important role in the analysis and understanding of temporal data. In the industrial robot assembly training system, the use of a three-dimensional convolutional network model can effectively extract the temporal features of assembly operations and reference operations for subsequent analysis and evaluation.

[0066] In one example, the preprocessing subunit is used to: perform sparse sampling on the assembly operation video and the reference operation video to obtain a sequence of assembly operation key frames and a sequence of reference operation key frames.

[0067] It is worth mentioning that sparse sampling is a method of selecting a small number of key frames from a continuous data sequence. In video processing, sparse sampling is used to select a small number of key frames from a video sequence to reduce the amount of data and retain key information. In the preprocessing subunit, sparse sampling is used to process the assembly operation video and the reference operation video to obtain a sequence of assembly operation key frames and a sequence of reference operation key frames. Usually, a video is composed of continuous image frames, and the purpose of sparse sampling is to select a part of representative key frames from these continuous frames. The sparse sampling algorithm selects key frames from continuous frames according to a certain strategy. These key frames usually have the following characteristics: 1. Representativeness: The key frames can effectively represent the entire video sequence and contain important information and actions in the video. 2. Diversity: There should be certain differences between key frames to cover different scenes and actions in the video sequence. 3. Compressibility: The number of selected key frames is relatively small to reduce the amount of data and computational complexity. Sparse sampling can adopt different strategies according to different needs and application scenarios, such as sampling based on time intervals, sampling based on motion information, etc. Through sparse sampling, continuous video sequences can be converted into a small number of key frame sequences, which is convenient for subsequent processing and analysis.

[0068] Subsequently, the transfer matrix between the feature matrices of each group of corresponding channel dimensions of the assembly operation timing feature graph and the reference operation timing feature graph is calculated to obtain the operation difference semantic transfer representation feature graph. That is, the difference between the user's assembly operation and the reference operation is quantified by calculating the transfer matrix. Specifically, the transfer matrix reflects the transfer relationship or transfer probability between the feature matrices of each group of corresponding channel dimensions, and this transfer relationship can be used to measure the correlation and difference between the assembly operation and the reference operation.

[0069] Accordingly, the semantic difference extraction unit 143 is used to calculate the transfer matrix between the feature matrices of each group of corresponding channel dimensions of the assembly operation timing feature graph and the reference operation timing feature graph to obtain the operation difference semantic transfer representation feature graph.

[0070] Furthermore, the operation difference semantic transfer representation feature map is passed through a local operation saliency device based on an adaptive attention layer to obtain a salient operation difference semantic transfer representation feature map. Here, the adaptive attention layer can dynamically adjust the weights of different positions according to the characteristics of the input data to achieve selective attention to the input data. That is, through the local operation saliency device based on the adaptive attention layer, the operation difference semantic transfer representation feature map can be weighted so that the operation differences and important semantic information of the key channels are highlighted. This can improve the perception of operation differences while reducing attention to irrelevant or minor information. Subsequently, the salient operation difference semantic transfer representation feature map is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

[0071] Accordingly, the deviation analysis unit 144 includes: a local operation saliency subunit, used for passing the operation difference semantic transfer representation feature map through a local operation saliency based on an adaptive attention layer to obtain a salient operation difference semantic transfer representation feature map; and a classification subunit, used for passing the salient operation difference semantic transfer representation feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

[0072] It should be understood that the main function of the local operation salient subunit is to process the operation difference semantic transfer representation feature map through the local operation salient based on the adaptive attention layer to obtain a salient operation difference semantic transfer representation feature map. The purpose of this subunit is to highlight the significant differences in the assembly operation so as to better understand and analyze the user's assembly operation. The adaptive attention layer is a deep learning module used to learn the importance weights of different positions in the input feature map. Through the adaptive attention layer, the local operation salient subunit can automatically focus on important areas according to the content of the operation difference semantic transfer representation feature map, thereby generating a salient operation difference semantic transfer representation feature map. The main function of the classification subunit is to process the salient operation difference semantic transfer representation feature map through a classifier to obtain a classification result. This subunit is used to determine whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold. The classifier is a machine learning model that performs classification by learning the relationship between the salient operation difference semantic transfer representation feature map and different categories (such as normal operation and abnormal operation). The classifier can determine the degree of difference between the user's assembly operation and the standard operation based on the information in the feature map and give the corresponding classification result. Through the classification subunit, the assembly operation can be compared with the standard operation and its deviation can be judged whether it exceeds the predetermined threshold. This can timely detect the user's operation errors or anomalies and help improve the accuracy and quality of the assembly operation.

[0073] In one example, the local operation saliency subunit is used to: process the operation difference semantic transfer representation feature map using the following adaptive attention formula to obtain the saliency operation difference semantic transfer representation feature map; wherein the adaptive attention formula is:

[0074] v c =pool(F)

[0075] A=σ(W a *v c +b a )

[0076]

[0077] F′=A′⊙F

[0078] Where F is the semantic transfer representation feature map of the operation difference, pool is the pooling process, and v c is the pooling vector, W a is the weight matrix, b a is the bias vector, σ is the activation process, A is the initial element weight feature vector, A iis the eigenvalue of the i-th position in the initial meta-weight feature vector, A′ is the corrected meta-weight feature vector, F′ is the significant operation difference semantic transfer representation feature map, and ⊙ represents the multiplication of each feature matrix of the operation difference semantic transfer representation feature map along the channel dimension with the eigenvalue in the corrected meta-weight feature vector as the weight.

[0079] In the technical solution of the present application, the assembly operation timing feature graph and the reference operation timing feature graph respectively express the image semantic features of the sequence of the assembly operation key frames and the timing association of the reference operation key frames. Thus, the assembly operation timing feature graph and the reference operation timing feature graph have feature distribution information representation differences caused by differences in the timing association of the semantic distribution of the source images.

[0080] Therefore, when calculating the transfer matrix between the feature matrices of each group of corresponding channel dimensions of the assembly operation timing feature graph and the reference operation timing feature graph, the difference in feature distribution information representation between the assembly operation timing feature graph and the reference operation timing feature graph may lead to poor consistency in the representation of channel association distribution information between the feature matrices of the operation difference semantic transfer representation feature graph, thereby further leading to sparse channel distribution after passing through the local operation saliency device based on the adaptive attention layer, affecting the expression effect of the salient operation difference semantic transfer representation feature graph. Based on this, in order to improve the channel distribution constraint of the salient operation difference semantic transfer representation feature graph, the present application optimizes the salient operation difference semantic transfer representation feature graph.

[0081] Correspondingly, the classification subunit includes: a feature graph optimization secondary subunit, which is used to optimize the significant operation difference semantic transfer representation feature graph to obtain an optimized significant operation difference semantic transfer representation feature graph; and a deviation classification secondary subunit, which is used to pass the optimized significant operation difference semantic transfer representation feature graph through a classifier to obtain a classification result, and the classification result is used to indicate whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

[0082] Among them, the feature map optimization secondary sub-unit includes: expanding the significant operation difference semantic transfer representation feature map into a significant operation difference semantic transfer representation feature vector; calculating the first weight matrix and the second weight matrix of the significant operation difference semantic transfer representation feature vector, wherein the value of each position of the first weight matrix is ​​the mean of the two eigenvalues ​​of the corresponding position of the significant operation difference semantic transfer representation feature vector, and the value of each position of the second weight matrix is ​​the variance of the two eigenvalues ​​of the corresponding position of the significant operation difference semantic transfer representation feature vector; matrix multiplying the transposed vector of the significant operation difference semantic transfer representation feature vector as a row feature vector with the first weight matrix to obtain a first intermediate vector and Perform matrix multiplication of the second weight matrix and the saliency operation difference semantic transfer representation feature vector to obtain a second intermediate vector; calculate the dot sum of the first intermediate vector and the transposed vector of the second intermediate vector to obtain a third intermediate vector; calculate the matrix product of the first weight matrix and the second weight matrix, and perform matrix multiplication of the transposed vector of the saliency operation difference semantic transfer representation feature vector and the matrix product to obtain a fourth intermediate vector; calculate the dot sum of the transposed vector of the third intermediate vector and the fourth intermediate vector to obtain an optimized saliency operation difference semantic transfer representation feature vector; and restore the optimized saliency operation difference semantic transfer representation feature vector to an optimized saliency operation difference semantic transfer representation feature graph.

[0083] That is, by using the weight matrix of the group aggregation statistical evaluation of the local distribution of the feature value granularity of the saliency operation difference semantic transfer representation feature map as the retrieval and response distribution enhancement of the saliency operation difference semantic transfer representation feature map, and constructing a reference-free distribution retrieval response framework under an open domain of the feature distribution based on group aggregation of the saliency operation difference semantic transfer representation feature map, the redundancy of distribution responses caused by the local overflow characteristics of the saliency operation difference semantic transfer representation feature map can be avoided through response superposition, so as to achieve the fidelity constraint related to the self-aggregation statistics of the response of the saliency operation difference semantic transfer representation feature map to the classification target domain, so as to improve the expression effect of the saliency operation difference semantic transfer representation feature map.

[0084] Furthermore, the deviation classification secondary sub-unit is used to: expand the optimized saliency operation difference semantic transfer representation feature map into a classification feature vector according to a row vector or a column vector; use the fully connected layer of the classifier to fully connect the classification feature vector to obtain an encoded classification feature vector; and input the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.

[0085] That is, in the technical solution of the present application, the label of the classifier includes the deviation between the user's assembly operation and the standard operation exceeding the predetermined threshold (first label), and the deviation between the user's assembly operation and the standard operation does not exceed the predetermined threshold (second label), wherein the classifier determines which classification label the optimized significant operation difference semantic transfer representation feature map belongs to through the soft maximum function. It is worth noting that the first label p1 and the second label p2 here do not contain artificially set concepts. In fact, during the training process, the computer model does not have the concept of "whether the deviation between the user's assembly operation and the standard operation exceeds the predetermined threshold". It only has two classification labels and the probability of the output feature under these two classification labels, that is, the sum of p1 and p2 is one. Therefore, the classification result of whether the deviation between the user's assembly operation and the standard operation exceeds the predetermined threshold is actually converted into a binary classification class probability distribution that conforms to the law of nature through the classification label. In essence, the physical meaning of the natural probability distribution of the label is used, rather than the linguistic text meaning of "whether the deviation between the user's assembly operation and the standard operation exceeds the predetermined threshold".

[0086] It should be understood that the role of the classifier is to use the given categories and known training data to learn classification rules and classifiers, and then classify (or predict) unknown data. Logistic regression and SVM are often used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-classification. However, this is prone to errors and is not efficient. Commonly used multi-classification methods include the Softmax classification function.

[0087] In summary, the virtual simulation-based industrial robot assembly training system 100 according to the embodiment of the present application is explained, which can achieve an objective evaluation of the user's assembly operation.

[0088] As described above, the industrial robot assembly training system 100 based on virtual simulation according to the embodiment of the present application can be implemented in various terminal devices, such as a server with an industrial robot assembly training algorithm based on virtual simulation. In one example, the industrial robot assembly training system 100 based on virtual simulation can be integrated into the terminal device as a software module and / or a hardware module. For example, the industrial robot assembly training system 100 based on virtual simulation can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the industrial robot assembly training system 100 based on virtual simulation can also be one of the many hardware modules of the terminal device.

[0089] Alternatively, in another example, the virtual simulation-based industrial robot assembly training system 100 and the terminal device may also be separate devices, and the virtual simulation-based industrial robot assembly training system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0090] Figure 3 FIG. 1 is a flowchart of an industrial robot assembly training method based on virtual simulation according to an embodiment of the present application. Figure 3 As shown, according to the virtual simulation-based industrial robot assembly training method of the embodiment of the present application, it includes: S110, providing a virtual scene and virtual equipment for industrial robot assembly training through a virtual simulation platform; S120, interacting with the virtual simulation platform through a virtual reality device to realize the operation and control of the virtual scene and the virtual equipment; S130, receiving an input signal of the virtual reality device through a data processing module, analyzing and processing the input signal, generating a corresponding feedback signal, and sending the feedback signal to the virtual reality device; and, S140, evaluating the user's assembly operation through an evaluation module.

[0091] In one possible implementation, Figure 4 As shown, the user's assembly operation is evaluated through an evaluation module, including: obtaining the user's assembly operation video captured by a camera, and a reference operation video; performing timing analysis on the assembly operation video and the reference operation video to obtain an assembly operation timing feature graph and a reference operation timing feature graph; extracting the operation semantic difference feature between the assembly operation timing feature graph and the reference operation timing feature graph to obtain an operation difference semantic transfer representation feature graph; and, based on the operation difference semantic transfer representation feature graph, determining whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

[0092] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned industrial robot assembly training method based on virtual simulation have been referred to above. Figure 1 to Figure 2 It has been introduced in detail in the description of the industrial robot assembly training system based on virtual simulation, and therefore, its repeated description will be omitted.

[0093] Figure 5 FIG. 1 shows an application scenario diagram of an industrial robot assembly training system based on virtual simulation according to an embodiment of the present application. Figure 5 As shown, in this application scenario, first, a user's assembly operation video captured by a camera is obtained (for example, Figure 5 D1 as shown in the figure), and, refer to the operation video (e.g., Figure 5Then, the assembly operation video and the reference operation video are input to a server (for example, Figure 5 In S) shown in , the server is able to use the virtual simulation-based industrial robot assembly training algorithm to process the assembly operation video and the reference operation video to obtain a classification result indicating whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

[0094] In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as a memory including computer program instructions, which can be executed by a processing component of an apparatus to perform the above method.

[0095] The present application may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0096] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0097] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the function or action of the specification, or can be realized by a combination of special-purpose hardware and computer instructions.

[0098] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An industrial robot assembly training system based on virtual simulation, characterized in that: include: Virtual simulation platform, used to provide virtual scenes and virtual equipment for industrial robot assembly training; A virtual reality device, used to interact with the virtual simulation platform to operate and control the virtual scene and the virtual device; A data processing module, used for receiving an input signal of the virtual reality device, analyzing and processing the input signal, generating a corresponding feedback signal, and sending the feedback signal to the virtual reality device; as well as The evaluation module is used to evaluate the user's assembly operation.

2. The industrial robot assembly training system based on virtual simulation according to claim 1 is characterized in that: The evaluation module comprises: A data acquisition unit, used to acquire the assembly operation video of the user acquired by a camera, and a reference operation video; A timing analysis unit, used for performing timing analysis on the assembly operation video and the reference operation video to obtain an assembly operation timing characteristic diagram and a reference operation timing characteristic diagram; A semantic difference extraction unit, used for extracting the operation semantic difference feature between the assembly operation timing feature graph and the reference operation timing feature graph to obtain an operation difference semantic transfer representation feature graph; and The deviation analysis unit is used to determine whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold based on the operation difference semantic transfer representation feature map.

3. The industrial robot assembly training system based on virtual simulation according to claim 2 is characterized in that: The timing analysis unit comprises: a preprocessing subunit, configured to preprocess the assembly operation video and the reference operation video to obtain a sequence of assembly operation key frames and a sequence of reference operation key frames; and The timing correlation feature extraction subunit is used to obtain the assembly operation timing feature graph and the reference operation timing feature graph by passing the sequence of the assembly operation key frames and the sequence of the reference operation key frames through an operation timing correlation feature extractor based on a three-dimensional convolutional network model.

4. The industrial robot assembly training system based on virtual simulation according to claim 3 is characterized in that: The pre-processing sub-unit is used for: The assembly operation video and the reference operation video are sparsely sampled to obtain a sequence of the assembly operation key frames and a sequence of the reference operation key frames.

5. The industrial robot assembly training system based on virtual simulation according to claim 4 is characterized in that: The semantic difference extraction unit is used to: The transfer matrix between the feature matrices of each group of corresponding channel dimensions of the assembly operation timing feature graph and the reference operation timing feature graph is calculated to obtain the operation difference semantic transfer representation feature graph.

6. The industrial robot assembly training system based on virtual simulation according to claim 5 is characterized in that: The deviation analysis unit comprises: A local operation saliency subunit, configured to pass the operation difference semantic transfer representation feature map through a local operation saliency device based on an adaptive attention layer to obtain a salient operation difference semantic transfer representation feature map; and The classification subunit is used to pass the semantic transfer representation feature map of the significant operation difference through a classifier to obtain a classification result, and the classification result is used to indicate whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

7. The industrial robot assembly training system based on virtual simulation according to claim 6 is characterized in that: The local operation significant subunit is used to: The operation difference semantic transfer representation feature map is processed by the following adaptive attention formula to obtain the salient operation difference semantic transfer representation feature map; wherein the adaptive attention formula is: v c =pool(F) A=σ(W a *v c +b a ) F′=A′⊙F Where F is the semantic transfer representation feature map of the operation difference, pool is the pooling process, and v c is the pooling vector, W a is the weight matrix, b a is the bias vector, σ is the activation process, A is the initial element weight feature vector, A i is the eigenvalue of the ith position in the initial meta-weight feature vector, A' is the corrected meta-weight feature vector, F' is the significant operation difference semantic transfer representation feature map, and ⊙ represents the multiplication of each feature matrix of the operation difference semantic transfer representation feature map along the channel dimension with the eigenvalue in the corrected meta-weight feature vector as the weight.

8. The industrial robot assembly training system based on virtual simulation according to claim 7 is characterized in that: The classification subunit comprises: A feature map optimization secondary subunit, used for optimizing the saliency operation difference semantic transfer representation feature map to obtain an optimized saliency operation difference semantic transfer representation feature map; and The deviation classification secondary subunit is used to pass the optimized significant operation difference semantic transfer representation feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether the deviation between the user's assembly operation and the standard operation exceeds a predetermined threshold.

9. The industrial robot assembly training system based on virtual simulation according to claim 8, characterized in that: The deviation classification secondary subunit is used for: Expanding the optimized saliency operation difference semantic transfer representation feature map into a classification feature vector according to a row vector or a column vector; Performing full connection encoding on the classification feature vector using the fully connected layer of the classifier to obtain an encoded classification feature vector; as well as The encoded classification feature vector is input into the Softmax classification function of the classifier to obtain the classification result.

10. A virtual simulation-based industrial robot assembly training method, characterized in that: include: Provide virtual scenes and virtual equipment for industrial robot assembly training through virtual simulation platform; The virtual scene and the virtual device are operated and controlled by interacting with the virtual simulation platform through a virtual reality device; receiving an input signal of the virtual reality device through a data processing module, analyzing and processing the input signal, generating a corresponding feedback signal, and sending the feedback signal to the virtual reality device; as well as The user's assembly operations are evaluated through the evaluation module.

Images