Intelligent assembly system and method for handicrafts and ornaments based on visual inspection

Abstract

The invention provides an intelligent assembly system and method for handicrafts and ornaments based on visual inspection, and relates to the field of intelligent recognized.The intelligent recognition technology based on machine vision is adopted, and firstly, images of a workpiece to be recognized at multiple angles are obtained; secondly, performing high-dimensional space feature extraction on the images of the to-be-recognized workpiece at multiple angles to obtain workpiece angle correlation features, and finally, judging the workpiece type according to the workpiece angle correlation features through a classifier. In this way, the accuracy of workpiece recognition can be improved, and therefore the assembly precision is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent identification, and more specifically, to a system and method for intelligent assembly of handicraft ornaments based on visual detection. Background Art

[0002] In modern manufacturing, automated assembly lines are key to improving production efficiency and product quality. However, accurate identification of workpieces is the primary challenge in achieving automated assembly. As a special product, handicrafts are often designed with a high degree of artistry and complexity. These ornaments contain fine textures, diverse shapes, and complex symmetrical structures. These characteristics bring additional difficulty to workpiece identification during automated assembly. For example, when some parts of the ornaments are visually difficult to distinguish, image recognition at a single angle will result in an inability to accurately distinguish the parts, leading to assembly errors.

[0003] Therefore, an optimized intelligent assembly solution for handicraft ornaments based on visual inspection is needed. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present application provides a system and method for intelligent assembly of handicraft ornaments based on visual inspection.

[0005] According to one aspect of the present application, there is provided a handicraft ornament intelligent assembly system based on visual detection, which includes:

[0006] A multi-angle workpiece image data acquisition module is used to obtain images of the workpiece to be identified taken at different angles;

[0007] A multi-angle workpiece image data encoding module is used to perform feature encoding on the workpiece images to be identified taken at different angles to obtain a multi-scale workpiece angle correlation feature map;

[0008] A multi-angle workpiece correlation feature enhancement module is used to perform channel feature enhancement on the multi-scale workpiece angle correlation feature map to obtain a multi-scale workpiece angle correlation enhanced feature map;

[0009] The workpiece type label result generation module is used to determine the workpiece type label of the workpiece to be identified based on the multi-scale workpiece angle association enhancement feature map.

[0010] According to another aspect of the present application, a method for intelligent assembly of handicraft ornaments based on visual detection is provided, which includes:

[0011] Acquire images of workpieces to be identified taken at different angles;

[0012] Performing feature encoding on the images of the workpiece to be identified taken at different angles to obtain a multi-scale workpiece angle-related feature map;

[0013] Performing channel feature enhancement on the multi-scale workpiece angle correlation feature map to obtain a multi-scale workpiece angle correlation enhanced feature map;

[0014] Based on the multi-scale workpiece angle association enhanced feature map, the workpiece type label to which the workpiece to be identified belongs is determined.

[0015] This application has significant technical effects due to the adoption of the above technical solutions:

[0016] The intelligent assembly system and method of handicraft ornaments based on visual inspection provided by the present application adopts intelligent recognition technology based on machine vision, firstly obtains images of multiple angles of the workpiece to be identified, then extracts features of the workpiece angle association features in high-dimensional space from the images of multiple angles of the workpiece to be identified, and finally passes the workpiece angle association features through a classifier to realize the judgment of the workpiece type. In this way, the accuracy of workpiece recognition can be improved, thereby improving the assembly accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 The system block diagram of the handicraft ornament intelligent assembly system based on visual inspection according to the embodiment of the present application.

[0019] Figure 2 It is a block diagram of a multi-angle workpiece image data encoding module in a handicraft ornament intelligent assembly system based on visual inspection according to an embodiment of the present application.

[0020] Figure 3 It is a block diagram of a multi-scale workpiece angle correlation feature generation unit in a handicraft ornament intelligent assembly system based on visual inspection according to an embodiment of the present application.

[0021] Figure 4 It is a block diagram of a workpiece type label result generation module in a handicraft ornament intelligent assembly system based on visual inspection according to an embodiment of the present application.

[0022] Figure 5 It is a flow chart of the intelligent assembly method of handicraft ornaments based on visual inspection according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0024] In modern manufacturing, automated assembly lines are a key factor in improving production efficiency and product quality. With the development of technology, automated assembly lines have been widely used in many fields such as automobiles, electronics, and food processing. Workpiece recognition is a vital part of automated assembly lines because the assembly system needs to accurately grasp, locate, and assemble workpieces based on the type, shape, size, and other characteristics of the workpiece to ensure the quality and consistency of the final product. In other words, workpiece recognition provides the assembly system with the necessary workpiece information, enabling the system to make correct decisions and operations. However, accurate identification of workpieces remains a major challenge in achieving automated assembly, especially in the manufacturing of highly artistic and complex products such as handicrafts.

[0025] The manufacture of handicraft ornaments usually requires extremely high precision and meticulous attention to detail. The design of these ornaments often combines the creativity of artists and the exquisite skills of craftsmen, and includes fine textures, diverse shapes, and complex symmetrical structures. These characteristics bring additional difficulties to the automated assembly process. For example, when some parts of the ornaments are visually difficult to distinguish, traditional single-angle image recognition technology may not be able to accurately distinguish the parts, resulting in assembly errors. Therefore, an optimized intelligent assembly solution for handicraft ornaments based on visual inspection is expected.

[0026] In recent years, deep learning and neural networks have been widely used in computer vision, natural language processing, text signal processing and other fields. In addition, deep learning and neural networks have also demonstrated a level close to or even beyond that of humans in image classification, object detection, semantic segmentation, text translation and other fields. The development of deep learning and neural networks has provided new solutions and solutions for the intelligent assembly of handicrafts based on visual inspection.

[0027] Figure 1 FIG. 1 is a system block diagram of an intelligent assembly system for handicraft ornaments based on visual detection according to an embodiment of the present application. Figure 1As shown, in the intelligent assembly system 100 of handicraft ornaments based on visual inspection, it includes: a multi-angle workpiece image data acquisition module 110, which is used to obtain the images of the workpiece to be identified taken at different angles; a multi-angle workpiece image data encoding module 120, which is used to perform feature encoding on the images of the workpiece to be identified taken at different angles to obtain a multi-scale workpiece angle association feature map; a multi-angle workpiece association feature enhancement module 130, which is used to perform channel feature enhancement on the multi-scale workpiece angle association feature map to obtain a multi-scale workpiece angle association enhancement feature map; a workpiece type label result generation module 140, which is used to determine the workpiece type label to which the workpiece to be identified belongs based on the multi-scale workpiece angle association enhancement feature map.

[0028] Specifically, in the technical solution of the present application, first, it is necessary to obtain images of the workpiece to be identified taken at different angles. It should be understood that the workpiece to be identified is a component of a handicraft ornament, which often has a complex shape and three-dimensional structure. It is difficult for an image from a single angle to fully reflect the overall characteristics of the workpiece to be identified. By taking images from different angles, the appearance information of the workpiece under various viewing angles can be obtained, so as to have a more comprehensive understanding of the shape, size and other characteristics of the workpiece, so as to identify the type of workpiece. In other words, images at different angles can provide different viewing angle features, which may not be obvious or easy to confuse under a single viewing angle. By integrating image information from multiple angles, the recognition between workpieces can be improved and the possibility of misidentification can be reduced.

[0029] In an embodiment of the present application, a possible implementation method for obtaining images of the workpiece to be identified taken at different angles may be: a. Determine the shooting angle: Generally, the front, left side, right side, top and other key directions of the workpiece to be identified will be photographed, and more angles may be determined according to actual needs; b. Adjust camera parameters: It is necessary to deploy industrial cameras at the angles selected for shooting, and adjust the camera parameters to obtain the best image quality; c. Synchronous shooting: Multiple industrial cameras capture images of the workpiece from different angles at the same time.

[0030] Figure 2 FIG. 1 is a block diagram of a multi-angle workpiece image data encoding module in a handicraft ornament intelligent assembly system based on visual inspection according to an embodiment of the present application. Figure 2As shown, the multi-angle workpiece image data encoding module 120 includes: a target recognition workpiece image feature extraction unit 121, which is used to pass the workpiece images to be recognized taken at different angles through a workpiece feature encoder to obtain multiple target recognition workpiece feature matrices; a multi-angle target recognition workpiece feature integration unit 122, which is used to aggregate the multiple target recognition workpiece feature matrices along the channel dimension into a target recognition workpiece three-dimensional feature tensor; a multi-scale workpiece angle association feature generation unit 123, which is used to perform multi-scale feature extraction on the target recognition workpiece three-dimensional feature tensor to obtain the multi-scale workpiece angle association feature map.

[0031] Considering that directly identifying from the original image of the workpiece is not only computationally intensive, but also easily interfered by external factors such as noise and illumination changes, in order to extract key features useful for the workpiece type recognition task from the workpiece image to be identified and reduce redundant information at the same time, the workpiece images to be identified taken at different angles can be respectively passed through the workpiece feature encoder to obtain multiple target recognition workpiece feature matrices. For workpieces with complex shapes and textures, images at different angles contain rich shape, texture and spatial position information. By extracting features from images at different angles through the feature encoder, a series of highly representative feature matrices can be obtained to comprehensively reflect the characteristics of the workpiece at different angles. In this application, the workpiece feature encoder is a convolutional neural network model including a deep and shallow feature fusion module. Those of ordinary skill in the art should know that in a convolutional neural network, shallow feature extraction usually involves the detection of basic visual elements such as edges, corners, and textures of the image. As the number of convolutional neural network layers increases, feature extraction gradually transitions from a low level to a high level, and the network can learn more abstract and complex feature representations. That is, shallow feature extraction can capture low-level features such as the basic outline and texture pattern of the workpiece, while deep features can capture high-level features such as the relationship between the part and the whole of the workpiece. By fusing deep and shallow features, the network can make comprehensive use of these two types of feature information, thereby improving the accuracy and robustness of workpiece recognition.

[0032] Considering that the process of processing the images of the workpiece to be identified taken at different angles through the workpiece feature encoder is the same, therefore, taking the processing of an image at one angle as an example, in an embodiment of the present application, a specific implementation method of passing the image of the workpiece to be identified taken at a certain angle through the workpiece feature encoder to obtain a target identification workpiece feature matrix is ​​as follows: extracting the target identification workpiece shallow feature matrix from the i-th layer of the workpiece feature encoder, the i-th layer is the first to sixth layers of the workpiece feature encoder; extracting the target identification workpiece deep feature matrix from the j-th layer of the workpiece feature encoder, the ratio between the j-th layer and the i-th layer is greater than or equal to 5; and, using the deep and shallow feature fusion module to fuse the target identification workpiece shallow feature matrix and the target identification workpiece deep feature matrix to obtain the target identification workpiece feature matrix.

[0033] Workpiece images taken at different angles provide visual information of the workpiece at different viewing angles. In order to make full use of the complementarity of this information and better identify the type of workpiece, the technical solution of the present application aggregates the multiple target recognition workpiece feature matrices along the channel dimension into a target recognition workpiece three-dimensional feature tensor. It should be understood that each target recognition workpiece feature matrix is ​​obtained by extracting features from the image of the workpiece to be identified at a certain angle. Aggregation along the channel dimension is actually to fuse the workpiece feature matrices extracted at different angles at the feature level to form a three-dimensional feature tensor containing more comprehensive information.

[0034] Figure 3 FIG. 1 is a block diagram of a multi-scale workpiece angle correlation feature generation unit in a handicraft ornament intelligent assembly system based on visual inspection according to an embodiment of the present application. Figure 3 As shown, the multi-scale workpiece angle association feature generating unit 123 includes: a first-scale workpiece angle association feature generating subunit 1231, used for passing the target identification workpiece three-dimensional feature tensor through a first three-dimensional workpiece angle association feature encoder to obtain a first-scale workpiece angle association feature map; a second-scale workpiece angle association feature generating subunit 1232, used for passing the target identification workpiece three-dimensional feature tensor through a second three-dimensional workpiece angle association feature encoder to obtain a second-scale workpiece angle association feature map; a multi-scale feature fusion subunit 1233, used for fusing the first-scale workpiece angle association feature map and the second-scale workpiece angle association feature map to obtain the multi-scale workpiece angle association feature map.

[0035] Although the target identification workpiece three-dimensional feature tensor contains workpiece image information from different angles, it still contains some redundant information, which does not contribute much to the identification of workpiece types. In order to remove redundant information and learn the common features and difference features of workpieces at different viewing angles, it is necessary to pass the target identification workpiece three-dimensional feature tensor through a first three-dimensional workpiece angle-associated feature encoder to obtain a first-scale workpiece angle-associated feature map. The essence of the first three-dimensional workpiece angle-associated feature encoder in this application is a convolutional neural network model using a first-scale three-dimensional convolution kernel. The three-dimensional convolution kernel has an additional channel dimension compared to the commonly used two-dimensional convolution kernel, which refers to the angle dimension in this application. The three-dimensional convolution kernel can slide directly on the target identification workpiece three-dimensional feature tensor to capture the associated features of the workpiece at different angles, thereby learning a more accurate and comprehensive workpiece feature representation.

[0036] In an embodiment of the present application, a possible implementation method of passing the target identification workpiece three-dimensional feature tensor through a first three-dimensional workpiece angle association feature encoder to obtain a first-scale workpiece angle association feature map may be as follows: each layer of the first three-dimensional workpiece angle association feature encoder performs the following on the input data during the forward transmission process of the layer: performing convolution processing on the input data based on a first-scale three-dimensional convolution kernel to obtain a convolution feature map; performing mean pooling processing on the convolution feature map based on a local feature matrix to obtain a pooling feature map; performing nonlinear activation on the pooling feature map to obtain an activation feature map; wherein the output of the last layer of the first three-dimensional workpiece angle association feature encoder is the first-scale workpiece angle association feature map, and the input of the first layer of the first three-dimensional workpiece angle association feature encoder is the target identification workpiece three-dimensional feature tensor.

[0037] At the same time, the target identification workpiece three-dimensional feature tensor is passed through a second three-dimensional workpiece angle association feature encoder to obtain a second-scale workpiece angle association feature map. In the present application, the second three-dimensional workpiece angle association feature encoder is a convolutional neural network model using a second-scale three-dimensional convolution kernel, and the first scale is different from the second scale. The scale of a three-dimensional convolution kernel generally refers to its size in the spatial dimension (i.e., width, height) and channel (i.e., angle in the present application) dimension. In deep learning, convolution kernels of different scales have different receptive fields and can capture feature information of different ranges. Specifically, small-scale convolution kernels are better at capturing local detail information, while large-scale convolution kernels can capture a wider range of spatial and channel correlations. The second three-dimensional workpiece angle association feature encoder uses a second-scale three-dimensional convolution kernel to extract features from the target identification workpiece three-dimensional feature tensor, and can obtain workpiece angle association feature information of different scales.

[0038] Next, the first-scale workpiece angle-related feature map and the second-scale workpiece angle-related feature map are fused to obtain a multi-scale workpiece angle-related feature map. It should be understood that the first-scale and second-scale workpiece angle-related feature maps are respectively obtained by extracting the three-dimensional feature map of the target recognition workpiece using three-dimensional convolution kernels of different sizes. In order to make full use of the feature information at different scales and more accurately capture the workpiece features, the workpiece angle-related feature maps of different scales can be fused. By analyzing the fused feature maps, the accuracy of workpiece type recognition can be improved. It is also worth mentioning that in practical applications, due to the influence of factors such as illumination changes, occlusion, and noise, a single-scale feature map may not be able to stably express the features of the workpiece. The fusion of multi-scale feature maps can complement each other through feature information at different scales, reduce the influence of these factors on the recognition results, and thus improve the robustness of the model.

[0039] In order to strengthen the expression of workpiece features in the multi-scale workpiece angle association feature map and help the model capture important features for workpiece type identification, the multi-scale workpiece angle association feature map is passed through a feature enhancer based on a channel attention mechanism to obtain a multi-scale workpiece angle association enhanced feature map. The essence of the feature enhancer based on the channel attention mechanism in this application is a convolutional neural network model using a channel attention mechanism. By introducing the channel attention mechanism in the convolutional neural network model, the importance of different channel features can be adaptively adjusted to achieve enhanced processing of the feature map. Specifically, it evaluates each channel of the multi-scale workpiece angle association feature map, assigns different weights according to the importance of the channel features, and then enhances the expression of important features and suppresses unimportant features. That is, for the feature channels that contribute more to the recognition task in the multi-scale workpiece angle association feature map, the model will assign higher weights to enhance the expression of these features; while for those feature channels that contribute less, the model will assign lower weights, or even suppress the expression of these features, so that the representation ability of the feature map can be effectively improved, providing better input for subsequent classification. And by reducing the weights of unimportant features, the channel attention mechanism can also reduce the waste of computing resources to a certain extent. In deep learning models, computing resources are usually limited, so the reasonable allocation of computing resources is of great significance to improving the efficiency and performance of the model. The channel attention mechanism optimizes the allocation of computing resources by adaptively adjusting feature weights, so that the model can reduce computing costs and time consumption while maintaining high recognition accuracy.

[0040] In an embodiment of the present application, a possible implementation method of passing the multi-scale workpiece angle association feature map through a feature enhancer based on a channel attention mechanism to obtain a multi-scale workpiece angle association enhanced feature map can be: each layer of the feature enhancer based on the channel attention mechanism performs the following on the input data in the forward transfer of the layer: convolution processing on the input data based on a two-dimensional convolution kernel to generate a convolution feature map; pooling processing on the convolution feature map to generate a pooling feature map; activation processing on the pooling feature map to generate an activation feature map; calculating the quotient of the eigenvalue mean of the feature matrix corresponding to each channel in the activation feature map and the sum of the eigenvalue means of the feature matrices corresponding to all channels as the weighting coefficient of the feature matrix corresponding to each channel; weighting the feature matrix of each channel with the weighting coefficient of each channel in the activation feature map to generate a channel attention feature map; wherein the input of the first layer of the feature enhancer based on the channel attention mechanism is the multi-scale workpiece angle association feature map, and the output of the last layer of the feature enhancer based on the channel attention mechanism is the multi-scale workpiece angle association enhanced feature map.

[0041] Figure 4 FIG. 1 is a block diagram of a workpiece type label result generation module in a handicraft ornament intelligent assembly system based on visual inspection according to an embodiment of the present application. Figure 4 As shown, the workpiece type label result generation module 140 includes: a feature map dimension reduction unit 141, which is used to pool the various feature matrices of the multi-scale workpiece angle association enhancement feature map along the channel dimension to obtain a multi-scale workpiece angle association enhancement feature vector; a workpiece feature optimization unit 142, which is used to perform backward learning-type attention optimization based on a feature analysis network on the multi-scale workpiece angle association enhancement feature vector to obtain an optimized multi-scale workpiece angle association enhancement feature vector; a workpiece feature classification unit 143, which is used to pass the optimized multi-scale workpiece angle association enhancement feature vector through a workpiece type classifier to obtain a classification result, and the classification result is the workpiece type label of the workpiece to be identified.

[0042] Considering that the original multi-scale workpiece angle correlation enhanced feature map contains a large number of data points, directly processing these data will result in a large amount of calculation. In order to significantly reduce the dimension of the feature map while retaining key features, in the technical solution of the present application, the multi-scale workpiece angle correlation enhanced feature map is pooled along each feature matrix of the channel dimension to obtain a multi-scale workpiece angle correlation enhanced feature vector. Through the pooling operation, the dimension of the feature map is reduced, which will greatly reduce the amount of calculation of subsequent processing steps, thereby making the model operation more efficient.

[0043] In particular, in the technical solution of the present application, it is considered that the workpiece feature encoder may not be able to fully capture the workpiece features at all angles, especially when certain features of the workpiece are more obvious at a specific angle. Aggregating multiple target recognition workpiece feature maps into a three-dimensional feature tensor along the channel dimension may lose some important spatial information. The first and second three-dimensional workpiece angle-related feature encoders may not be able to uniformly encode features at all angles, resulting in over-emphasis or neglect of features at certain angles. Fusion of workpiece angle-related feature maps of different scales is a complex process that may introduce additional noise or errors. The feature enhancer based on the channel attention mechanism may not be able to uniformly enhance all features, and some important features may be ignored. Global pooling based on the feature matrix may lose some local features, especially when certain key features of the workpiece are concentrated in a specific area of ​​the image. Due to the above reasons, the multi-scale workpiece angle-related enhanced feature vector will have a class harmony offset relative to the predetermined class label of the workpiece type classifier. Therefore, in order to reduce the class harmony offset, in the technical solution of the present application, the multi-scale workpiece angle-related enhanced feature vector is subjected to backward learning-like attention optimization based on the feature analysis network.

[0044] Specifically, the multi-scale workpiece angle association enhancement feature vector is subjected to backward learning-like attention optimization based on a feature analysis network to obtain an optimized multi-scale workpiece angle association enhancement feature vector, including: performing probabilistic activation based on a Sigmoid function on the multi-scale workpiece angle association enhancement feature vector to obtain a probabilistic multi-scale workpiece angle association enhancement feature vector; inputting the multi-scale workpiece angle association enhancement feature vector into the workpiece type classifier to obtain an initial class label probability value; calculating the absolute value of the difference between the feature value of each position in the probabilistic multi-scale workpiece angle association enhancement feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error metric vector; and using the backward The machine learning fine-grained error metric vector is used as a key vector, and the multi-scale artifact angle association enhancement feature vector is used as a query vector and a value vector, and the key vector, the query vector and the value vector are input into a converter structure to obtain a backward learning type attention optimized multi-scale artifact angle association enhancement feature vector; the ratio between the sum of the backward learning type attention optimized multi-scale artifact angle association enhancement feature vector is calculated, and the natural exponential function value with the ratio as a power is calculated as an optimized scaling factor; the product between the optimized scaling factor and the backward learning type attention optimized multi-scale artifact angle association enhancement feature vector is calculated to obtain the optimized multi-scale artifact angle association enhancement feature vector.

[0045] In an embodiment of the present application, another expression of performing backward learning-type attention optimization based on a feature analysis network on the multi-scale workpiece angle association enhancement feature vector to obtain an optimized multi-scale workpiece angle association enhancement feature vector can be: processing the multi-scale workpiece angle association enhancement feature vector using the following formula to obtain the optimized multi-scale workpiece angle association enhancement feature vector; wherein the formula is:

[0046]

[0047] Among them, V c represents the probabilistic multi-scale workpiece angle association enhancement feature vector, P represents the initial class label probability value, Indicates subtraction by position, V e represents the fine-grained error metric vector for backward machine learning, v i represents the eigenvalue of the ith position of the backward machine learning fine-grained error metric vector, L represents the length of the backward machine learning fine-grained error metric vector, T represents the transpose of the vector, represents vector multiplication, Softmax represents the normalized exponential function, ⊙ represents point multiplication by position, V' c represents the optimized multi-scale workpiece angle association enhanced feature vector.

[0048] Based on this, in the technical solution of the present application, the multi-scale workpiece angle association enhancement feature vector is optimized by backward learning-like attention based on the feature analysis network. First, the multi-scale workpiece angle association enhancement feature vector is activated by the probabilistic activation based on the Sigmoid function to obtain a probabilistic multi-scale workpiece angle association enhancement feature vector, and the multi-scale workpiece angle association enhancement feature vector is input into the classifier to obtain the initial class label probability value. Then, the error between the probabilistic multi-scale workpiece angle association enhancement feature vector and the initial class label probability value is calculated by backward error analysis to generate a fine-grained error metric vector, which quantifies the viewpoint metric contribution of each feature position to the classification error.

[0049] Then, the backward machine learning fine-grained error metric vector is used as a key vector, and the multi-scale workpiece angle association enhancement feature vector is used as a query vector and a value vector, and the key vector, the query vector and the value vector are input into a transformer structure to perform error metric-based attention-based backward machine learning on the multi-scale workpiece angle association enhancement feature vector through a Transformer-like architecture. In this way, the multi-scale workpiece angle association enhancement feature vector is subjected to a backward interpretability analysis so that the obtained backward learning-like attention-optimized multi-scale workpiece angle association enhancement feature vector has a backward learning sensitive preference relative to the predetermined class label of the workpiece type classifier. And further, the ratio of the sum of the error metric vector and the vector length is calculated, and the optimized scaling factor obtained by calculating the natural exponential function value with this ratio as an exponent is used to perform backward scaling optimization (BackwardOptimization) on the backward learning-like attention-optimized multi-scale workpiece angle association enhancement feature vector, so that the sensitivity preference and robustness of the multi-scale workpiece angle association enhancement feature vector relative to the predetermined class label of the workpiece type classifier are improved, thereby improving the accuracy of the classification result.

[0050] Finally, the optimized multi-scale workpiece angle association enhanced feature vector is passed through the workpiece type classifier to obtain a classification result, and the classification result is the workpiece type label of the workpiece to be identified. Although the optimized multi-scale workpiece angle association enhanced feature vector contains rich feature information, it corresponds to the representation form of high-dimensional space features and is not directly a workpiece type label. In order to convert the feature information into a specific workpiece type label, a classifier is needed to establish a mapping relationship between features and categories. The classifier can determine which category the workpiece to be identified belongs to based on the information in the feature vector. That is, it is necessary to pass the optimized multi-scale workpiece angle association enhanced feature vector through the workpiece type classifier. The workpiece type label classifier is a machine learning model for mapping input features to predefined workpiece type labels. This model is usually obtained through training, and a large amount of labeled data is used during the training process to learn the mapping relationship between features and categories. The classification result representing the workpiece type label directly determines the assembly path, tools and equipment of the workpiece in the subsequent assembly process. In this way, the automation and high-precision identification of the workpiece type can be achieved, greatly improving the efficiency and accuracy of the assembly process.

[0051] In an embodiment of the present application, a possible implementation method of passing the optimized multi-scale workpiece angle association enhanced feature vector through a workpiece type classifier to obtain a classification result for representing a workpiece type label of the workpiece to be identified may be: using the fully connected layer of the workpiece type classifier to fully connect encode the optimized multi-scale workpiece angle association enhanced feature vector to obtain a fully connected encoded feature vector; and inputting the fully connected encoded feature vector into the Softmax classification function of the workpiece type classifier to obtain the classification result.

[0052] In summary, the intelligent assembly system 100 of handicraft ornaments based on visual inspection according to the embodiment of the present application is explained, which adopts intelligent recognition technology based on machine vision, firstly obtains images of multiple angles of the workpiece to be identified, then extracts features of the workpiece angle association features in high-dimensional space for the images of multiple angles of the workpiece to be identified, and finally passes the workpiece angle association features through a classifier to realize the judgment of the workpiece type. In this way, the accuracy of workpiece identification can be improved, thereby improving the assembly accuracy.

[0053] As described above, the intelligent assembly system 100 for handicraft ornaments based on visual detection according to the embodiment of the present application can be implemented in various terminal devices, such as a server for intelligent assembly of handicraft ornaments based on visual detection, etc. In one example, the intelligent assembly system 100 for handicraft ornaments based on visual detection according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the intelligent assembly system 100 for handicraft ornaments based on visual detection can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the intelligent assembly system 100 for handicraft ornaments based on visual detection can also be one of the many hardware modules of the terminal device.

[0054] Alternatively, in another example, the intelligent assembly system 100 for handicraft ornaments based on visual detection and the terminal device may also be separate devices, and the intelligent assembly system 100 for handicraft ornaments based on visual detection may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0055] Figure 5 FIG. 1 is a flow chart of a method for intelligently assembling handicraft ornaments based on visual inspection according to an embodiment of the present application. Figure 5As shown, in the intelligent assembly method of handicraft ornaments based on visual inspection, the method includes: S110, acquiring images of the workpiece to be identified taken at different angles; S120, performing feature encoding on the images of the workpiece to be identified taken at different angles to obtain a multi-scale workpiece angle association feature map; S130, performing channel feature enhancement on the multi-scale workpiece angle association feature map to obtain a multi-scale workpiece angle association enhanced feature map; S140, judging the workpiece type label to which the workpiece to be identified belongs based on the multi-scale workpiece angle association enhanced feature map.

[0056] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned method for intelligent assembly of handicraft ornaments based on visual detection have been described in the above reference. Figures 1 to 4 The invention has been introduced in detail in the description of the intelligent assembly system for handicraft ornaments based on visual inspection, and therefore, its repeated description will be omitted.

[0057] In summary, the intelligent assembly method of handicraft ornaments based on visual inspection according to the embodiment of the present application is explained, which adopts intelligent recognition technology based on machine vision, firstly obtains images of multiple angles of the workpiece to be identified, then extracts features of the workpiece angle association features in high-dimensional space for the images of multiple angles of the workpiece to be identified, and finally passes the workpiece angle association features through a classifier to realize the judgment of the workpiece type. In this way, the accuracy of workpiece identification can be improved, thereby improving the assembly accuracy.

[0058] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0059] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0060] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent assembly system for handicraft ornaments based on visual inspection, characterized in that: include: A multi-angle workpiece image data acquisition module is used to obtain images of the workpiece to be identified taken at different angles; A multi-angle workpiece image data encoding module is used to perform feature encoding on the workpiece images to be identified taken at different angles to obtain a multi-scale workpiece angle correlation feature map; A multi-angle workpiece correlation feature enhancement module is used to perform channel feature enhancement on the multi-scale workpiece angle correlation feature map to obtain a multi-scale workpiece angle correlation enhanced feature map; The workpiece type label result generation module is used to determine the workpiece type label of the workpiece to be identified based on the multi-scale workpiece angle association enhancement feature map.

2. The intelligent assembly system for handicraft ornaments based on visual inspection according to claim 1 is characterized in that: The multi-angle workpiece image data encoding module comprises: A target recognition workpiece image feature extraction unit is used to pass the images of the workpiece to be recognized taken at different angles through a workpiece feature encoder to obtain a plurality of target recognition workpiece feature matrices; A multi-angle target recognition workpiece feature integration unit is used to aggregate the multiple target recognition workpiece feature matrices along the channel dimension into a target recognition workpiece three-dimensional feature tensor; The multi-scale workpiece angle correlation feature generation unit is used to perform multi-scale feature extraction on the target recognition workpiece three-dimensional feature tensor to obtain the multi-scale workpiece angle correlation feature map.

3. The intelligent assembly system for handicraft ornaments based on visual inspection according to claim 2 is characterized in that: The multi-scale workpiece angle association feature generation unit comprises: A first-scale workpiece angle-associated feature generating subunit, configured to pass the target recognition workpiece three-dimensional feature tensor through a first three-dimensional workpiece angle-associated feature encoder to obtain a first-scale workpiece angle-associated feature map; A second-scale workpiece angle-associated feature generating subunit, configured to pass the target recognition workpiece three-dimensional feature tensor through a second three-dimensional workpiece angle-associated feature encoder to obtain a second-scale workpiece angle-associated feature map; The multi-scale feature fusion subunit is used to fuse the first-scale workpiece angle correlation feature map and the second-scale workpiece angle correlation feature map to obtain the multi-scale workpiece angle correlation feature map.

4. The intelligent assembly system for handicraft ornaments based on visual inspection according to claim 3 is characterized in that: The multi-angle workpiece association feature enhancement module is used to: pass the multi-scale workpiece angle association feature map through a feature enhancer based on a channel attention mechanism to obtain the multi-scale workpiece angle association enhanced feature map.

5. The intelligent assembly system for handicraft ornaments based on visual inspection according to claim 4 is characterized in that: The workpiece feature encoder is a convolutional neural network model including a deep and shallow feature fusion module.

6. The intelligent assembly system for handicraft ornaments based on visual inspection according to claim 5 is characterized in that: The first three-dimensional workpiece angle-associated feature encoder is a convolutional neural network model using a three-dimensional convolution kernel of a first scale, and the second three-dimensional workpiece angle-associated feature encoder is a convolutional neural network model using a three-dimensional convolution kernel of a second scale, and the first scale is different from the second scale.

7. The intelligent assembly system for handicraft ornaments based on visual inspection according to claim 6 is characterized in that: The feature enhancer based on the channel attention mechanism is a convolutional neural network model using the channel attention mechanism.

8. The intelligent assembly system for handicraft ornaments based on visual inspection according to claim 7 is characterized in that: The artifact type label result generation module includes: A feature map dimension reduction unit, used for pooling each feature matrix of the multi-scale workpiece angle correlation enhancement feature map along the channel dimension to obtain a multi-scale workpiece angle correlation enhancement feature vector; A workpiece feature optimization unit, used for performing a backward learning-type attention optimization based on a feature analysis network on the multi-scale workpiece angle association reinforcement feature vector to obtain an optimized multi-scale workpiece angle association reinforcement feature vector; The workpiece feature classification unit is used to pass the optimized multi-scale workpiece angle association enhanced feature vector through a workpiece type classifier to obtain a classification result, and the classification result is a workpiece type label of the workpiece to be identified.

9. The intelligent assembly system for handicraft ornaments based on visual inspection according to claim 8 is characterized in that: The workpiece feature optimization unit is used to: Performing probabilistic activation based on a Sigmoid function on the multi-scale workpiece angle association enhancement feature vector to obtain a probabilistic multi-scale workpiece angle association enhancement feature vector; Inputting the multi-scale workpiece angle association enhanced feature vector into the workpiece type classifier to obtain an initial class label probability value; Calculating the absolute value of the difference between the feature value of each position in the probabilistic multi-scale workpiece angle association enhanced feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error metric vector; Using the backward learning fine-grained error metric vector as a key vector and the multi-scale artifact angle association enhancement feature vector as a query vector and a value vector, the key vector, the query vector and the value vector are input into a converter structure to obtain a backward learning class attention optimized multi-scale artifact angle association enhancement feature vector; Calculating a ratio between a sum of the backward machine learning fine-grained error metric vector and a length of the backward machine learning fine-grained error metric vector, and calculating a natural exponential function value raised to a power of the ratio as an optimization scaling factor; The product of the optimized scaling factor and the backward learning-like attention-optimized multi-scale artifact angle-related enhanced feature vector is calculated to obtain the optimized multi-scale artifact angle-related enhanced feature vector.

10. A method for intelligent assembly of handicraft ornaments based on visual inspection, characterized in that: include: Acquire images of workpieces to be identified taken at different angles; Performing feature encoding on the images of the workpiece to be identified taken at different angles to obtain a multi-scale workpiece angle-related feature map; Performing channel feature enhancement on the multi-scale workpiece angle correlation feature map to obtain a multi-scale workpiece angle correlation enhanced feature map; Based on the multi-scale workpiece angle association enhanced feature map, the workpiece type label to which the workpiece to be identified belongs is determined.

Images