A method for identifying the assembly state of in-process products based on template contour learning

Abstract

The present invention relates to a method for identifying the assembly state of in-process products based on template contour learning. First, an assembly state bounding box of the product digital model is established and expanded after obtaining the virtual viewpoints on the bounding box; the projection of the assembly state of the digital model is completed by using the digital model and the virtual viewpoints, and a feature descriptor is constructed; the image of the in-process product captured is optimized to obtain a high-quality in-process product image with a complete outer contour, clear features, and appropriate exposure; based on the obtained digital model feature descriptor and the in-process product image, a data set for convolutional neural network training and testing is established; a convolutional neural network is constructed, and the network is trained by using the digital model feature descriptor training data set. This method is realized by analyzing the actual captured image of the in-process product without identification; at the same time, the computer automatically identifies the assembly state of the in-process product during the assembly process, which can effectively improve the efficiency of product assembly operations.

Description

Technical Field

[0001] The present invention belongs to the field of computer-aided manufacturing, and particularly relates to a method for identifying the assembly state of in-process products based on template contour learning. Background Art

[0002] Manual assembly, with its high flexibility and strong adaptability, is an indispensable link in the assembly scenarios of highly customized and small-batch mechanical products such as airplanes and rockets. With the development of intelligent manufacturing, the assembly process difficulty of complex products has increased, and enterprises have further improved their requirements for the quality and efficiency of manual assembly. Therefore, the technology for identifying the assembly state of products without identification has become a research hotspot in the field of product assembly. By analyzing the assembly situation based on the real-time assembly scenario, pushing the assembly assistance guidance information to the assembly personnel in a timely manner and carrying out manual operations, and adopting the augmented reality-assisted assembly (Augmented Assembly) mode of virtual-real fusion can significantly reduce the learning cost of manual understanding of the assembly progress and performing part installation operations. Currently, some model similarity analysis methods for different descriptors have emerged:

[0003] In the patent "A method for identifying the assembly state based on three-dimensional feature points (CN1131117411A, publication date: 20210713)", a method for identifying the assembly state based on three-dimensional feature points is proposed. The extraction of feature points is realized according to the relationship between the structural features and curvature of the assembly body. The point cloud registration of the digital model and the in-process product is realized with the mapping relationship of the feature points as the input, and the calculation of the point cloud similarity is carried out. The identification of the product assembly state is realized by analyzing the point cloud similarity, and the pose and assembly progress information of the in-process product are output. However, further research finds that this method is greatly affected by the occlusion relationship of the depth point cloud, and the recognition accuracy of the depth image taken from some specific perspectives for the assembly state is not high.

[0004] The literature (Mustafa W, Pugeault N, Buch A G, et al. Multi-View ObjectInstanceRecognition in an Industrial Context, 2017, 35(2): 271–292) combines machine vision algorithms and proposes to use an identification method without identification to identify objects in the assembly scenario. The position of the object is obtained by using a deep convolutional neural network for a single input color image, and a loss function is proposed to support the neural network framework to achieve real-time recognition. However, this method is based on the recognition of physical templates and requires a large amount of time for image shooting for network training, and is not suitable for the recognition of small-batch and highly customized mechanical products.

[0005] In the literature "Luo R C, Kuo C W. Intelligent Seven-doF Robot with Dynamic Obstacle Avoidance and 3-D Object Recognition for Industrial Cyber–Physical Systems in Manufacturing Automation[J]. Proceedings of the IEEE, 2016, 104(5):1102-1113.", the Point Cloud Library (PCL) is used to preprocess the three-dimensional point cloud obtained by the Kinect sensor. The alternative recognition hypothesis results are obtained according to the matching degree of the generated geometric descriptors, and then the second matching is performed according to the pose matching degree in the alternative hypothesis results. However, the judgment accuracy and speed of this method for complex assemblies need to be improved. Summary of the Invention

[0006] In order to solve the problem of too high learning cost for manual assembly progress and part installation operations in the production process of complex products, the present invention takes augmented reality-assisted assembly as the application background, establishes a method for identifying the assembly state of in-process products without identification based on template contour learning, generates a template contour feature descriptor for describing the assembly state by using the digital assembly model of the product, and trains the SqueezeNet convolutional neural network. On this basis, by comparing the template features of the digital model with the actual photographed photos, the assembly state of the in-process products in the augmented reality-assisted assembly process is finally identified.

[0007] The technical solution of the present invention is: a method for identifying the assembly state of in-process products based on template contour learning, including the following steps:

[0008] Step 1: Define the product to be assembled composed of n parts as: A = {part1, part2,..., part n}, by sorting out its product assembly process, an assembly sequence SE composed of n point cloud models representing its assembly states is established;

[0009]

[0010] Among them, ASi represents the assembly body point cloud model when the product is in the i-th assembly state, and parti represents the part to be installed in the i-th assembly state;

[0011] Step 2: Establish a circumscribed regular polyhedron for the digital models of different assembly states of the assembly body, and the set of vertices and face center points obtained is the initial virtual view point set in this assembly state;

[0012] Step 3: For each assembled state point cloud, obtain its template feature descriptor through view projection;

[0013] Step 4: To highlight the external shape features of the product in assembly, it is necessary to optimize the collected images of the product in assembly;

[0014] Step 5: Use a convolutional neural network to realize the recognition of the assembled state; the digital model descriptor obtained in Step 2 is used as the training set; the image of the product in assembly obtained in Step 3 is used as the test set, where the input layer is used to receive image input, and after the input layer, a convolutional layer is connected in sequence, then an activation function layer, and then a max pooling layer and other subsequent network levels are connected; the output layer is used to output the recognition result information, that is, the classification layer;

[0015] Step 6: Perform forward propagation and backpropagation network training on the established neural network. Among them, the output value obtained by forward propagation also needs to calculate the error with the target value. If the calculation result does not meet the requirements, it enters the backpropagation process. If it meets the requirements, the result is directly output. It is necessary to set the loss function of backpropagation;

[0016] Step 7: Use the trained convolutional neural network to recognize the test data set of the images of the product in assembly and output the corresponding recognition results.

[0017] A further technical solution of the present invention is that the following sub-steps are included in Step 2:

[0018] Step 2.1: Establish a circumscribed regular polyhedron for each assembled state of the digital model, so that the contour vertices of the digital model are approximately located on the faces of the circumscribed regular polyhedron;

[0019] Step 2.2: Virtual view point expansion: Let CV i be the i-th virtual view point, and CV_new ij be the j-th virtual view point expanded on the basis of CV i ; First, with CV i as the origin, obtain CV_new ij under this virtual view point by adding offset amounts (Δx, Δy, Δz) with different weights; Then, iterate n times to expand each CV i to n + 1 virtual view points; Finally, traverse all view points in the initial virtual view point set to obtain an expanded virtual view point set for constructing the digital model descriptor.

[0020] A further technical solution of the present invention is that the following sub-steps are included in Step 3:

[0021] Step 3.1: Rotate the projection plane to the xoy plane in the three-dimensional coordinate system and then perform projection; o_model is the centroid of the assembly model, with (x o, y o , z o ) represents the centroid coordinates; CVi is the i-th virtual viewpoint, represented by (x i , y i , z i ) represents the virtual viewpoint coordinates. Project the original point cloud along the direction L = (x i - x0, y i - y0, z i - z0) onto the xoy plane, where (x i , y i , z i ) represents the three-dimensional coordinates of the i-th virtual viewpoint, and (x0, y0, z0) represents the three-dimensional coordinates of the centroid point of the assembled point cloud; the three-dimensional point cloud coordinates after projection are expressed as:

[0022] AS' i = AS i * R j

[0023] Among them, is the rotation matrix for rotating the point cloud model to the required position under the i-th virtual viewpoint, and can be expressed as:

[0024]

[0025] Step S3.2: Construct a two-dimensional histogram and initialize it so that each value is 0, where m represents the dimension;

[0026] Step S3.3: Count the positions where each point in AS' i falls within , and count its maximum value:

[0027]

[0028] Among them, id x , id y = 1, 2,..., m is the position index where the point (x, y, z) falls within , and is calculated in the following way:

[0029]

[0030] Among them, [·] is the ceiling function;

[0031] Step S3.4: Repeat Step S3.3 until all points in AS' i are traversed, and use as the projection contour descriptor of the assembled state point cloud in the direction of the virtual viewpoint fp i in AS j ;

[0032] Step S3.5: Repeat steps S3.1 - S3.4 until states AS1 - AS are obtained. n For all descriptors k under the virtual viewpoints fp1 - fp

[0033] A further technical solution of the present invention is that step 4 includes the following sub - steps:

[0034] Step 4.1: Use the linear spectral clustering image segmentation algorithm to highlight the in - process product image and move it to the center position of the image.

[0035] Step 4.2: Use the asymptotically non - local mean denoising algorithm to reduce the noise in the image.

[0036] Step 4.3: Use the irradiance algorithm to fuse underexposed and overexposed images to obtain an optimized image with appropriate exposure.

[0037] A further technical solution of the present invention is that step 5 includes the following sub - steps:

[0038] Step 5.1: The main function of the network convolutional layer is to take the output of the previous layer as the input, perform convolution operation, and after obtaining the calculation result, output the result of the convolution operation to the next layer. These results are also the input of the next layer; the size g×g of the convolution kernel needs to be set much smaller than the size of the input matrix of the convolutional layer.

[0039] Step 5.2: Establish a non - linear activation function layer to enable the convolutional neural network to obtain the ability to complete non - linear tasks.

[0040] Step 5.3: Network pooling layer, using max - pooling operation to reduce the size of the model, improve the calculation speed of the convolutional neural network, and maintain the feature robustness of the input image.

[0041] Step 5.4: Classification layer, select the Softmax function layer for classification.

[0042] Invention effects

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The product assembly state recognition method proposed by the present invention can be realized by analyzing the actual - shot image of the in - process product without identification, with fewer operation constraints compared to the current identification methods with identification.

[0045] 2. The training data for learning the product assembly state proposed by the present invention is automatically obtained from the 3D digital model, which is easier to obtain compared to the actual - shot images in the physical environment.

[0046] 3. The method of the present invention addresses the problem of identifying the assembly status of in-process products in the augmented reality-assisted assembly process. It proposes a method for identifying the assembly status of in-process products without identification based on deep learning, realizing the automatic identification of the assembly status of in-process products by a computer during the assembly process, and effectively improving the efficiency of product assembly operations. Description of the Drawings

[0047] Figure 1 It is the overall flowchart for identifying the assembly status of in-process products in the present invention.

[0048] Figure 2 It is an example of an in-process product for status identification in the specific implementation manner of the method of the present invention.

[0049] Figure 3 It is the point cloud of each product assembly status in the specific implementation manner of the method of the present invention.

[0050] Figure 4 It is an example of the virtual viewpoint and its coordinates of the product assembly status point cloud in the specific implementation manner of the method of the present invention.

[0051] Figure 5 It is an example of the projection contour descriptor of the product assembly status in the specific implementation manner of the method of the present invention.

[0052] Figure 6 It is an example of some data in the test set in the specific implementation manner of the method of the present invention.

[0053] Figure 7 It is the SqueezeNet neural network structure constructed in the specific implementation manner of the method of the present invention.

[0054] Figure 8 It is the recognition result of the actual shooting images of each assembly status in the specific implementation manner of the method of the present invention. Specific Implementation Manner

[0055] Refer to Figures 1 - 8 , the main content of the solution of the present invention is to establish a bounding box for the assembly status of the product digital model, appropriately expand it after obtaining the virtual viewpoints on the bounding box; finally, use the digital model and virtual viewpoints to complete the projection of the assembly status of the digital model, construct the corresponding feature descriptors; optimize the images of the in-process products captured, and obtain high-quality in-process product images with complete outer contours, clear features, and appropriate exposure; based on the obtained digital model feature descriptors and in-process product images, establish a data set that can be used for the training and testing of the convolutional neural network; construct a convolutional neural network, and use the data set of the digital model feature descriptors to train the network, and finally complete the identification of the assembly status of the in-process products.

[0056] The technical solution of the present invention includes the following steps:

[0057] Step S1: For the product A to be assembled composed of n parts A = {part1, part2, …, part n}, by sorting out its product assembly process, an assembly sequence SE composed of n point cloud models representing its assembly states is established:

[0058]

[0059] where ASi represents the assembly body point cloud model when the product is in the i-th assembly state, and parti represents the part to be installed in the i-th assembly state.

[0060] Step S2: In this embodiment, a virtual view point set is obtained by using an inscribed regular icosahedron, which specifically includes the following steps:

[0061] Step S2.1: Obtain the original view point set CV from the inscribed regular icosahedron, including 12 vertices and 20 face center points:

[0062] CV = {fp1, fp2, …, fp 12 , fp 13 , …, fp 32}

[0063] where the first 12 points are the vertices of the regular icosahedron, and their coordinate values are expressed as:

[0064] fp i = {(±0.526r, 0, ±0.831r), (0, ±0.831r, ±0.526r), (±0.831r, ±0.526r, 0)}

[0065] where r is the farthest distance from the centroid of the assembly body to the outer contour, that is, the length of the circumradius of the bounding box of the point cloud in the n-th assembly state. fp 13 -fp 32 represents the coordinate values of the centroid points of the equilateral triangles on each face of the regular icosahedron, which can be expressed as follows:

[0066]

[0067] where fp a , fp b , fp c represent an equilateral triangle, and fp i represents the coordinate value of the centroid point.

[0068] Step S2.2: Expand each original view point, and obtain the expanded view point coordinates by adding offsets with different weights to each original view point:

[0069] fp = (x i+Δx,y i +Δy,z i +Δz)

[0070] where (x i , y i , z i ) represents the three-dimensional coordinates of the i-th original viewpoint;

[0071] Step S2.3: Repeat Step S2.2 five times for each original viewpoint to obtain the augmented viewpoint set CV_new;

[0072] Step S2.4: Finally, obtain the virtual viewpoint set FP = CV ∪ CV_new.

[0073] Step S3: For each point cloud of the assembly state, obtain its template feature descriptor through view projection. Specifically, it includes the following steps:

[0074] Step S3.1: Project the original point cloud along the direction L = (x i - x0, y i - y0, z i - z0) onto the xoy plane, where (x i , y i , z i ) represents the three-dimensional coordinates of the i-th virtual viewpoint, and (x0, y0, z0) represents the three-dimensional coordinates of the centroid point of the assembly point cloud. The three-dimensional point cloud coordinates after projection are expressed as:

[0075] AS' i = AS i * R j

[0076] where is the rotation matrix for rotating the point cloud model to the required position under the i-th virtual viewpoint, which can be expressed as:

[0077]

[0078] Step S3.2: Construct a 227-dimensional two-dimensional histogram and initialize it so that each value in it is 0;

[0079] Step S3.3: Count the positions where each point in AS' i falls within and count its maximum value:

[0080]

[0081] where id x , id y = 1, 2, …, 227 is the position where the point (x, y, z) falls within The position index in it is calculated as follows:

[0082]

[0083] where [·] is the ceiling function.

[0084] Step S3.4: Repeat Step S3.3 until all points in AS' i are traversed, and is used as the projection contour descriptor of the assembled state point cloud in AS i in the virtual view point fp j direction;

[0085] Step S3.5: Repeat Steps S3.1 - S3.4 until all descriptors in the virtual view points fp1 - fp n under 192 are obtained for the states AS1 - AS

[0086] Step S4: Construct the model framework of the SqueezeNet convolutional neural network, including the input layer, convolutional layer, activation function, and pooling layer. Specifically, it includes the following steps:

[0087] Step S4.1: Construct the training set. Use all descriptors of the product to be assembled as the training set, divide them into n categories according to the assembled state, and mark them;

[0088] Step S4.2: Input the training set into the neural network, perform network convolution operations and non - linear activation operations, and calculate the loss function between the output value and the target value:

[0089]

[0090] where p c is the result probability corresponding to the current network parameters, n is the number of categories in this classification task, and y c is a variable indicating whether the classification result is correct, 1 for correct and 0 for incorrect.

[0091] Step S4.3: Repeat the loop of forward propagation and backward propagation until the error L p calculated by the loss function is less than the set error threshold, and fix the parameters of the neural network.

[0092] Step S5: Establish a test set for the assembled state of the product in - process to test the network recognition effect, and verify the network recognition performance. Specifically, it includes the following steps:

[0093] Step S5.1: In the physical environment, obtain the images of each actual assembled state of the product in - process to form the test set TestData:

[0094] TestData = {IM i,j |AS i} (0 - 1)

[0095] where: IM i,j —— the j-th actual shot image of the i-th assembly state AS i .

[0096] Step S5.2: For each actual shot image, use the linear spectral clustering algorithm to segment the image so that the target product is located at the center of the image;

[0097] Step S5.3: For each actual shot image, use the linear spectral clustering algorithm to segment the image so that the target product is located at the center of the image;

[0098] Step S5.4: If the pixel scale of the image is not 227 * 227, use the bilinear interpolation algorithm to enlarge or reduce it to a 227 * 227 two-dimensional matrix as the input of the SqueezeNet network.

[0099] Step S6: Input the assembly state picture into the trained SqueezeNet network for classification, and use the category with the highest probability in the output layer as the classification result of the assembly state to complete the recognition of the actual assembly state in the physical environment.

[0100] The following further explains the technical solution with reference to the attached drawings.

[0101] Figure 1 The following is a schematic diagram of the recognition process of the assembly state of the product in the process based on template contour feature learning provided by the embodiment of the present invention, which includes the following steps:

[0102] Step S1: For the aircraft cabin door model A = {part1, part2,..., part8} composed of 8 parts as shown in Figure 2 , by sorting out its product assembly process, establish an assembly sequence SE composed of 8 point cloud models representing its assembly states:

[0103] SE = {AS1, AS2,..., AS8}

[0104] where, AS i represents the assembly body point cloud model when the product is in the i-th assembly state, and the assembly body point cloud models corresponding to AS1 - AS8 are as shown in Figure 3 .

[0105] Step S2: Use the circumscribed regular icosahedron to obtain the virtual view point set, which specifically includes the following steps:

[0106] Step S2.1: Obtain the original viewpoint set CV from the circumscribed regular icosahedron, including 12 vertices and 20 face centers:

[0107] CV = {fp1, fp2, …, fp 12 , fp 13 , …, fp 32}

[0108] Among them, the first 12 points are the vertices of the regular icosahedron, and their coordinate values are expressed as:

[0109] fp i = {(±0.526r, 0, ±0.831r), (0, ±0.831r, ±0.526r), (±0.831r, ±0.526r, 0)}

[0110] Among them, r = 190.45 is the maximum distance from the centroid of the assembled body to the outer contour, that is, the length of the circumradius of the bounding box of the nth assembled state point cloud. fp 13 -fp 32 represents the coordinate values of the centroid points of the equilateral triangles on each face of the regular icosahedron:

[0111]

[0112] Among them, fp a , fp b , fp c represent an equilateral triangle, and fp i represents the coordinate values of the centroid point.

[0113] Step S2.2: Expand each original viewpoint, and obtain the expanded viewpoint coordinates by adding offsets with different weights to each original viewpoint:

[0114] fp = (x i *Δx, y i *Δy, z i *Δz)

[0115] Among them, (x i , y i , z i ) represents the three-dimensional coordinates of the ith original viewpoint, and Δx, Δy, and Δz represent random values from 1 to 1.1;

[0116] Step S2.3: Repeat Step S3.2 for each original viewpoint 5 times to obtain the expanded viewpoint set CV_new;

[0117] Step S2.4: Finally, obtain the virtual viewpoint set FP = CV ∪ CV_new, and the virtual viewpoint coordinates formed finally are as Figure 4 shown.

[0118] Step S3: For each assembled state point cloud, obtain its template feature descriptor through view projection. Specifically, it includes the following steps:

[0119] Step S3.1: Project the assembled point cloud along the direction L=(x j -x0, y j -y0, z j -z0) onto the xoy plane, where (x j , y j , z j ) represents the three-dimensional coordinates of the j-th virtual view point, and (x0, y0, z0) represents the three-dimensional coordinates of the centroid point of the assembled point cloud. The three-dimensional point cloud coordinates after projection are expressed as:

[0120] AS' i = AS i * R j

[0121] where R j is the rotation matrix that rotates the point cloud model to the required position under the j-th virtual view point, and can be expressed as:

[0122]

[0123] Step S3.2: Construct a 227-dimensional two-dimensional histogram and initialize it so that each value in it is 0;

[0124] Step S3.3: Count the positions where each point in AS' i falls within , and count its maximum value:

[0125]

[0126] where id x , id y = 1, 2,..., 227 is the position index where a point (x, y, z) in AS' i falls within , and is calculated in the following way:

[0127]

[0128] where, [·] is the ceiling function.

[0129] Step S3.4: Repeat Step S3.3 until all points in AS' i are traversed, and take as the assembled state point cloud in AS i under the virtual view point fp jProjection contour descriptor in the direction;

[0130] Step S3.5: Repeat steps S3.1 - S3.4 until all descriptors of states AS1 - AS8 at virtual viewpoints fp1 - fp 192 are obtained. The finally formed partial descriptor data is as Figure 5 shown.

[0131] Step S4: Construct the model framework of the SqueezeNet convolutional neural network, including an input layer, convolutional layers, activation functions, and pooling layers, with a total of 68 layers. The network structure is as Figure 7 shown. The middle part is a continuous connection of multiple Fire modules, and a pooling layer needs to be connected after every two Fire modules.

[0132] Specifically, it includes the following steps:

[0133] Step S4.1: Construct a training set. Use all descriptors of the product to be assembled as the training set, divide them into n categories according to the assembly state, and mark them;

[0134] Step S4.2: Input the training set into the neural network, perform network convolution operations and non - linear activation operations, and calculate the loss function between the output value and the target value:

[0135]

[0136] where p c is the result probability corresponding to the current network parameters, n is the number of categories in this classification task, and y c is a variable indicating whether the classification result is correct or not, 1 for correct and 0 for incorrect.

[0137] Step S4.3: Repeat the loop of forward propagation and backward propagation. After 1000 iterations, fix the parameters of the neural network.

[0138] Step S5: Establish an assembly state test set for testing the network recognition effect for product A and verify the network recognition performance. Specifically, it includes the following steps:

[0139] Step S5.1: In the physical environment, obtain the images of the in - process product A in each actual assembly state to form the test set TestData:

[0140] TestData = {IM i,j |AS i} (0 - 2)

[0141] where: IM i,j —— The j - th actual - shot image of the i - th assembly state AS i AsFigure 5 as shown

[0142] Step S5.2: For each real-shot image, use the linear spectral clustering algorithm to segment the image so that the target product is located at the center of the image;

[0143] Step S5.3: For each real-shot image, use the linear spectral clustering algorithm to segment the image so that the target product is located at the center of the image;

[0144] Step S5.4: If the pixel scale of the image is not 227*227, use the bilinear interpolation algorithm to enlarge or reduce it to a 227*227 two-dimensional matrix as the input of the SqueezeNet network. An example of the assembled state image after the above processing is as Figure 6 shown

[0145] Step S6: Input the assembled state image into the trained SqueezeNet network for classification, and use the category with the highest probability in the output layer as the classification result of the assembled state, thus completing the recognition of the actual assembled state in the physical environment. The product state recognition results and accuracy rates for each real-shot image are as Figure 8 shown

[0146] This embodiment shows that the product assembly state recognition method proposed by the present invention can be used to classify the product assembly states of real-shot images in the physical environment and can achieve good results.

[0147] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for identifying the assembly state of in-process products based on template contour learning, characterized in that It includes the following steps: Step 1: Define the product to be assembled consisting of n parts as: A = {part1, part2, …, part n}, by sorting out its product assembly process, establish an assembly sequence SE consisting of n point cloud models representing its assembly states; Where, ASi represents the assembly point cloud model when the product is in the i-th assembly state, and parti represents the part to be installed in the i-th assembly state; Step 2: Establish a circumscribed regular polyhedron for the digital models of different assembly states of the assembly, and the set of vertices and face center points obtained is the initial virtual view point set in this assembly state; Step 3: For the point cloud of each assembly state, obtain its template feature descriptor through view projection; it includes the following sub-steps: Step 3.1: Rotate the projection plane to the xoy plane in the three-dimensional coordinate system and then project; o_model is the centroid of the assembly model, represented by (x o , y o , z o ); CVi is the i-th virtual viewpoint, represented by (x i , y i , z i ); Project the original point cloud onto the xoy plane along the direction L = (x i - x0, y i - y0, z i - z0), where (x i , y i , z i ) represents the three-dimensional coordinates of the i-th virtual viewpoint, and (x0, y0, z0) represents the three-dimensional coordinates of the centroid point of the assembly point cloud; The coordinates of the projected three-dimensional point cloud are expressed as: AS' i = AS i * R j Among them, is the rotation matrix for rotating the point cloud model to the required position under the i-th virtual viewpoint, which can be expressed as: Step S3.2: Construct a two-dimensional histogram and initialize it so that each value in it is 0, where m represents the dimension; Step S3.3: Count AS' i the position where each point in falls, and count its maximum value: where id x , id y = 1, 2, …, m is the position index where the point (x, y, z) falls within , and is calculated as follows: Where, [·] is the ceiling function; Step S3.4: Repeat Step S3.3 until all points in AS' are traversed, and use as the projection contour descriptor of the assembled state point cloud in AS i in the virtual view point fp j direction. i All points in are used as the projection contour descriptor of the assembled state point cloud in AS i in the virtual view point fp j direction. Step S3.5: Repeat steps S3.1 - S3.4 until states AS1 - AS are obtained n at virtual viewpoints fp1 - fp k all descriptors Step 4: To highlight the external shape features of the product in assembly, it is necessary to optimize the collected image of the product in assembly; Step 5: Use a convolutional neural network to realize the recognition of the assembly state; the digital model descriptor obtained in Step 2 is used as the training set; the image of the product in assembly obtained in Step 3 is used as the test set, where the input layer is used to receive image input, and after the input layer, a convolutional layer is connected in sequence, then an activation function layer, and then a max pooling layer and other subsequent network levels are connected; the output layer is used to output the recognition result information, that is, the classification layer; Step 6: Perform forward propagation and backpropagation network training on the established neural network. Among them, the output value obtained by forward propagation also needs to calculate the error with the target value. If the calculation result does not meet the requirements, it enters the backpropagation process. If it meets the requirements, the result is directly output. It is necessary to set the loss function of backpropagation; Step 7: Use the trained convolutional neural network to recognize the test data set of the image of the product in assembly and output the corresponding recognition result.

2. The method for identifying the assembly state of in-process products based on template contour learning according to claim 1, wherein, The following sub-steps are included in Step 2: Step 2.1: Establish a circumscribed regular polyhedron for each assembly state of the digital model, so that the contour vertices of the digital model are approximately located on the faces of the circumscribed regular polyhedron; Step 2.2: Virtual viewpoint expansion: Let CV i be the i-th virtual viewpoint, and CV_new ij be the j-th virtual viewpoint obtained by expanding based on CV i ; First, taking CV i as the origin, CV_new ij under this virtual viewpoint is obtained by adding offsets (Δx, Δy, Δz) with different weights; Then, each CV i is expanded to n + 1 virtual viewpoints by iterating n times; Finally, all viewpoints in the initial virtual viewpoint set are traversed to obtain an expanded virtual viewpoint set for constructing the digital model descriptor.

3. The method for identifying the assembly state of products in production as claimed in claim 1, characterized in that, The following sub-steps are included in Step 4: Step 4.1: Use the linear spectral clustering image segmentation algorithm to highlight and move the image of the product in assembly to the center position of the image; Step 4.2: Use the asymptotic non-local mean denoising algorithm to reduce the noise in the image; Step 4.3: Use the irradiance algorithm to fuse the underexposed and overexposed images to obtain an optimized image with appropriate exposure.

4. The method for identifying the assembly state of a product in production according to claim 1, based on template contour learning, is characterized in that The following sub-steps are included in Step 5: Step 5.1 The main function of the network convolutional layer is to use the output of the previous layer as the input, perform convolutional operation, and after obtaining the calculation result, output the result of the convolutional operation to the next layer. These results are also the input of the next layer at the same time; the size g×g of the convolutional kernel needs to be set to be much smaller than the size of the input matrix of the convolutional layer; Step 5.2 Establish a non-linear activation function layer to enable the convolutional neural network to obtain the ability to complete non-linear tasks; Step 5.3 Network pooling layer, use max pooling operation to reduce the size of the model, improve the calculation speed of the convolutional neural network at the same time, and maintain the feature robustness of the input image; Step 5.4 Classification layer, select the Softmax function layer for classification.

Images