Method for mechanical assembly variation detection and assembly step identification based on difference images

Abstract

The present application relates to a kind of mechanical assembly body change detection and assembly step identification method based on difference image, including training phase and assembly monitoring phase;Training phase is trained to target detection network by the synthetic depth image of each assembly step before and after, each angle in assembly process, obtain component class identification network and assembly step identification network;Assembly monitoring phase obtains the physical depth image of each assembly step before and after in assembly process, and the component of current assembly step assembly and the assembly step where it is by component class identification network and assembly step identification network are identified.The present application extracts the label of the coordinate of change area to make target detection data set, without manually drawing target frame again, can automatically generate training set.Change component in assembly process is rendered and fused with the image of next time, increase the feature information of change component, improve the accuracy of assembly step identification.

Description

TECHNICAL FIELD

[0001] The application relates to a mechanical assembly body change detection and assembly step identification method based on a difference image and belongs to the technical field of intelligent manufacturing. BACKGROUND

[0002] Assembly is an important production process of product manufacturing, which refers to the process of assembling and connecting mechanical parts according to design requirements. With the continuous change of product types, the difficulty of assembly is also increasing. In the assembly process of complex mechanical products, errors (such as assembly sequence errors, missing parts, and misassembly) are prone to occur in the assembly process of small parts, which will directly affect the assembly efficiency and service life of mechanical products. Mechanical assembly body change detection can accurately detect the changes of assembly parts from the mechanical assembly body image, and can timely find errors in the assembly process, thereby reducing the unqualified rate of products and shortening the production cycle of products, which has important research significance for realizing intelligent detection of mechanical assembly bodies.

[0003] Assembly step identification can effectively monitor changed parts, identify the current assembly stage, push the next step of assembly parts, and improve assembly efficiency. SUMMARY

[0004] In order to overcome the above problems, the application provides a mechanical assembly body change detection and assembly step identification method based on a difference image, which extracts the coordinates of the change area to make labels of the target detection data set, without manually drawing the target frame, and can automatically generate a training set. The changed parts in the assembly process are rendered and fused with the image at the next time, increasing the feature information of the changed parts and improving the accuracy of assembly step identification.

[0005] The technical scheme of the application is as follows:

[0006] The mechanical assembly body change detection and assembly step identification method based on a difference image comprises a training stage and an assembly monitoring stage.

[0007] The training stage trains a target detection network through synthetic depth images before and after each assembly step and at each angle in the assembly process, to obtain a part class recognition network and an assembly step recognition network.

[0008] The assembly monitoring stage obtains physical depth images before and after each assembly step in the assembly process, and identifies the assembly parts and the assembly step through the part class recognition network and the assembly step recognition network.

[0009] Further, the training stage comprises:

[0010] Assembling parts;

[0011] Obtain two synthesized depth images P1 and P2 of the same perspective, wherein P1 is a pre-assembly assembly body depth image, and P2 is a post-assembly assembly body depth image;

[0012] Obtain a change image 1 by a difference method from the synthesized depth image P1 and the synthesized depth image P2, and determine a change region of the change image 1;

[0013] Render the change region of the change image 1 to obtain a rendered image 1;

[0014] Extract coordinates of the change region in the rendered image 1;

[0015] Fuse the change region in the rendered image 1 with a change region of the synthesized depth image P2 in pixel value to obtain a fused image 1;

[0016] Make a part category recognition dataset label and an assembly step recognition dataset label;

[0017] Store the change image 1 and the corresponding part category recognition dataset label in the part category recognition dataset;

[0018] Store the fused image 1 and the corresponding assembly step recognition dataset label in the assembly step recognition dataset;

[0019] Rotate the assembly body by a degree, obtain synthesized depth images of different perspectives, and determine whether the assembly body has been rotated for one revolution, if not, repeat the above steps until the assembly body is rotated for one revolution;

[0020] Assemble the next step part, and determine whether the assembly step is completed, if not, repeat the above steps until the assembly step is completed;

[0021] Train the part category recognition network and the assembly step recognition network;

[0022] Train the part category recognition network and the assembly step recognition network to a preset batch, save the optimal model of the part category recognition network and the optimal model of the assembly step recognition network;

[0023] Further, the assembly detection stage comprises:

[0024] Assemble a part;

[0025] Obtain two physical depth images Q1 and Q2 of the same perspective, wherein Q1 is a pre-assembly assembly body depth image, and Q2 is a post-assembly assembly body depth image, obtain a change image 2 by a difference method from the physical depth image Q1 and the physical depth image Q2, and determine a change region of the change image 2;

[0026] The category of the changed part represented by the changed region in the changed image 2 is identified through the part category identification network optimal model;

[0027] The changed region in the changed image 2 is rendered to obtain a rendered image 2;

[0028] The changed region in the rendered image 2 is fused with the changed region in the physical depth image Q2 in terms of pixel values to obtain a fused image 2;

[0029] The fused image 2 is input into the assembly step identification network optimal model to identify the assembly step in which the assembly part represented by the changed region in the changed image 2 is located.

[0030] Further, the changed image 1 is obtained through a difference method from the synthesized depth image P1 and the synthesized depth image P2, and the changed region of the changed image 1 is determined, specifically as follows:

[0031] The pixel values of the same coordinates of the synthesized depth image P1 and the synthesized depth image P2 are subtracted to obtain the changed image 1, and the region in the changed image 1 with a pixel value less than a threshold t is defined as the changed region in the changed image 1, and the changed region in the changed image 1 is the region of the changed part before and after assembly.

[0032] Further, the changed region of the changed image 1 is rendered to obtain a rendered image 1, specifically as follows:

[0033] The changed region in the changed image 1 is rendered according to the category of the changed part represented by the changed region in the changed image 1 to obtain the rendered image 1;

[0034] The region in the rendered image 1 with the same coordinates as the changed region in the changed image 1 is defined as the changed region of the rendered image 1.

[0035] Further, the coordinates of the changed region in the rendered image 1 are extracted, specifically as follows:

[0036] The boundary coordinates of the changed region of the rendered image 1 are extracted;

[0037] The minimum values of the horizontal and vertical coordinates of the changed region are defined as X min and Y min , and the minimum values of the horizontal and vertical coordinates of the target frame are X min and Y min ;

[0038] The maximum values of the horizontal and vertical coordinates of the changed region are defined as X max and Y max , and the maximum values of the horizontal and vertical coordinates of the target frame are X max and Y max ;

[0039] According to the coordinate points (X min ,Y min ) and (X max ,Y max ), draw the target frame;

[0040] Normalize the extracted coordinates:

[0041] The normalized center coordinates (x, y) of the target frame:

[0042]

[0043]

[0044] The normalized width w and height h of the target frame:

[0045]

[0046]

[0047] Where width and height are the width and height of the rendered image 1.

[0048] Further, make a parts category recognition dataset label and an assembly step recognition dataset label, specifically:

[0049] Save the category of the changed part represented by the changed area, the normalized center coordinates, and the normalized width and height of the target frame in the txt1 file as the label of a sample in the parts category recognition dataset;

[0050] Save the assembly step in which the changed part represented by the changed area in the changed image 1 is located, the normalized center coordinates, and the normalized width and height of the target frame in the txt2 file as the label of a sample in the assembly step recognition dataset.

[0051] Further, train the parts category recognition network and the assembly step recognition network, specifically:

[0052] Use the target detection network as the parts category recognition network and the assembly step recognition network;

[0053] Train the parts category recognition network through the parts category recognition dataset, which is used to identify the category of the changed part;

[0054] Train the assembly step recognition network through the assembly step recognition dataset, which is used to identify the assembly step in which the changed part is located.

[0055] Further, render the changed area in the changed image 2 to obtain a rendered image 2, specifically:

[0056] According to the category of the changed part represented by the changed region in the changed image 2, the rendering color is determined, and the changed region in the changed image 2 is rendered to obtain a rendered image 2; the color of each part category rendering is the same as that in the training stage;

[0057] The region in the rendered image 2 with the same coordinates as the changed region in the changed image 2 is defined as the changed region of the rendered image 2.

[0058] Preferably, the target detection network is yolov5.

[0059] The present application has the following beneficial effects:

[0060] 1. The present application extracts the coordinates of the changed region to make labels of the target detection data set, without manually drawing the target box, and can automatically generate the training set.

[0061] 2. The present application renders the changed parts in the assembly process and fuses them with the image at the next time, increases the feature information of the changed parts, and improves the accuracy of assembly step recognition.

[0062] 3. The present application can accurately determine the assembly stage of the parts when the same parts are assembled at different assembly steps. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 The training stage flowchart of the embodiment of the present application.

[0064] Figure 2 The assembly monitoring stage flowchart of the embodiment of the present application.

[0065] Figure 3 The target box schematic diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0066] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0067] Reference Figures 1-2 , the mechanical assembly body change detection and assembly step recognition method based on the difference image includes a training stage and an assembly monitoring stage;

[0068] The training stage trains the target detection network through the synthetic depth images before and after each assembly step and at each angle in the assembly process, to obtain a part category recognition network and an assembly step recognition network.

[0069] The assembly monitoring stage acquires physical depth images before and after each assembly step in the assembly process, and identifies the parts assembled in the current assembly step and the assembly step through the part type identification network and the assembly step identification network.

[0070] Embodiment one

[0071] Reference Figure 1 , a mechanical assembly change detection and assembly step identification method based on difference image, the training stage comprises:

[0072] Assembling parts;

[0073] Obtain two synthesized depth images P1 and P2 with the same perspective, wherein P1 is the depth image of the assembly before assembly, and P2 is the depth image of the assembly after assembly;

[0074] The change image 1 is obtained by the difference method from the synthesized depth image P1 and the synthesized depth image P2, and the change area of the change image 1 is determined;

[0075] Render the change area of the change image 1 to obtain the rendered image 1;

[0076] Extract the coordinates of the change area in the rendered image 1;

[0077] In the mechanical assembly process, the parts assembled in different assembly steps may be the same, and the current assembly step needs to be judged in combination with the features around the new assembly parts. In order to effectively improve the accuracy of the target detection network and accurately judge the assembly step of the changed parts, the change area in the rendered image 1 is fused with the change area of the synthesized depth image P2 to obtain the fusion image 1. The fusion image 1 not only retains the depth features of the changed parts, but also increases the feature information of the changed parts before and after assembly.

[0078] Make part type identification dataset labels and assembly step identification dataset labels;

[0079] Store the change image 1 and the corresponding part type identification dataset label in the part type identification dataset;

[0080] Store the fusion image 1 and the corresponding assembly step identification dataset label in the assembly step identification dataset;

[0081] Rotate the assembly by a degree to obtain synthesized depth images with different perspectives, and determine whether the assembly has been rotated one round, if not, repeat the above steps until the assembly is rotated one round;

[0082] Assemble the next step parts, and determine whether the assembly step is completed, if not, repeat the above steps until the assembly step is completed;

[0083] training the part category recognition network and the assembly step recognition network;

[0084] training the part category recognition network and the assembly step recognition network to a preset batch, saving the optimal model of the part category recognition network and the optimal model of the assembly step recognition network;

[0085] Embodiment Two

[0086] Reference Figure 2 , based on the difference image-based mechanical assembly body change detection and assembly step recognition method, on the basis of Embodiment One, the assembly detection stage comprises:

[0087] assembly parts;

[0088] obtaining two physical depth images Q1 and Q2 of the same perspective, Q1 being a pre-assembly depth image of the assembly body and Q2 being a post-assembly depth image of the assembly body, obtaining a change image 2 from the physical depth image Q1 and the physical depth image Q2 by a difference method, and determining a change region of the change image 2;

[0089] identifying the category of the changed part represented by the change region in the change image 2 through the optimal model of the part category recognition network;

[0090] rendering the change region in the change image 2 to obtain a rendered image 2;

[0091] fusing the change region in the rendered image 2 with the change region of the physical depth image Q2 in terms of pixel value to obtain a fused image 2;

[0092] inputting the fused image 2 into the optimal model of the assembly step recognition network to identify the assembly step in which the assembly part represented by the change region in the change image 2 is located.

[0093] Embodiment Three

[0094] based on the difference image-based mechanical assembly body change detection and assembly step recognition method, on the basis of Embodiment Two, obtaining a change image 1 from the synthetic depth image P1 and the synthetic depth image P2 by a difference method, and determining a change region of the change image 1, specifically:

[0095] obtaining the change image 1 by subtracting the pixel values of the same coordinates of the synthetic depth image P1 and the synthetic depth image P2, defining the region in which the pixel value is less than the threshold value t in the change image 1 as the change region in the change image 1, and the change region in the change image 1 being the region of the changed part before and after assembly.

[0096] Embodiment Four

[0097] The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images, building upon Example 3, renders the changed region of the changed image 1 to obtain rendered image 1, specifically as follows:

[0098] The rendering color is determined based on the category of the changed component represented by the changed area in the changed image 1, and the changed area in the changed image 1 is rendered to obtain the rendered image 1.

[0099] The region in rendered image 1 with the same coordinates as the changed region in the transformed image 1 is defined as the changed region of rendered image 1.

[0100] Example 5

[0101] The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images, building upon Example 4, extracts the coordinates of the changed regions in the rendered image 1, specifically as follows:

[0102] Extract the boundary coordinates of the changed region in the rendered image 1;

[0103] The minimum value of the x and y coordinates of the region of change is defined as X. min and Y min Let the minimum x and y coordinates of the target bounding box be X. min and Y min ;

[0104] The maximum values ​​of the x and y coordinates of the region of change are defined as X. max and Y max Let the maximum value of the x and y coordinates of the target bounding box be X. max and Y max ;

[0105] Based on coordinate point (X) min ,Y min ) and (X max ,Y max Draw the target bounding box;

[0106] Normalize the extracted coordinates:

[0107] The normalized center coordinates (x, y) of the bounding box:

[0108]

[0109]

[0110] Normalized width w and height h of the bounding box:

[0111]

[0112]

[0113] wherein width and height are the width and height of the rendered image 1.

[0114] Since it is time-consuming to manually draw the target box to make the dataset label, the application automatically draws the target box by using the rendered image 1, thereby improving the efficiency of making the dataset.

[0115] Embodiment six

[0116] Based on the mechanical assembly body change detection and assembly step recognition method based on the difference image, on the basis of embodiment five, the part category recognition dataset label and the assembly step recognition dataset label are made, specifically:

[0117] The category of the changed part represented by the change area, the normalized center coordinates and the normalized width and height of the target box are saved in the txt1 file as the label of one sample in the part category recognition dataset;

[0118] The assembly step in which the changed part represented by the change area in the change image 1 is located, the normalized center coordinates, and the normalized width and height of the target box are saved in the txt2 file as the label of one sample in the assembly step recognition dataset.

[0119] Embodiment seven

[0120] Based on the mechanical assembly body change detection and assembly step recognition method based on the difference image, on the basis of embodiment six, the part category recognition network and the assembly step recognition network are trained, specifically:

[0121] The target detection network is used as the part category recognition network and the assembly step recognition network;

[0122] The part category recognition network is trained through the part category recognition dataset, and is used for recognizing the category of the changed part;

[0123] The assembly step recognition network is trained through the assembly step recognition dataset, and is used for recognizing the assembly step in which the changed part is located.

[0124] Preferably, the target detection network is yolov5.

[0125] Embodiment eight

[0126] Based on the mechanical assembly body change detection and assembly step recognition method based on the difference image, on the basis of embodiment seven, the change area in the change image 2 is rendered to obtain a rendered image 2, specifically:

[0127] According to the category of the changed part represented by the changed region in the changed image 2, the rendering color is determined, the changed region in the changed image 2 is rendered, and a rendered image 2 is obtained; the rendering color of each part category is the same as that in the training stage;

[0128] The region in the rendered image 2 with the same coordinates as the changed region in the changed image 2 is defined as the changed region of the rendered image 2.

[0129] The above description is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure made by using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images, characterized in that, This includes the training phase and the assembly monitoring phase; The training phase uses synthesized depth images from before and after each assembly step and from various angles during the assembly process to train the target detection network, resulting in a component category recognition network and an assembly step recognition network. The assembly monitoring phase acquires physical depth images before and after each assembly step during the assembly process, and identifies the components being assembled in the current assembly step and the current assembly step through the component category recognition network and the assembly step recognition network. The training phase includes: Assemble parts; Acquire two composite depth images P1 and P2 from the same viewpoint, where P1 is the depth image of the assembly before assembly and P2 is the depth image of the assembly after assembly. A changed image 1 is obtained from the synthesized depth image P1 and the synthesized depth image P2 using a difference method, and the changed region of the changed image 1 is determined. Render the changed area of ​​the changed image 1 to obtain the rendered image 1; Extract the coordinates of the changing region in the rendered image 1; The pixel values ​​of the changed regions in the rendered image 1 and the changed regions in the synthesized depth image P2 are fused to obtain the fused image 1. Create dataset labels for component category identification and assembly step identification; Store the changed image 1 and its corresponding component category recognition dataset labels into the component category recognition dataset; Store the fused image 1 and its corresponding assembly step recognition dataset label into the assembly step recognition dataset; Rotate the assembly by 'a' degrees to obtain composite depth images from different perspectives, and determine whether the assembly has rotated one full circle. If not, repeat the above steps until the assembly has rotated one full circle. Assemble the components for the next step and determine if all assembly steps are complete. If not, repeat the above steps until all assembly steps are complete. Train the component category recognition network and the assembly step recognition network; The component category recognition network and the assembly step recognition network are trained to a preset batch, and the optimal models of the component category recognition network and the assembly step recognition network are saved.

2. The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images according to claim 1, characterized in that, The assembly monitoring phase includes: Assemble parts; Two physical depth images Q1 and Q2 from the same viewpoint are acquired. Q1 is the depth image of the assembly before assembly, and Q2 is the depth image of the assembly after assembly. A change image 2 is obtained from the physical depth images Q1 and Q2 by the difference method, and the change area of ​​the change image 2 is determined. The optimal model of the component category identification network identifies the category of the changed components represented by the changed regions in image 2. Render the changed areas in the changed image 2 to obtain the rendered image 2; The pixel values ​​of the changed regions in the rendered image 2 and the changed regions in the physical depth image Q2 are fused to obtain the fused image 2. The fused image 2 is input into the optimal model of the assembly step recognition network to identify the assembly step in which the assembly parts represented by the changing regions in the changed image 2 are located.

3. The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images according to claim 2, characterized in that, A changed image 1 is obtained from the synthesized depth image P1 and the synthesized depth image P2 using a difference method, and the changed region of the changed image 1 is determined, specifically as follows: The difference between the pixel values ​​at the same coordinates in the synthetic depth image P1 and the synthetic depth image P2 is used to obtain the changed image 1. The region in the changed image 1 where the pixel value is less than the threshold t is defined as the changed region in the changed image 1. The changed region in the changed image 1 is the region where the changed parts are located before and after assembly.

4. The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images according to claim 3, characterized in that, The changed area of ​​the changed image 1 is rendered to obtain the rendered image 1, specifically as follows: The rendering color is determined based on the category of the changed component represented by the changed area in the changed image 1, and the changed area in the changed image 1 is rendered to obtain the rendered image 1. The region in rendered image 1 with the same coordinates as the changed region in the transformed image 1 is defined as the changed region of rendered image 1.

5. The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images according to claim 4, characterized in that, Extracting the coordinates of the changing region in the rendered image 1, specifically: Extract the boundary coordinates of the changed region in the rendered image 1; The minimum values ​​of the horizontal and vertical coordinates of the region of change are defined as and Let the minimum x and y coordinates of the target box be 1. and ; The maximum values ​​of the horizontal and vertical coordinates of the region of change are defined as... and Let the maximum values ​​of the horizontal and vertical coordinates of the target bounding box be . and ; Based on coordinate points and Draw the target bounding box; Normalize the extracted coordinates: Normalized center coordinates of the bounding box : ； ； normalized width of the target box and high : ； ； in and The width and height of the rendered image 1.

6. The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images according to claim 5, characterized in that, Create labels for component category recognition datasets and assembly step recognition datasets, specifically as follows: The category of the changed component represented by the changed region, the normalized center coordinates, and the normalized width and height of the target box are saved in the txt1 file as the label of a sample in the component category recognition dataset. The assembly step of the changed component represented by the changed region in the changed image 1, the normalized center coordinates, and the normalized width and height of the target box are saved in the txt2 file as the label of a sample in the assembly step identification dataset.

7. The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images according to claim 6, characterized in that, The training of the component category recognition network and the assembly step recognition network is as follows: The target detection network is used as a component category recognition network and an assembly step recognition network; A component category recognition network is trained using a component category recognition dataset to identify the categories of varying components. An assembly step recognition network is trained using an assembly step recognition dataset to identify the assembly steps in which different parts are located.

8. The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images according to claim 7, characterized in that, The changed areas in the changed image 2 are rendered to obtain the rendered image 2, specifically: The rendering color is determined based on the category of the changed component represented by the changed area in the changed image 2, and the changed area in the changed image 2 is rendered to obtain the rendered image 2; the color rendered for each component category is the same as that in the training phase. The region in rendered image 2 that has the same coordinates as the changed region in the transformed image 2 is defined as the changed region of rendered image 2.

9. The method for detecting changes in mechanical assemblies and identifying assembly steps based on difference images according to claim 8, characterized in that, The target detection network is YOLOv5.

Images