Vision-based method for detecting assembly state of micro parts of complex mechatronic products

Abstract

The present application belongs to the field of mechanical and electrical product assembly, and particularly relates to a complex mechanical and electrical product micro part assembly state detection method based on vision. The method comprises the following steps: image acquisition: multiple images are taken, and the images cover the complex mechanical and electrical product completely; the original image is corrected in tilt and is spliced; the image of an assembly unit is acquired; the size of a first type of target object is 10mm-50mm, and the first type of target object is a parent part of a second type of target object; the image of the assembly unit is cut out and is input to a trained target detection model to detect the state of the first type of target object; the image of the first type of target object is extracted, the image of the second type of target object is extracted according to the position information of the second type of target object on the image of the first type of target object, and the assembly state of the second type of target object is detected; the second type of target object is a micro part with a diameter less than 10mm. The present application reduces the misassembly and missing assembly of parts on the complex mechanical and electrical product, and improves the assembly quality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of mechanical and electrical product assembly, and particularly relates to a visual-based complex mechanical and electrical product micro part assembly state detection method. BACKGROUND

[0002] Many processes of mechanical and electrical products have realized automatic production, but manual operation is still needed in some complex, narrow and other environments where mechanical arms are not easy to operate, which inevitably leads to missed or misassembled situations, so error-proof detection in the assembly process is very important. The traditional manual visual inspection method is easily affected by the subjective factors of the inspectors, and cannot guarantee the reliability and stability of the results, thereby affecting the assembly quality of the products. In recent years, image recognition technology has been gradually applied in assembly quality detection, but the current detection methods are mostly used for products or structural parts with small sizes and simple structures. For complex mechanical and electrical products with sizes of 3-5 meters or even larger, the existing assembly state detection methods cannot quickly and accurately detect the parts on the assembly body. Taking a radar array as an example, the size span of the parts on the array is large, the assembly unit plane size is about 30cm x 40cm, and the appearance diameter of the filter fixing part after assembly is only 6-10mm. Larger targets can be recognized by existing mature algorithms, but the recognition of smaller targets has always been a difficulty in target detection. SUMMARY

[0003] The purpose of the present application is to provide a visual-based complex mechanical and electrical product micro part assembly state detection method to reduce the misassembly and missed assembly of parts on complex mechanical and electrical products and improve the assembly quality.

[0004] The technical solution for achieving the purpose of the present application is as follows: a visual-based complex mechanical and electrical product micro part assembly state detection method, comprising the following steps:

[0005] Step (1): image acquisition: multiple images are taken, and the images cover the complex mechanical and electrical product completely;

[0006] Step (2): tilt correction is performed on the original images collected in step (1), and image splicing is performed;

[0007] Step (3): image acquisition of the assembly unit: the complete product image is input into the trained target detection model, the image position of the assembly unit is obtained, and the image of the assembly unit is cropped;

[0008] Step (4): acquisition of the first type of target object image: the size of the first type of target object is 10mm-50mm, which is the parent part of the second type of target object, and the image of the assembly unit cropped in step (3) is input into the trained target detection model to detect the state of the first type of target object;

[0009] Step (5): Detection of the assembly state of the second type of target object image: Extract the image of the first type of target object, extract the image of the second type of target object according to the position information of the second type of target object on the first type of target object image, and detect the assembly state of the second type of target object. The second type of target object is a micro part with a diameter less than 10 mm.

[0010] Further, in step (1), a plurality of image acquisition devices are used for image acquisition, and the distances of the plurality of image acquisition devices to the plane to be acquired are consistent. The images acquired by adjacent acquisition devices have overlapping regions, and the proportion of the overlapping regions to the images is 1 / 6-1 / 5.

[0011] Further, according to the target detection accuracy P v Determine the parameters of the acquisition device, the target detection accuracy P v is the number of pixels of the target to be detected on the image P n , and the ratio of the actual size h to the target detection accuracy P v is greater than 3 pixels per millimeter.

[0012] Further, the specific method of step (2) is as follows:

[0013] Step (21): Based on the original image, the straight lines in the image are detected by Hough transform;

[0014] Step (22): Calculate the inclination angle of each straight line and find its average value;

[0015] Step (23): Rotate the original image by the average value of the inclination angle obtained, so that the outer frame of the complex mechanical and electrical product is in an approximately horizontal or vertical state on the image.

[0016] Further, the specific method of image splicing in step (2) is as follows:

[0017] Step (24): Image feature point extraction: SURF is used as a feature point extraction algorithm;

[0018] Step (25): Image feature point matching: further filter the feature points in the two images to obtain better image matching points;

[0019] Step (26): Image registration: the matching point set of the two images to be spliced is obtained by step (25), and the two images are converted to the same coordinate;

[0020] The RANSAC algorithm is used to find the best homography matrix H, and the matrix size is 3x3. The equation is as follows:

[0021]

[0022] Wherein (x, y) represents the target image corner point position, (x', y') is the scene image corner point position, s is the scale parameter, the RANSAC algorithm randomly extracts 4 samples from the matching data set and ensures that the 4 samples are not collinear, calculates the homography matrix, then uses this model to test all data, and calculates the number of data points that meet this model and the projection error, i.e. The cost function is the minimum if the model is the optimal model, then the corresponding cost function is the minimum;

[0023] The calculation formula of the cost function is as follows:

[0024]

[0025] Step (27): image fusion: transforming the image by using the homography matrix obtained by image registration, and splicing the transformed image.

[0026] Further, the training of the target detection model in step (3) is specifically:

[0027] Obtain a plurality of complete images of complex electromechanical products;

[0028] Label the complete images of complex electromechanical products, and the labeling content is the image position of the assembly unit, which is in the form of (Px, Py, m, n), wherein Px and Py are the coordinates of the center point of the assembly unit on the image, and m and n are the proportions of the horizontal and vertical dimensions of the labeling frame to the horizontal and vertical directions of the image, respectively.

[0029] Based on the labeled complete images of complex electromechanical products, an assembly unit target detection model is trained and obtained.

[0030] Further, the training of the target detection model in step (4) is specifically:

[0031] The cropped assembly unit image is labeled for the first type of target object, and the labeling content is the image position of the first type of target object; The form is (Px, Py, m, n), wherein Px and Py are the coordinates of the center point of the first type of target object on the image, and m and n are the proportions of the horizontal and vertical dimensions of the labeling frame to the horizontal and vertical directions of the image, respectively.

[0032] Based on the labeled assembly unit image, a first type of target object detection model is trained and obtained.

[0033] Further, the target detection model adopts a convolutional neural network as the network skeleton.

[0034] Further, the assembly state detection of the second type of target object further includes:

[0035] Obtain a plurality of images of the second type of target object correctly installed, incorrectly installed and not installed as training samples;

[0036] Extract key point information of the second type of target object; train a classification model to classify the image at the position of the second type of target object according to the key point information; the extraction of the key point information refers to extracting image features according to pixels, and the implementation manner is as follows:

[0037] Perform size normalization processing on the image of the second type of target object;

[0038] The HOG feature of the gradient direction histogram of the local region of the image is calculated and statistically analyzed as the key point information; first, the image is divided into small connected regions, then the direction histogram of the gradient or edge of each pixel point in the connected region unit is extracted, and finally the histograms are combined to form the key point information; the calculation formula of the direction gradient is as follows:

[0039] G x (x,y)=H(x+1,y)-H(x-1,y)

[0040] G y (x,y)=H(x,y+1)-H(x,y-1)

[0041] Wherein Gx(x,y) represents the horizontal direction gradient of the pixel point (x,y), Gy(x,y) represents the vertical direction gradient of the pixel point (x,y), and H(x,y) represents the result of the pixel point (x,y) after gamma regularization processing;

[0042] The implementation manner of classifying the image according to the key point information is as follows:

[0043] After extracting the key point information of the image, the image is classified by using a support vector machine, the basic model of the support vector machine is defined as a linear classifier with the largest interval in the feature space, and finally the problem is converted into the solution of a convex quadratic programming problem.

[0044] A use of the above method is used for a large radar array, the first type of target object is a filter, and the second type of target object is a filter fixing part, i.e., a fastening screw and a positioning screw.

[0045] Compared with the prior art, the present application has the following advantages:

[0046] The present application only needs to collect the image of a complex electromechanical product once, and the operation is simple; in addition, the product image is inclined and corrected through correction processing, which reduces the false detection problem caused by the placement deviation of the collection equipment; the detection target is recognized through the convolutional neural network, which improves the speed and accuracy of recognition; the second type of detection target is detected through the positional relationship between parts, which improves the detection capability of small size targets. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1The flow chart of the large radar array micro part assembly state detection of the application.

[0048] Figure 2 The example diagram of the complex electromechanical product to be detected of the application.

[0049] Figure 3 The example diagram of the assembly unit to be detected of the application.

[0050] Figure 4 The correct assembly image of the part to be detected of the application.

[0051] Figure 5 The incorrect assembly image of the part to be detected of the application. DETAILED DESCRIPTION

[0052] The application will be further described in detail below with reference to the accompanying drawings.

[0053] With reference to the accompanying Figure 1 , the flow of the complex electromechanical product micro part assembly state detection method based on vision provided by the embodiment of the application is as follows: multiple local photos of the complex electromechanical product are taken to ensure complete coverage of the entire product, and the photos are corrected; the large radar array local photos after correction are spliced into the complex electromechanical product; the complete product image is sent to the pre-trained target detection model to detect the assembly unit, extract the part image, and perform the next operation; the cropped assembly unit image is sent to the pre-trained target detection model to detect the second type of target object, extract the part image, and perform the next operation; according to the relative position of the second type of target object and the first type of target object, the image of the second type of target object is extracted, and the assembly state detection of the second type of target object is performed.

[0054] In the embodiment, the complex electromechanical product to be detected is taken as an example of a large radar array, and the example diagram is shown in FIG. 1. Figure 2 During assembly, the clamping tool (numbered 2 in the figure) places the radar array (numbered 1 in the figure) vertically to the ground, and the target to be detected is the radar assembly unit (numbered 3 in the figure) and the parts inside it.

[0055] The internal structure of the radar assembly unit is shown in FIG. 2. Figure 3 The target to be detected in the radar assembly unit is the filter (numbered 4 in the figure), the fastening screw (numbered 5 in the figure), and the positioning screw (numbered 6 in the figure).

[0056] In the above, the first type of detection target is the filter, and the second type of detection target is the filter fixing part, i.e., the fastening screw and the positioning screw. Figure 4 A sample image of the correct installation of the fastener of the filter of the embodiment is shown in FIG. 3. Figure 5A sample image of an embodiment filter incorrectly installed fastener is shown, with a red box indicating the location of the incorrectly installed part.

[0057] In the specific implementation process, the collection of the original image of the radar array also includes:

[0058] According to the detection accuracy, the pixels, interfaces, sensor types, etc. of the collection device are determined;

[0059] When multiple devices are used for collection, the distance between the several image collection devices and the plane to be collected should be consistent.

[0060] In the specific implementation process, the tilt correction method for the original image includes:

[0061] Based on the original image, the straight lines in the image are detected by Hough transformation;

[0062] The tilt angle of each straight line is calculated, and the average value is obtained;

[0063] The original image is rotated by the obtained average value of the tilt angle, thereby obtaining the corrected image.

[0064] In the specific implementation process, the image splicing operation includes:

[0065] Image feature point extraction. The extraction of features is to match the images, and the selection of features as the basis for registration should be considered according to the characteristics of the images to be registered;

[0066] Image feature point matching. The purpose of image feature point matching is to further filter the feature points in the two images and obtain more excellent image matching points;

[0067] Image registration. Through the operation of the previous step, the matching point set of the two images to be spliced is obtained, and then the two images need to be converted to the same coordinate;

[0068] Image fusion. This step is to transform the image by the homography matrix obtained by image registration, and then splice the transformed image.

[0069] In the specific implementation process, three aspects need to be considered in the selection of feature points:

[0070] First, select the features that exist in both images to be registered. Only the features that can be extracted from both images can be used for image feature matching;

[0071] Second, a sufficient number of features must be extracted from both images simultaneously to be effective;

[0072] Third, the selected features must have good uniqueness and be easy for the next step of feature matching.

[0073] In the implementation process, as many sample images as possible should be obtained for training to ensure the accuracy of the model. After ensuring sufficient samples, the structure and weight of the network are adjusted to make the output of the network consistent with the expected value. Under the condition of ensuring the quality and distribution balance of sample data, the size of sample data determines the accuracy of neural network training results. The larger the sample data, the higher the accuracy.

[0074] In the implementation process, the detection of the assembly state of the fastening screw and the positioning screw also includes:

[0075] Obtain several correct installation and incorrect installation images of the fastening screw and the positioning screw as training samples.

[0076] Extract the key point information of the fastening screw and the positioning screw.

[0077] Train the classification model to classify the images of the positions of the fastening screw and the positioning screw according to the key point information.

[0078] In the implementation process, the extraction of the key point information refers to extracting image features according to pixels. One possible implementation way is:

[0079] By calculating and counting the gradient direction histogram of the local area of the image, specifically, first divide the image into small connected regions, then the direction histogram of the gradient or edge of each pixel point in these connected region units. Finally, these histograms are combined to form the key point information.

[0080] In the implementation process, one possible implementation way of classifying the image according to the key point information is:

[0081] After extracting the key point information of the image of the fastening screw and the positioning screw, the image is classified by using a support vector machine. The basic model of the support vector machine is defined as a linear classifier with the largest interval in the feature space, and finally the problem is converted into the solution of a convex quadratic programming problem. Through the classifier, the image is distinguished, and the purpose of part assembly state detection is achieved.

Claims

1. A vision-based method for detecting the assembly status of micro-parts in complex electromechanical products, characterized in that, Includes the following steps: Step (1): Image acquisition: Take multiple images to completely cover the complex electromechanical product; Step (2): The original image acquired in step (1) is tilted and then stitched together; Step (3): Image acquisition of assembly unit: Input the complete product image into the trained target detection model, obtain the image position of the assembly unit, and crop out the image of that part; Step (4): Acquisition of the first type of target object image: The size of the first type of target object is 10mm~50mm, which is the parent part of the second type of target object. The image of the assembly unit cropped in step (3) is input into the trained target detection model to perform state detection on the first type of target object; Step (5): Detection of assembly status of the second type of target object image: Extract the image of the first type of target object, extract the image of the second type of target object based on the position information of the second type of target object on the image of the first type of target object, and perform assembly status detection on the second type of target object. The second type of target object is a small part with a diameter of less than 10mm. The specific method for tilt correction in step (2) is as follows: Step (21): Based on the original image, detect straight lines in the image using Hough transform; Step (22): Calculate the angle of inclination of each line and find its average value; Step (23): Rotate the original image by the average tilt angle obtained, so that the outer frame of the complex electromechanical product is approximately horizontal or vertical in the image. The specific method for image stitching in step (2) is as follows: Step (24): Image feature point extraction: SURF is used as the feature point extraction algorithm; Step (25): Image feature point matching: Further filter the feature points in the two images to obtain better image matching points; Step (26): Image registration: The matching point set of the two images to be stitched is obtained through step (25), and the two images are transformed into the same coordinate system; The RANSAC algorithm is used to find the optimal homography matrix H, which has a size of 3×3. The equation is as follows: ， Where (x, y) represents the corner position of the target image, (x', y') represents the corner position of the scene image, and s is the scale parameter. The RANSAC algorithm randomly selects 4 samples from the matching dataset and ensures that these 4 samples are not collinear. It calculates the homography matrix, then uses this model to test all data, and calculates the number of data points that satisfy this model and the projection error. If this model is the optimal model, then the corresponding cost function is minimized. The formula for calculating the cost function is as follows: ； Step (27): Image fusion: Transform the image using the homography matrix obtained from image registration, and then stitch the transformed images together.

2. The method according to claim 1, characterized in that, In step (1), several image acquisition devices are used to acquire images. The distance between the several image acquisition devices and the plane to be acquired is kept consistent. The images acquired by adjacent acquisition devices have overlapping areas, and the overlapping area accounts for 1 / 6 to 1 / 5 of the image.

3. The method according to claim 2, characterized in that, Based on target detection accuracy Determine the parameters of the data acquisition equipment and the target detection accuracy. The number of pixels of the target to be detected in the image Compared to actual size The ratio of target detection accuracy Greater than 3 pixels per millimeter.

4. The method according to claim 3, characterized in that, The training of the target detection model in step (3) is specifically as follows: Acquire several complete images of complex electromechanical products; The complete image of a complex electromechanical product is annotated. The annotation content is the image position of the assembly unit, in the form of (Px, Py, m, n), where Px and Py are the coordinates of the center point of the assembly unit on the image, and m and n are the proportions of the horizontal and vertical dimensions of the annotation box to the horizontal and vertical dimensions of the image, respectively. An assembly unit target detection model is trained based on the complete image of the labeled complex electromechanical product.

5. The method according to claim 4, characterized in that, Step (4) The training of the object detection model is specifically as follows: The cut-out assembly unit image is used to annotate the first type of target object. The annotation content is the image position of the first type of target object; the form is (Px, Py, m, n), where Px and Py are the coordinates of the center point of the first type of target object on the image, and m and n are the proportions of the horizontal and vertical dimensions of the annotation box to the horizontal and vertical dimensions of the image, respectively. Based on the labeled assembly unit images, a first-class target object detection model is trained.

6. The method according to claim 5, characterized in that, All object detection models use convolutional neural networks as the network skeleton.

7. The method according to claim 6, characterized in that, The assembly status detection of the second type of target object also includes: Several images of the second type of target object that are correctly installed, incorrectly installed, or not installed are obtained as training samples; Extract key point information of the second type of target object; train a classification model to classify the image of the location of the second type of target object according to the key point information; key point information extraction refers to extracting image features based on pixels, and the implementation method is as follows: The size of the second type of target object image is normalized; Keypoint information is obtained by calculating and statistically analyzing the gradient orientation histogram (HOG) of local image regions. First, the image is divided into small connected regions. Then, the gradient or edge orientation histograms of each pixel within these connected region units are extracted. Finally, these histograms are combined to form the keypoint information. The formula for calculating the directional gradient is as follows: ， ， Where Gx(x,y) represents the horizontal gradient of pixel (x,y), Gy(x,y) represents the vertical gradient of pixel (x,y), and H(x,y) represents the result of pixel (x,y) after gamma regularization. The method for classifying images based on key point information is as follows: After extracting key point information from the image, a support vector machine is used to classify the image. The basic model of the support vector machine is defined as a linear classifier with the largest margin in the feature space. Finally, the problem is transformed into solving a convex quadratic programming problem.

8. An application of the method according to any one of claims 1-7, characterized in that, For use in large radar arrays, the first type of target object is the filter, and the second type of target object is the filter fixing parts, namely fastening screws and positioning screws.

Images