A multi-view screw missing detection method and system

Abstract

The present invention discloses a multi-view screw missing detection method and system, including: S1: Obtain multi-view images of each screw, where the multi-view images include multiple single-view images; S2: Collect non-missing images of each screw and perform preprocessing to obtain a template image, mark the template screw and template key points, and calculate the first included angle and the length of the first side; S3: Identify the multi-view images of the screws to be measured in S1, mark the screws therein with a first mark, and mark two key points with a second mark; S4: Calculate the position i of the template screw in the single-view image of the screw to be measured, and determine a measurement region with i as the center and r as the radius. If there is a first mark in the measurement region, it is considered that the screw is not missing, otherwise it is considered that the screw is missing; S5: Statistically analyze the detection results of multiple single-view images of the screws to be measured to obtain a multi-view detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of screw detection, and particularly relates to a multi-view screw missing detection method and system. Background Art

[0002] In industrial production, a large number of parts are assembled by manually installing screws. In this process, individual screws are likely to be missed due to factors such as staff negligence and fatigue, which will affect the product quality. Therefore, it is necessary to detect whether the screws are installed or missing.

[0003] Currently, in factories, manual methods are mostly used for screw missing detection. This method has problems such as low efficiency, slow speed, and waste of human resources. Moreover, due to the fast production rhythm, it cannot be taken into account, resulting in a low actual effective detection rate and accuracy in the real environment.

[0004] Currently, there are a small number of methods for screw missing detection by computer vision. It mainly uses traditional image processing methods (such as blob detection) to find the positions of screw holes in the image and matches them with the preset screw hole positions in the template to determine whether there is a missing screw. However, this type of method relies on traditional image processing methods. In practical applications, the light, the camera shooting angle, and the position and spatial relationship between the object to be measured and the camera are variable, resulting in poor robustness. And in the process, it relies too much on manually designed matching rules, resulting in insufficient flexibility and being unable to be effectively used in multiple detection scenarios. Summary of the Invention

[0005] Aiming at the technical problem of low accuracy in screw missing detection in the prior art, the present invention proposes a multi-view screw missing detection method and system. By finding key points of the image to calculate the position of the template screw in the image to be measured, and then checking whether there is a screw at this position, the detection accuracy is improved.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-view screw missing detection method specifically includes the following steps:

[0008] S1: Obtain multi-view images of each screw, and the multi-view images include multiple single-view images;

[0009] S2: Collect non-missing images of each screw, preprocess them to obtain template images, train the template images, mark the template screws and template key points, and calculate the first included angle and the length of the first side;

[0010] S3: Identify the multi-view images of the screws to be measured in S1, mark the screws therein with the first mark, and mark the two key points with the second mark;

[0011] S4: Calculate the position i of the template screw in the single-view image of the screw to be measured based on the two key points in S3 and the first included angle and the length of the first side in S2, and determine the area to be measured with i as the center and r as the radius. If there is a first mark in the area to be measured, it is considered that the screw is not missing; otherwise, it is considered that the screw is missing.

[0012] S5: Statistically analyze the detection results of multiple single-view images of the screw to be measured to obtain the multi-view detection result. If the detection result of any single-view image shows that it is not missing, the multi-view detection result is not missing; if the detection results of all single-view images show that it is missing, the multi-view detection result is missing.

[0013] Preferably, in S1, at least one image acquisition device is provided at each screw, and the image acquisition device is a camera.

[0014] Preferably, in S2, S2 includes:

[0015] S2-1: Obtain the non-missing image of each screw through the image acquisition device as the template image of the screw.

[0016] S2-2: In each template image, mark n screws with a first mark as template screws, then mark the first template key point and the second template key point with a second mark, and connect the first template key point and the second template key point to obtain the first image.

[0017] S2-3: Establish a coordinate system with the lower left corner of the first image as the coordinate origin, obtain the coordinates of the first template key point, the second template key point and the template screw, and calculate the first included angle and the length of the first side.

[0018] Preferably, in S2-3, the included angle formed by the connection line between the first template key point and the template screw and the connection line between the second template key point and the template screw is the first included angle, and its calculation formula is:

[0019]

[0020] In formula (1), α i represents the first included angle of the i-th template screw; (x a , y a ) represents the coordinates of the first template key point k a ; (x b , y b ) represents the coordinates of the second template key point k b ; (x i , y i ) represents the coordinates of the i-th template screw.

[0021] Preferably, in S2-3, the first side is the line connecting the first template key point and the template screw, and its length calculation formula is:

[0022]

[0023] In formula (2), l i represents the length of the first side; (x a , y a ) represents the coordinates of the first template key point k a ; (x i , y i ) represents the coordinates of the i-th template screw.

[0024] Preferably, in S4, the calculation formula for the position i of the template screw in the single-view image of the screw to be measured is:

[0025] x' i = l i * cos(α i ), y' i = l i * sin(α i ) (3)

[0026] In formula (3), (x' i , y' i ) represents the position of the template screw in the multi-view image of the screw to be measured; l i represents the length of the first side; α i represents the first included angle.

[0027] Preferably, in S5, the image acquisition device set Ci = {ci1, ci2,.., cij,.., cim}, cim represents the m-th view image of the i-th screw, where 1 ≤ j ≤ m; when there is no missing in cij for the screw, it indicates that there is no missing in the image acquisition device set, and the multi-view detection result is 1; when all view images of the screw are missing, it indicates that there is a missing in the image acquisition device set, and the multi-view detection result is 0.

[0028] The present invention also provides a multi-view screw missing detection system, including:

[0029] An image acquisition device for acquiring images of screws;

[0030] An image training module for training the acquired screw non-missing images and marking the template screws and template key points;

[0031] A detection module for performing screw marking and key point marking on the single-view image of the screw to be measured;

[0032] A matching module, configured to calculate the position of the template screw in the single-view image of the screw to be measured, and detect whether there is a screw at this position.

[0033] A display module, configured to display whether there is a missing screw in the multi-view image of the screw to be measured. If there is a missing screw, display "0", and if there is no missing screw, display "1".

[0034] In summary, due to the adoption of the above technical solutions, compared with the prior art, the present invention has at least the following beneficial effects:

[0035] The present invention takes pictures from multiple perspectives, is less affected by light, etc., and the screws in the area with more components are not easily blocked, and has a strong adaptability to the external environment, so as to detect various production devices on different production lines with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 FIG. is a schematic diagram of a multi-view screw missing detection method according to an exemplary embodiment of the present invention.

[0037] Figure 2 FIG. is a schematic diagram of the first image after processing the image of the screw without missing obtained by the image acquisition device according to an exemplary embodiment of the present invention.

[0038] Figure 3 FIG. is a schematic diagram of a multi-view screw missing detection system according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The present invention will be further described in detail below in conjunction with embodiments and specific implementation manners. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments. All technologies implemented based on the content of the present invention belong to the scope of the present invention.

[0040] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the present invention.

[0041] As Figure 1 shown, the present invention provides a multi-view screw missing detection method, which specifically includes the following steps:

[0042] S1: Set at least one image acquisition device at each screw to acquire multi-view images of each screw, and the multi-view images include multiple single-view images.

[0043] In this embodiment, at least one image acquisition device (such as a camera) is installed at each screw position to acquire images; and for screws that are easily blocked (such as where wires pass and there are many installation components), at least two image acquisition devices are set to acquire multi-view images (that is, images from multiple perspectives, each image is taken by an image acquisition device), so as to avoid the images at a certain angle being blocked.

[0044] S2: Collect the complete images of each screw and perform preprocessing to obtain a template image, and train the template image, mark the template screws and template key points, and calculate the first included angle and the length of the first side.

[0045] S2-1: Use the image acquisition device to acquire the complete image of each screw as the template image of the screw;

[0046] In this embodiment, the template image obtained by each camera may contain multiple screws. For example, in the top view, many screws are not blocked and can be detected all in one template image.

[0047] S2-2: In each template image, mark n screws with a first mark (the screws can be marked with a rectangular frame) as template screws, and then mark the first template key point and the second template key point with a second mark, and connect the first template key point and the second template key point to obtain a first image.

[0048] In this embodiment, the first mark is a rectangle and the second mark is a circle.

[0049] In this embodiment, the key points are the points set manually during image annotation. It is required that the environment around the key point part has a large difference and is easy to detect; in this patent, the inflection points of the components are mainly used as key points.

[0050] As Figure 2 shown, it is the first image after processing the complete image of the screw acquired by the image acquisition device, where S 1 、S 2 、S 3 、S 4 、S 5 、S 6 are template screws and are marked with the first mark; k a is the first template key point in the template image, and k b is the second template key point in the template image.

[0051] S2-3: Establish a coordinate system with the lower left corner of the first image as the coordinate origin, obtain the coordinates of the first template key point, the second template key point and the template screws, and calculate the first included angle and the length of the first side.

[0052] In this embodiment, the included angle formed by the line connecting the first template key point and the template screw and the line connecting the second template key point and the template screw is the first included angle, and its calculation formula is:

[0053]

[0054] In formula (1), α i represents the first included angle of the i-th template screw; (x a , y a ) represents the coordinates of the first template key point k a ; (x b , y b ) represents the coordinates of the second template key point k b ; (x i , y i ) represents the coordinates of the i-th template screw.

[0055] In this embodiment, the first side is the line connecting the first template key point and the template screw, and its length calculation formula is:

[0056]

[0057] In formula (2), l i represents the length of the first side; (x a , y a ) represents the coordinates of the first template key point k a ; (x i , y i ) represents the coordinates of the i-th template screw.

[0058] S3: Use the trained marking method to identify the multi-view images of the screws to be measured in S1, mark the screws with the first mark, and mark the two key points (the inflection points of the components) with the second mark.

[0059] In this embodiment, screw detection is a computer vision task, the purpose of which is to find all the targets of interest in the image and use a rectangular detection frame to determine their categories and positions. Screw detection is divided into algorithms based on anchor boxes and anchor-free boxes. As an anchor-free box algorithm, FCOS does not require manual setting of hyperparameters and can achieve similar or even better accuracy than the algorithm based on anchor boxes. Therefore, this patent uses the FCOS algorithm to detect screws.

[0060] In this embodiment, the two key points are marked as k a ’ and k b ’.

[0061] S4: Calculate the position i of the template screw in the single-view image of the screw to be measured based on the two key points in S3 and the length of the first side and the first included angle in S2, and determine the measurement area with i as the center and r (r is a threshold value, which is the radius of the screw) as the radius. If there is a screw in the measurement area, it is considered that the screw is not missing; otherwise, it is considered that the screw is missing.

[0062] In this embodiment, with the first key point k a ’ as the origin and k a ’k b ’ as the x-axis to establish a coordinate system, and use the length l i of the first side and the first included angle α i in S2 to calculate the position (x' i , y' i ) of the template screw in the multi-view image of the screw to be measured, as shown in formula (3):

[0063] x' i = l i * cos(α i ), y' i = l i * sin(α i ) (3)

[0064] In formula (3), (x' i , y' i ) represents the position of the template screw in the multi-view image of the screw to be measured; l i represents the length of the first side; α i represents the first included angle.

[0065] In this embodiment, in the measurement area, if the first mark is found, it is considered that the screw exists; if the first mark is not found, it is considered that the screw is missing.

[0066] In this embodiment, if there are multiple screws in the measurement area, it is considered that the screw closest to point i matches the template screw.

[0067] S5: Statistically analyze the detection results of multiple single-view images of the screw to be measured to obtain the multi-view detection result. If the detection result of any single-view image shows not missing, the multi-view detection result is not missing; if the detection results of all single-view images show missing, the multi-view detection result is missing.

[0068] In this embodiment, if the multi-view detection result is not missing, mark the screw with a third mark (a square can be used).

[0069] In this embodiment, the set of image acquisition devices for the screw Ci = {ci1, ci2,.., cij,.., cim}, where cim represents the m-th perspective image of the i-th screw, where 1 ≤ j ≤ m; the matching result of the screw in cij, 0 indicates missing, and 1 indicates not missing; when there is no missing screw in cij, it means there is no missing in the set of image acquisition devices, and the multi-perspective detection result is 1; when all perspective images of the screw are missing, it means there is a missing in the set of image acquisition devices, and the multi-perspective detection result is 0.

[0070] The present invention has the following advantages:

[0071] High accuracy: The screw missing detection method used in the present invention has almost no detection errors in industrial production. The correct rate can reach more than 99.8% in actual production;

[0072] Good robustness: The present invention is less affected by light and the like, and has a strong adaptability to the external environment;

[0073] Good flexibility: The method used in the present invention can detect various production devices on different production lines;

[0074] Less affected by occlusion between components: For screws in areas with many components, cameras at multiple angles are used for joint detection, so that the detection result of the present invention is less affected by occlusion between components.

[0075] Based on the above method, as Figure 3 shown, the present invention also provides a multi-perspective screw missing detection system, including an image acquisition device, an image training module, a detection module, a matching module, and a display module; the output end of the image acquisition device is respectively connected to the input end of the image training module and the input end of the detection module, the output end of the detection module is connected to the first input end of the matching module, the output end of the image training module is connected to the second input end of the matching module, and the output end of the matching module is connected to the input end of the display module.

[0076] The image acquisition device is used to acquire images of screws;

[0077] The image training module is used to train the acquired images of screws without missing, and mark the template screws and template key points;

[0078] The detection module is used to mark the screws and key points for the single perspective image of the screw to be detected;

[0079] The matching module is used to calculate the position of the template screw in the single perspective image of the screw to be detected, and detect whether there is a screw at this position;

[0080] A display module is configured to display whether there is a missing part in the multi-view images of the screw to be measured. If there is a missing part, it displays "0", and if there is no missing part, it displays "1".

[0081] Those of ordinary skill in the art can understand that the above-described embodiments are specific examples for implementing the present invention. In actual applications, various changes can be made to its form and details without departing from the spirit and scope of the present invention.

Claims

1. A multi-view screw missing detection method, characterized in that , specifically including the following steps: S1: Obtain multi-view images of each screw, where the multi-view images include multiple single-view images; S2: Collect non-missing images of each screw and preprocess them to obtain template images, and train the template images, mark the template screws and template key points, and calculate the first included angle and the length of the first side; the first side is the line connecting the first template key point and the template screw; The S2 includes: S2-1: Obtain non-missing images of each screw through an image acquisition device as the template images of the screws; S2-2: In each template image, mark n screws with the first mark as template screws, and then mark the first template key point and the second template key point with the second mark, and connect the first template key point and the second template key point to obtain the first image; S2-3: Establish a coordinate system with the lower left corner of the first image as the coordinate origin, obtain the coordinates of the first template key point, the second template key point and the template screw, and calculate the first included angle and the length of the first side; S3: Identify the multi-view images of the screws to be measured in S1, mark the screws with the first mark, and mark two key points with the second mark; S4: Calculate the position i of the template screw in the single-view image of the screw to be measured according to the two key points in S3 and the first included angle and the length of the first side in S2, and determine the measurement area with i as the center and r as the radius. If there is a first mark in the measurement area, it is considered that the screw is not missing, otherwise it is considered that the screw is missing; In the S4, the calculation formula for the position i of the template screw in the single-view image of the screw to be measured is: x' i = l i * cos(α i ), y' i = l i * sin(α i )(3) In formula (3), (x' i , y' i ) represents the position of the template screw in the multi-view images of the screw to be measured; l i represents the length of the first side; α i represents the first included angle; S5: Statistically analyze the detection results of multiple single-view images of the screws to be measured to obtain multi-view detection results. If the detection result of any single-view image shows no missing, the multi-view detection result is no missing; if the detection results of all single-view images show missing, the multi-view detection result is missing.

2. A multi-view screw missing detection method according to claim 1, characterized in that , in the S1, at least one image acquisition device is provided at each screw, and the image acquisition device is a camera.

3. A multi-view screw missing detection method according to claim 1, characterized in that , in the S2-3, the included angle formed by the line connecting the first template key point and the template screw and the line connecting the second template key point and the template screw is the first included angle, and its calculation formula is: In formula (1), α i represents the first included angle of the i-th template screw; (x a , y a ) represents the coordinates of the first template key point k a ; (x b , y b ) represents the coordinates of the second template key point k b ; (x i , y i ) represents the coordinates of the i-th template screw.

4. A multi-view screw missing detection method according to claim 1, characterized in that , in the S2-3, the first side is the line connecting the first template key point and the template screw, and its length calculation formula is: In formula (2), l i represents the length of the first side; (x a , y a ) represents the coordinates of the key point k a of the first template; (x i , y i ) represents the coordinates of the i-th template screw.

5. A multi-view screw missing detection method according to claim 1, characterized in that , in S5, the set of image acquisition devices \(C_i=\{c_{i1}, c_{i2},.., c_{ij},.., c_{im}\}\), where \(c_{im}\) represents the \(m\)-th perspective image of the \(i\)-th screw, and \(1\leq j\leq m\); when there is no missing screw in \(c_{ij}\), it indicates that there is no missing screw in the set of image acquisition devices, and the multi-perspective detection result is 1; when all perspective images of the screw are missing, it indicates that there is a missing screw in the set of image acquisition devices, and the multi-perspective detection result is 0.

6. A multi-perspective screw missing detection system based on the method according to any one of claims 1-5, characterized in that, it includes: an image acquisition device for acquiring images of screws; an image training module for training the acquired images of screws without missing, and marking the template screws and template key points; a detection module for marking screws and key points on the single-perspective image of the screw to be measured; a matching module for calculating the position of the template screw in the single-perspective image of the screw to be measured, and detecting whether there is a screw at this position; a display module for displaying whether there is a missing screw in the multi-perspective image of the screw to be measured. If there is a missing screw, it displays "0", and if there is no missing screw, it displays "1".

Images