Method for detecting missing or incorrect inspection points on workpieces

Abstract

The method for detecting missing or incorrect inspection points on a workpiece of the present invention belongs to the technical field of image recognition methods and solves the technical problem of low efficiency of using the naked eye to detect workpieces in the prior art. It includes using a binocular stereo camera to photograph the workpiece to obtain an image to be detected; converting the coordinates between the binocular stereo camera and the detection plane to determine the conversion matrix of the binocular stereo camera; determining the parameters of the inspection points to be measured in the image to be detected, and determining the depth of each inspection point based on the parameters of the inspection points to be measured; the depth and the coordinates of each inspection point are used to determine the coordinates of each inspection point in the world coordinate system through the conversion matrix; obtaining the standard parameters of all inspection points of the workpiece standard part in the world coordinate system, and completing the detection by comparison. The present invention replaces the traditional manual recognition method with camera image recognition to improve detection efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile workpiece detection methods, and in particular relates to a method for detecting misassembly or missing of detection points on a workpiece. Background Art

[0002] Detecting incorrect or missing parts in automotive parts is a crucial step in ensuring their quality. Currently, this process on production lines relies primarily on visual inspection by workers, requiring them to review each inspection point. This process can be time-consuming and inefficient, requiring workers to inspect each inspection point repeatedly. Over time, this process can lead to fatigue and errors. Larger parts require a greater number of inspection points, potentially reaching dozens or even hundreds. This process is time-consuming and inefficient, slowing production and reducing efficiency.

[0003] In view of this, the present invention is proposed. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for detecting misalignment and missing parts of a workpiece, thereby resolving the technical problem of low efficiency in visual inspection of workpieces in the prior art. The technical solution of this case has many technical benefits, as described below:

[0005] A method for detecting misalignment or missing assembly of a workpiece is provided, which is suitable for detecting misalignment or missing assembly of a workpiece. The method comprises:

[0006] Using a binocular stereo camera to photograph the workpiece to obtain an image to be inspected, and calibrating the binocular stereo camera before photographing;

[0007] Coordinate conversion between the binocular stereo camera and the detection plane, determining a conversion matrix of the binocular stereo camera, wherein the conversion matrix is ​​used to convert the image coordinates taken by the binocular stereo camera into world coordinates with the detection plane as the coordinate system;

[0008] Determining parameters of the detection points to be measured in the image to be detected, and determining the depth of each detection point according to the parameters of the detection points to be measured; the parameters of the detection points to be measured include at least the position, size and category of the detection points in the image coordinates;

[0009] Obtain the coordinates of each detection point in the image coordinate system, and determine the coordinates of each detection point in the world coordinate system through the transformation matrix, that is, the first coordinates;

[0010] Obtain the standard parameters of all detection points of the workpiece standard part in the world coordinates, and determine whether the workpiece is qualified based on the first coordinates of all detection points of the workpiece to be tested and the parameters of the detection points to be tested.

[0011] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0012] The method of the present invention aims to improve the efficiency of workpiece detection by recognizing images taken by a camera, replacing manual recognition. The workpiece is detected to determine whether it is qualified by converting image coordinates to world coordinates and judging the coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A schematic diagram of a method for detecting misalignment or missing assembly of a detection point on a workpiece according to the present invention; DETAILED DESCRIPTION

[0015] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0016] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present invention, those skilled in the art will appreciate that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0017] It should also be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0018] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that aspects can be practiced without these specific details. In order to enable those skilled in the art to better understand the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. The terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise stated, "multiple" means two or more.

[0019] The workpiece is a machined and formed workpiece used in vehicles, and is equipped with inserts, screws, nuts, tape or rubber strips, etc. When inspecting on the inspection platform, each inspection point of the workpiece needs to be inspected to see if there are any missing or incorrect installations.

[0020] like Figure 1 The method for detecting misalignment or missing assembly of a workpiece is suitable for detecting misalignment or missing assembly of a workpiece. The method includes:

[0021] S101: Use a binocular stereo camera to photograph a workpiece to obtain an image to be inspected, and calibrate the binocular stereo camera before photographing. Specifically, the method for calibrating the binocular stereo camera includes:

[0022] Make sure the binocular stereo camera is set up in parallel with two monocular industrial cameras of the same model;

[0023] Two monocular industrial cameras are fixed by external mechanical devices and the standard focal length is adjusted;

[0024] Use binocular stereo calibration method to perform calibration and determine the internal parameters of two monocular industrial cameras;

[0025] Through the stereo correction matrix, the two intrinsic parameters are corrected so that the images taken by the two monocular industrial cameras are completely aligned in the Y-axis direction.

[0026] Specifically, a binocular stereo camera consists of two identical monocular industrial cameras placed in parallel. These cameras are mechanically secured and their focal lengths adjusted. Binocular stereo calibration is then used to obtain the intrinsic parameters of the two cameras. The two cameras are then corrected using a stereo calibration matrix to ensure that the images are perfectly aligned along the y-axis. This initial calibration ensures that the binocular stereo camera image coordinates are aligned, paving the way for subsequent conversion to world coordinates.

[0027] S102: Coordinate conversion between the binocular stereo camera and the detection plane. Determine the conversion matrix of the binocular stereo camera. The conversion matrix is ​​used to convert the image coordinates taken by the binocular stereo camera into the world coordinates with the detection plane as the coordinate system. Specifically, during the detection process, the number of cameras is set according to the actual situation as follows:

[0028] 1) When the number of detections is small, a single binocular camera is sufficient. The method for determining the conversion matrix of the binocular stereo camera includes: when setting up a binocular stereo camera, calibrating the coordinate conversion relationship between the binocular stereo camera and the detection plane, and determining the conversion matrix of the binocular stereo camera.

[0029] 2) A plurality of binocular stereo cameras are installed on the detection plane. Coordinate conversion between the binocular stereo cameras and the detection plane and a method for determining a conversion matrix of the binocular stereo cameras include:

[0030] Select any binocular stereo camera as the main camera and the rest as secondary cameras;

[0031] Calibrate the transformation matrix between the secondary camera and the main camera to transform the image coordinates of the secondary camera into the world coordinates with the detection plane on the detection platform as the coordinate system;

[0032] Calibrate the coordinate transformation relationship between the main camera and the detection plane, and determine the transformation matrix of the main camera;

[0033] The transformation matrix between the primary camera and the detection plane and the transformation matrix between the secondary camera and the primary camera determine the secondary camera's world coordinate transformation matrix with the detection plane as its coordinate system. This also determines the coordinates of all secondary cameras in world coordinates. The image coordinates of all cameras can be transformed into the same world coordinate system with the detection plane as its coordinate system, completing the coordinate system unity for all cameras.

[0034] Multiple cameras can be placed relatively freely on the detection platform, and can be calibrated through subsequent methods to solve the problem of troublesome camera installation.

[0035] In the prior art, machine vision inspection products used on production lines are fixture-based. The workpiece must pass through a custom fixture, which secures the workpiece and the camera in a fixed position. The position of the inspection point on the workpiece in the image is also fixed. The ROI (region of interest) of the inspection point can be extracted from a preset location, and then inspection is performed through image coordinate comparison. However, this method requires the camera to be fixed. Furthermore, when the workpiece is too large or the inspection point is too small, the camera's field of view must cover the entire workpiece, resulting in a very low pixel density at the inspection point. This makes image-based comparisons highly inaccurate, leading to erroneous results. Therefore, the existing solution for this situation involves using a robotic arm to move the camera and take pictures. Each time, the robotic arm moves the camera to the area where the inspection point is located, capturing an image with a sufficiently high pixel density at the inspection point, which is then used for image comparison. The use of a robotic arm results in slow inspection speeds, and operational errors in the robotic arm can lead to high image comparison errors, impacting inspection efficiency.

[0036] S103: Determine the parameters of the detection points to be tested in the image to be tested, and determine the depth of each detection point according to the parameters of the detection points to be tested; the parameters of the detection points to be tested include at least the position, size and category of the detection points in the image coordinates, and the binocular stereo camera includes a first camera and a second camera, specifically:

[0037] Acquire a first image to be detected and a second image to be detected from the first camera and the second camera;

[0038] The first camera and the second camera perform image transformation respectively, and the transformation results determine that the first image to be detected and the second image to be detected are images with translation only in the x-axis direction;

[0039] The first image to be detected is processed using a deep learning model to obtain each detection point, and the corresponding position is found on the second image to be detected. Since the two images are offset only in the x-direction, the corresponding position is only found in the x-direction;

[0040] Determine a maximum and a minimum parallax between the first image to be detected and the second image to be detected in the image coordinate system according to a detection height range;

[0041] Searching for the coordinate position of the right camera corresponding to each detection point in the first image to be detected within a fixed x-direction range;

[0042] Using template matching, the image block with the highest matching degree between the second camera and the first camera is found within the range determined by the first camera. The position of the image block is the position of the detection point corresponding to the second camera.

[0043] The difference in the x direction between the detection point in the first image to be detected and the second image to be detected is the parallax;

[0044] Determine depth based on parallax, specifically:

[0045] According to the previous binocular camera stereo calibration, the focal length f of the camera and the baseline distance b between the binocular cameras are obtained. The depth can be calculated through the parallax d, and it satisfies: z = (f*b) / d.

[0046] This method aims to convert the plane coordinates of the original image into three-dimensional space coordinates. The traditional method only performs detection on a two-dimensional plane. However, the method in this case performs detection in three-dimensional coordinates and converts the detection of detection points into 3D coordinate detection, avoiding the two-dimensional detection of image pixel detection, which requires expensive hardware costs and is time-consuming and easily affected by the light in the environment. The detection of 3D coordinates can greatly improve the detection efficiency and save time.

[0047] S104: Obtain the coordinates of each detection point in the image coordinate system. The depth and the coordinates of each detection point are used to determine the coordinates of each detection point in the world coordinate system through the transformation matrix, which is the first coordinate. The image to be detected is processed by the deep learning image model to determine the parameters of the detection point to be detected. The parameters include the position, size and category of the detection part in the image coordinate system, which is:

[0048] Using the image object detection model in deep learning, the position, category and size of the detection points to be detected in the image captured by the binocular camera can be output.

[0049] S105: Obtain standard parameters of all detection points of the workpiece standard part in the world coordinates, and determine whether the workpiece is qualified according to the first coordinates of all detection points of the workpiece to be tested and the parameters of the detection points to be tested.

[0050] For example, obtain the standard 3D coordinates and standard categories of all inspection points of the workpiece standard part in the world coordinates;

[0051] Determine whether the first coordinates and the category to be measured of all monitoring points are the same as the standard 3D coordinates and standard category or meet the preset deviation. If so, the currently detected workpiece is qualified and a feedback signal is given. If not, the currently detected workpiece is defective and a feedback signal is given.

[0052] For example, if the detection point is a screw of a fixed model on the workpiece, it is determined whether the size of the screw is the same as that of the standard screw, or whether a screw is set at the corresponding position of the standard part.

[0053] Whether the position of the screw on the workpiece is correct can be determined by comparing the first coordinate corresponding to the screw with the standard coordinate.

[0054] As a specific implementation provided in this case, when multiple binocular cameras are used as described above, there will be overlapping areas in the image. For example, since the coordinates of the detection points captured by all cameras have been converted to the same coordinate system, the same world coordinate point corresponds to multiple overlapping monitoring points. It is necessary to merge the duplicate detection points and leave only one detection point. These detection points with the same world coordinates are the same detection point on the workpiece, which is only captured by multiple cameras and does not require parallel processing. Specifically:

[0055] Determine multiple overlapping regions of all binocular stereo cameras;

[0056] Determine multiple detection points in all overlapping areas that have the same coordinates in the world coordinate system, that is, multiple detection points overlap in the same world coordinate system;

[0057] The duplicate detection points are merged, leaving only one detection point.

[0058] The overall technical effects of the present invention are:

[0059] 1) When inspecting workpieces, no tooling is required; workpieces can be placed anywhere on the inspection table within a certain range. Therefore, a single inspection device can inspect a wide variety of workpieces. Furthermore, by adjusting the camera position and adding cameras, the system can accommodate inspection points at various angles and ranges. Therefore, adapting to different workpieces requires only changing the camera position and number, as well as the inspection model. Without the need for tooling or software changes, flexible manufacturing on the production line is possible.

[0060] 2) The image coordinates (two-dimensional coordinates) are converted into three-dimensional coordinates. By judging the position of the monitoring point coordinates, the detection of whether the monitoring point position is correctly set on the workpiece, as well as the detection of its model and size are completed.

[0061] The above is a detailed introduction to the product provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the core ideas of the present invention. It should be pointed out that, for those skilled in the art, without departing from the principles of the invention, several improvements and modifications can be made to the invention, and these improvements and modifications also fall within the scope of protection of the invention claims.

Claims

1. A method for detecting mis-assembly or missing assembly of a workpiece, suitable for detecting mis-assembly or missing assembly of workpieces on a production line, characterized in that: The method comprises: Using a binocular stereo camera to photograph the workpiece to obtain an image to be inspected, and calibrating the binocular stereo camera before photographing; Coordinate conversion between the binocular stereo camera and the detection plane, determining a conversion matrix of the binocular stereo camera, wherein the conversion matrix is ​​used to convert the coordinates of an image photographed by the binocular stereo camera into a world coordinate system with the detection plane as the coordinate system; Determining parameters of the detection points to be measured in the image to be detected, and determining the depth of each detection point according to the parameters of the detection points to be measured; the parameters of the detection points to be measured include at least the position, size and category of the detection points in the image coordinate system; Obtain the coordinates of each detection point in the image coordinate system, and determine the coordinates of each detection point in the world coordinate system through the transformation matrix, that is, the first coordinates; Obtaining the standard parameters of all detection points of the workpiece standard part in the world coordinate system, and determining whether the workpiece is qualified based on the first coordinates of all detection points of the workpiece to be tested and the parameters of the detection points to be tested; Among them, the image coordinate system of the binocular stereo camera is guaranteed to be one by verification, so that the image coordinates of all cameras are converted into the same world coordinate system with the detection plane as the coordinate system, thereby completing the coordinate system of all cameras.

2. The method according to claim 1, characterized in that The method for calibrating the binocular stereo camera includes: The binocular stereo camera is composed of two monocular industrial cameras of the same model arranged in parallel; The two monocular industrial cameras are fixed by an external mechanical device and the standard focal length is adjusted; Using a binocular stereo calibration method to perform calibration, and determining the internal parameters of the two monocular industrial cameras; The two internal parameters are corrected by a stereo correction matrix so that the images taken by the two monocular industrial cameras are completely aligned in the Y-axis direction.

3. The method according to claim 1, characterized in that The coordinate conversion between the binocular stereo camera and the detection plane, and the method for determining the conversion matrix of the binocular stereo camera include: When setting up a binocular stereo camera, the coordinate conversion relationship between the binocular stereo camera and the detection plane is calibrated to determine the conversion matrix of the binocular stereo camera.

4. The method according to claim 1, wherein The coordinate conversion between the binocular stereo camera and the detection plane, and the method for determining the conversion matrix of the binocular stereo camera include: A plurality of binocular stereo cameras are installed on the detection plane, and any one binocular stereo camera is selected as the main camera, and the rest are auxiliary cameras; Calibrate the transformation matrix between the secondary camera and the main camera to transform the image coordinates of the secondary camera into the world coordinate system with the detection plane on the detection platform as the coordinate system; Calibrate the coordinate transformation relationship between the main camera and the detection plane, and determine the transformation matrix of the main camera; The transformation matrix of the world coordinate system of the secondary camera with the detection plane as the coordinate system is determined by the transformation matrix between the main camera and the detection plane and the transformation matrix between the secondary camera and the main camera, and the coordinates of all secondary cameras in the world coordinate system are determined.

5. The method according to claim 4, characterized in that Determine multiple overlapping regions of all binocular stereo cameras; Determine a plurality of detection points in all the overlapping areas that have the same coordinates in the world coordinate system, that is, the plurality of detection points overlap in the same world coordinate system; The duplicate detection points are merged, leaving only one detection point.

6. The method according to claim 4 or 5, characterized in that The method for determining the parameters of the detection point to be detected in the image to be detected includes: The image to be detected is processed by a deep learning image model to determine the parameters of the detection points to be detected, wherein the parameters include the position, size and category of the detection part in the image coordinate system.

7. The method according to claim 4 or 5, wherein the binocular stereo camera comprises a first camera and a second camera, wherein: The method for determining the depth of each detection point according to the parameters of the detection point to be measured includes: Acquire a first image to be detected and a second image to be detected from the first camera and the second camera; The first camera and the second camera respectively perform image transformation, and the transformation results determine that the first image to be detected and the second image to be detected are images with translation only in the x-axis direction; The first image to be detected is processed using a deep learning model to obtain each detection point, and the corresponding position is found on the second image to be detected, and the corresponding position is only found in the x-direction; Determine a maximum and a minimum parallax between the first image to be detected and the second image to be detected in the image coordinate system according to a detection height range; Searching for the coordinate position of the right camera corresponding to each detection point in the first image to be detected within a fixed x-direction range; Searching for the image block with the greatest matching degree between the second camera and the first camera within the range determined by the first camera by template matching. The position of the image block is the position of the detection point corresponding to the second camera. The difference in the x direction between the detection point in the first image to be detected and the second image to be detected is the parallax; The depth is determined based on the disparity.

8. The method according to claim 1, characterized in that The method of obtaining the standard parameters of all detection points of a workpiece standard part in the world coordinate system and determining whether the workpiece is qualified according to the first coordinates of all detection points of the workpiece to be tested and the parameters of the detection points to be tested includes: Obtain the standard 3D coordinates and standard categories of all inspection points of the workpiece standard parts in the world coordinate system; Determine whether the first coordinates and the category to be tested of all detection points are the same as the standard 3D coordinates and the standard category or meet the preset deviation. If so, the currently detected workpiece is qualified and a feedback signal is given. If not, the currently detected workpiece is defective and a feedback signal is given.

Images