Full-size detection method based on multi-view vision, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of size measurement, and provides a full-size detection method based on multi-view vision, electronic equipment and a storage medium, and the method comprises the steps: collecting images of a to-be-detected workpiece through a camera cluster, constructing a target image set, and determining feature information, each measuring point on the to-be-measured workpiece is at least collected by two cameras in the camera cluster, and the feature information comprises a corresponding relationship between each measuring point and the camera and a pixel coordinate in a corresponding camera coordinate system; obtaining point cloud data of each measuring point and constructing a point cloud data set; constructing a three-dimensional coordinate system based on the target image set and the feature information, and optimizing internal parameters and / or external parameters of each camera based on the point cloud data set; determining three-dimensional coordinate information of each measuring point based on the optimized internal parameters and external parameters of each camera; based on the three-dimensional coordinate information and the point cloud data set of each measuring point, the size information of the to-be-measured workpiece is determined, and the accuracy of full-size detection of the workpiece is improved.

Description

Technical Field

[0001] This application belongs to the technical field of dimensional measurement, and in particular, relates to a full-dimensional detection method, an electronic device, and a storage medium based on multi-view vision. Background Art

[0002] In the current industrial manufacturing field, the demand for on-line full-dimensional detection of large-sized workpieces is increasing day by day. To meet this demand, vision-based detection technologies have been widely applied. Among them, the camera-based detection system is favored because of its non-contact, high-precision, high-efficiency and other advantages.

[0003] Currently, the commonly used method is to use a 2D camera cluster for three-dimensional reconstruction. The specific steps include: collecting images of the holes to be measured on the workpiece to be measured through a camera array, and determining the correspondence between the cameras in the camera array and the holes to be measured; obtaining the pixel coordinates of the hole centers by using methods such as edge detection or template matching; performing sparse three-dimensional reconstruction based on the Structure from Motion (SFM) method, and using Bundle Adjustment (BA) for optimization during the reconstruction process to improve the reconstruction accuracy; then, performing coordinate transformation by solving the Iterative Closest Point (ICP) algorithm to convert the reconstructed hole center coordinates to the workpiece coordinate system; finally, performing datum alignment, calculating the deviation between each point position and the theoretical corresponding point position of the workpiece, and outputting the deviation value as the measurement result. Although this method has high robustness and is not easily reconstructed to fail, errors will gradually accumulate during the reconstruction process, resulting in a drift problem in the final reconstruction result. This error accumulation not only affects the accuracy of three-dimensional reconstruction, but may also mislead subsequent measurement and analysis; in addition, due to the existence of algorithm errors such as camera imaging distortion, edge detection or template matching, the detected hole center coordinates often only have high repeatability accuracy, but may not be able to fully represent the real hole center coordinates in the absolute scale, which limits the accuracy and reliability of the detection results; moreover, there is a situation where the depth direction estimation accuracy of multi-view reconstruction is insufficient, resulting in poor three-dimensional reconstruction effect and low accuracy of the detection results.

[0004] Therefore, how to improve the accuracy of full-dimensional detection has become an urgent problem to be solved. Summary of the Invention

[0005] Embodiments of this application provide a full-dimensional detection method, an electronic device, and a storage medium based on multi-view vision, aiming to improve the accuracy of full-dimensional detection.

[0006] In a first aspect, an embodiment of the present application provides a full-size detection method based on multi-view vision. The method includes: collecting images of a workpiece to be measured through a camera cluster to construct a target image set, and determining feature information, where each measurement point on the workpiece to be measured is collected by at least two cameras in the camera cluster, and the feature information includes the correspondence between each measurement point and the camera and the pixel coordinates in the corresponding camera coordinate system; obtaining the point cloud data of each measurement point and constructing a point cloud data set; constructing a three-dimensional coordinate system based on the target image set and the feature information, and optimizing the internal parameters and / or external parameters of each camera based on the point cloud data set; determining the three-dimensional coordinate information of each measurement point based on the optimized internal parameters and external parameters of each camera; and determining the size information of the workpiece to be measured based on the three-dimensional coordinate information of each measurement point and the point cloud data set.

[0007] In a possible implementation manner, constructing a three-dimensional coordinate system based on the target image set and the feature information, and optimizing the internal parameters and / or external parameters of each camera based on the point cloud data set includes:

[0008] S31, selecting an initial image and a first adjacent image from the target image set;

[0009] S32, based on the initial image, the first adjacent image, and the feature information, determining the external parameters of the first camera, the external parameters of the second camera, the three-dimensional coordinate system, and the three-dimensional reconstruction coordinates of each common measurement point in the first common measurement point set. The first camera is the camera that collects the initial image, the second camera is the camera that collects the first adjacent image, and the first common measurement point set includes all common measurement points of the initial image and the first adjacent image;

[0010] S33, registering the three-dimensional coordinate system with the point cloud data set to determine the rigid body transformation parameters between the three-dimensional coordinate system and the workpiece coordinate system;

[0011] S34, based on the point cloud data set, the three-dimensional reconstruction coordinates of each common measurement point in the first common measurement point set, the rigid body transformation parameters, the feature information, the initial internal parameters and external parameters of the first camera, and the initial internal parameters and external parameters of the second camera, using a preset objective function to locally optimize the internal parameters and / or external parameters of the first camera and the second camera;

[0012] S35, selecting a second adjacent image of the initial image from the target image set, and determining the external parameters of the third camera and the three-dimensional reconstruction coordinates of each common measurement point in the second common measurement point set based on the feature information and the three-dimensional coordinate system. The second common measurement point set includes all common measurement points of the initial image and the second adjacent image, and the third camera is the camera that collects the second adjacent image;

[0013] S36. Based on the point cloud data set, the three-dimensional reconstruction coordinates of each common measurement point in the second common measurement point set, the rigid body transformation parameters, the internal and external parameters of the first camera, and the initial internal and external parameters of the third camera, use the preset objective function to locally optimize the internal and / or external parameters of the first camera and the third camera.

[0014] S37. Repeat steps S35 - S36 until all images in the target image set are traversed, to obtain the three-dimensional reconstruction coordinates of each measurement point and the internal and external parameters of each camera after local optimization.

[0015] In a possible implementation, after step S37, the method further includes:

[0016] S38. Based on the point cloud data set, the three-dimensional reconstruction coordinates of each measurement point, the rigid body transformation parameters, and the internal and external parameters of each camera after local optimization, use the preset objective function to globally optimize the internal and / or external parameters of each camera.

[0017] In a possible implementation, the preset objective function is:

[0018] min∑ i∈C ∑ j∈p α||K i *[R i │t i *[R m │t m X j -x ij || 2 +(1-α)||K i *[R i │t i *[R m │t m X 3d-j -x ij || 2 ,

[0019] where C is the index set of cameras or images, P is the index set of measurement points, K i is the internal parameter of the i-th camera, [R i │t i is the external parameter of the i-th camera, [R m │t m is the rigid body transformation parameter, Xj represents the three-dimensional reconstruction coordinate, xij represents the pixel coordinate, X 3d-j represents the point cloud data, α is the weight coefficient, 0 ≤ α ≤ 1.

[0020] In a possible implementation, selecting the initial image and the first adjacent image from the target image set includes: selecting two images with the most common measurement points from the target image set as the initial image and the first adjacent image respectively.

[0021] In a possible implementation, based on the initial image, the first adjacent image, and the feature information, determining the external parameters of the first camera, the external parameters of the second camera, the three-dimensional coordinate system, and the three-dimensional reconstruction coordinates of each common measurement point in the first set of common measurement points includes: determining the first camera and the second camera based on the corresponding relationship between each measurement point in the initial image, the first adjacent image, and the feature information and the camera; constructing the three-dimensional coordinate system based on the first camera and the second camera; determining the external parameters of the first camera and the external parameters of the second camera by solving the fundamental matrix and the essential matrix between the initial image and the first adjacent image; and determining the three-dimensional reconstruction coordinates of each common measurement point in the first set of common measurement points based on the triangulation reconstruction method.

[0022] In a possible implementation, based on the three-dimensional coordinate information of each measurement point and the point cloud data set, determining the size information of the workpiece to be measured includes: replacing the depth information in the three-dimensional coordinate information with the depth information of each measurement point in the point cloud data set to determine the target three-dimensional coordinate information of each measurement point; and determining the size information of the workpiece to be measured based on the target three-dimensional coordinate information of each measurement point.

[0023] In a possible implementation, obtaining the point cloud data of each measurement point and constructing a point cloud data set includes: using a line laser sensor or a structured light sensor to scan and sample the workpiece to be measured, and obtaining the point cloud data of each measurement point and constructing the point cloud data set.

[0024] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in the first aspect or any one of its implementation manners is implemented.

[0025] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method described in the first aspect or any one of its implementation manners is implemented.

[0026] Fourthly, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the method described in the first aspect or any one of its implementation manners are implemented.

[0027] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: An image of a workpiece to be measured is collected by a camera cluster to construct a target image set, and feature information is determined. When collecting the image, it is ensured that each measurement point on the workpiece to be measured is collected by at least two cameras in the camera cluster, which ensures the richness of the target image set and improves the accuracy of subsequent three-dimensional reconstruction and workpiece size detection; The point cloud data of each measurement point is obtained and a point cloud data set is constructed. A three-dimensional coordinate system is constructed based on the target image set and the feature information, and the internal parameters and / or external parameters of each camera are optimized based on the point cloud data set. Based on the optimized internal parameters and external parameters of each camera, the three-dimensional coordinate information of each measurement point is determined. Using the point cloud data of each measurement point to replace the value of the three-dimensional reconstruction for the optimization of the internal and external parameters of the camera can timely correct the three-dimensional reconstruction points with large errors caused by drift effects, improve the optimization effect of the internal and external parameters of the camera, and thus improve the accuracy of the three-dimensional coordinate information of each measurement point and the accuracy of subsequent workpiece size measurement; Then, based on the three-dimensional coordinate information of each measurement point and the point cloud data set, the size information of the workpiece to be measured is determined. When measuring the workpiece size based on the three-dimensional coordinate information of each measurement point, the point cloud data set is introduced, which effectively supplements the weak constraint in the optical axis direction of the camera and improves the accuracy in the depth direction during multi-view reconstruction, and thus improves the accuracy of workpiece size measurement.

[0028] It can be understood that the electronic device, computer-readable storage medium, and computer program product provided by the embodiments of the present application have the same beneficial effects as the above-mentioned full-size detection method based on multi-view vision, and will not be elaborated here. Description of the Drawings

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0030] Figure 1 It is a schematic flowchart of a full-size detection method based on multi-view vision provided by an embodiment of the present application;

[0031] Figure 2 It is a schematic diagram of multi-view vision sampling provided by an embodiment of the present application;

[0032] Figure 3Schematic diagram of 3D line laser sensor sampling provided by an embodiment of the present application;

[0033] Figure 4 Schematic diagram of reprojection residuals during BA optimization based on a preset objective function provided by an embodiment of the present application;

[0034] Figure 5 Schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0035] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0036] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0037] It should also be understood that the term "and / or" as used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0038] As used in the specification and appended claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0039] In addition, in the description of the specification and appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that specific features, structures, or characteristics described in connection with that embodiment are included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0041] In the current vision-based large-size online full-size inspection method and system, the system hardware solutions mainly include 3D cameras based on, such as structured light cameras or 3D laser sensors, and three-dimensional reconstruction based on a cluster of 2D cameras. However, the field of view of 3D cameras is usually small and the cost is high, making it unable to adapt to large-field measurements; while the cluster of 2D cameras has high precision and good flexibility, and can be dynamically arranged according to the actual situation, which is suitable for large-field measurements. However, the following problems exist in this scenario:

[0042] 1) The problem of error accumulation in the incremental SFM method: Most of the SFM methods used in the prior art are incremental. Although this method has high robustness and is not easily reconstructed unsuccessfully, errors will gradually accumulate during the reconstruction process, resulting in a drift problem in the final reconstruction result. This error accumulation not only affects the accuracy of three-dimensional reconstruction but may also mislead subsequent measurements and analyses;

[0043] 2) The absolute scale problem of hole center coordinates: Due to the existence of algorithm errors such as camera imaging distortion, edge detection, or template matching, the detected hole center coordinates often only have high repeatability accuracy, but may not be able to fully represent the true hole center coordinates in the absolute scale, which limits the accuracy and reliability of the detection results;

[0044] 3) Insufficient accuracy in depth direction estimation for multi-view reconstruction: The core of three-dimensional reconstruction is triangulation, and its accuracy is directly related to the baseline length (camera spacing) and parallax (pixel displacement of the same object at different perspectives). However, in multi-view reconstruction, the following factors affect the accuracy of depth estimation:

[0045] Sensitivity of parallax to depth: When the difference in the optical axis directions of the cameras is large, the component of the effective baseline in the depth direction may be small, resulting in insufficient parallax along the optical axis direction, making it difficult to accurately estimate the depth;

[0046] Relationship between the baseline direction and the optical axis: If the optical axes of the multi-camera tend to converge (such as in a circular arrangement), the baseline direction is close to perpendicular to the optical axis direction. At this time, the horizontal parallax is significant, but the parallax in the depth direction is small, resulting in an amplified error in depth estimation.

[0047] Matching ambiguity in the optical axis direction: In the optical axis direction, the appearance of the target object changes little at different viewing angles, resulting in an increased ambiguity in feature matching. A slight matching error will be amplified into a larger depth error along the optical axis direction.

[0048] To solve the above problems, the present application provides a full-size detection method based on multi-view vision. Images of the workpiece to be measured are collected by a camera cluster to construct a target image set, and feature information is determined. Each measurement point on the workpiece to be measured is collected by at least two cameras in the camera cluster. The feature information includes the corresponding relationship between each measurement point and the camera and the pixel coordinates in the corresponding camera coordinate system; the point cloud data of each measurement point is obtained and a point cloud data set is constructed; a three-dimensional coordinate system is constructed based on the target image set and the feature information, and the internal parameters and / or external parameters of each camera are optimized based on the point cloud data set; based on the optimized internal parameters and external parameters of each camera, the three-dimensional coordinate information of each measurement point is determined; based on the three-dimensional coordinate information of each measurement point and the point cloud data set, the size information of the workpiece to be measured is determined, improving the accuracy of the full-size detection of the workpiece.

[0049] For ease of understanding, the technical solution of the present application will be described in detail below with reference to the accompanying drawings.

[0050] Figure 1 The following is a schematic flow chart of a full-size detection method based on multi-view vision provided by an embodiment of the present application. For ease of description, only the parts related to this embodiment are shown. The method provided by this embodiment includes the following steps:

[0051] S1. Images of the workpiece to be measured are collected by a camera cluster to construct a target image set, and feature information is determined. Each measurement point on the workpiece to be measured is collected by at least two cameras in the camera cluster. The feature information includes the corresponding relationship between each measurement point and the camera and the pixel coordinates in the corresponding camera coordinate system.

[0052] In a possible implementation, as Figure 2 shown, each camera in the camera cluster is installed at a different position, and the pixel size and initial internal parameters of each camera are obtained in advance. Images of all measurement points on the workpiece to be measured are collected by each camera, and it is necessary to ensure that each measurement point is successfully captured by at least 2 cameras. All the collected images are constructed into a target image set. Based on the target image set, the pixel coordinates of each measurement point in the corresponding camera coordinate system are obtained by methods such as edge detection or template matching, and the corresponding relationship between each measurement point and the camera is recorded to obtain the feature information.

[0053] In specific implementation, the poses of the cameras in the camera cluster can be adjusted according to the actual situation of the workpiece to be measured, so as to ensure that each measurement point is successfully captured by at least 2 cameras.

[0054] In another possible implementation, a camera is driven to move by using tools such as a robotic arm, and the pose of the camera is continuously adjusted to collect images of all measurement points on the workpiece to be measured, and it is ensured that each measurement point exists in at least 2 images, so as to construct a target image set. Then, based on the target image set, methods such as edge detection or template matching are used to obtain the pixel coordinates of each measurement point in the corresponding camera coordinate system, and the corresponding relationship between each measurement point and different camera positions is recorded to obtain feature information.

[0055] S2. Obtain the point cloud data of each measurement point and construct a point cloud data set.

[0056] In one possible implementation, a line laser sensor or a structured light sensor is used to scan and sample the workpiece to be measured, and the point cloud data of each measurement point on the workpiece to be measured is obtained and a point cloud data set is constructed. Figure 3 It is a schematic diagram of using a 3D line laser sensor to collect the point cloud data of the workpiece to be measured.

[0057] In addition, the point cloud data of each measurement point can also be obtained through other similar devices such as a coordinate measuring machine, but this method cannot perform real-time calibration, that is, the sampling of the coordinate measuring machine is an independent process and requires a large amount of time to be spent separately.

[0058] S3. Based on the target image set and the feature information, construct a three-dimensional coordinate system, and optimize the internal parameters and / or external parameters of each camera based on the point cloud data set.

[0059] Preferably, step S3 may optionally but not limited to include:

[0060] S31. Select an initial image and a first adjacent image from the target image set.

[0061] In specific implementation, two images with the most common measurement points in the target image set are respectively selected as the initial image and the first adjacent image. Usually, the number of common measurement points between the initial image and the first adjacent image is not less than 8 pairs.

[0062] S32. Based on the initial image, the first adjacent image and the feature information, determine the external parameters of the first camera, the external parameters of the second camera, the three-dimensional coordinate system, and the three-dimensional reconstruction coordinates of each common measurement point in the first common measurement point set. The first camera is the camera that collects the initial image, the second camera is the camera that collects the first adjacent image, and the first common measurement point set includes all the common measurement points of the initial image and the first adjacent image.

[0063] In a possible implementation manner, based on the corresponding relationships between the measuring points in the initial image, the first adjacent image, and the feature information and the camera, the first camera and the second camera are determined; a three-dimensional coordinate system is constructed based on the first camera and the second camera; by solving the fundamental matrix and the essential matrix between the initial image and the first adjacent image, the external parameters of the first camera and the external parameters of the second camera are determined; based on the triangulation reconstruction method, the three-dimensional reconstruction coordinates of each common measuring point in the first set of common measuring points are determined.

[0064] In a specific implementation, the first set of common measuring points is determined from the initial image and the first adjacent image. Combining the corresponding relationships between the measuring points in the feature information and the camera, the first camera and the second camera are determined. Based on the initial internal parameters of the first camera and the second camera, by solving the fundamental matrix and the essential matrix between the initial image and the first adjacent image, the external parameters of the first camera and the second camera are respectively determined. When solving the fundamental matrix, the ransac method can be used if necessary to exclude the interference of outliers; and based on the initial internal parameters and external parameters of the first camera and the initial internal parameters and external parameters of the second camera, the triangulation reconstruction method is used to reconstruct the three-dimensional reconstruction coordinates of each common measuring point in the first set of common measuring points. At this time, the three-dimensional coordinate system based on the initial view and the first adjacent view is successfully constructed.

[0065] S33, register the three-dimensional coordinate system with the point cloud data set to determine the rigid body transformation parameters between the three-dimensional coordinate system and the workpiece coordinate system.

[0066] S34, based on the point cloud data set, the three-dimensional reconstruction coordinates of each common measuring point in the first set of common measuring points, the rigid body transformation parameters, the feature information, and the initial internal parameters and external parameters of the first camera and the initial internal parameters and external parameters of the second camera, use a preset objective function to locally optimize the internal parameters and / or external parameters of the first camera and the second camera.

[0067] In a specific implementation, based on the point cloud data set, the three-dimensional reconstruction coordinates of each common measuring point in the first set of common measuring points, the rigid body transformation parameters, the feature information, and the initial internal parameters and external parameters of the first camera and the initial internal parameters and external parameters of the second camera, local BA is performed by minimizing the preset objective function using the Levenberg-Marquardt method to locally optimize the internal parameters and / or external parameters of the first camera and the second camera.

[0068] In a possible implementation manner, the preset objective function is:

[0069] min∑ i∈C ∑ j∈p α||K i *[R i │t i *[R m │t m Xj -x ij || 2 +(1 - α)||K i *[R i │t i *[R m │t m X 3d-j -x ij || 2 ,

[0070] Among them, C is the index set of cameras or images, P is the index set of measurement points, K i is the internal parameter of the i-th camera, [R i │t i is the external parameter of the i-th camera, [R m │t m is the rigid body transformation parameter, Xj represents the 3D reconstruction coordinates, xij represents the pixel coordinates, X 3d-j represents the point cloud data, {(Xj, xij)} is the set of 2D-3D correspondences, that is, the correspondence set between pixel coordinates and 3D reconstruction coordinates, and α is the weight coefficient used to determine the weight ratio of each part in the alternating optimization process, where 0 ≤ α ≤ 1.

[0071] Specifically, when performing step S34 according to the above preset objective function, C is the index set of the first camera and the second camera, and P is the index set of the first common measurement point set.

[0072] As an example, Figure 4 is a schematic diagram of the reprojection error during BA optimization based on the preset objective function. Among them, π k (X j ) and π i (X j ) are the reprojection errors of the measurement point j under different cameras respectively, and π = K i *[R i │t i *[R m │t m X j -x ij 。

[0073] It should be noted that during the optimization process, the internal parameters and external parameters can be optimized simultaneously according to the actual situation, or only the external parameters or internal parameters can be optimized. In this case, only the values of the corresponding parameters in the solver need to be fixed, and the optimization logic is essentially the same.

[0074] S35. Select the second adjacent image of the initial image from the target image set, and determine the external parameters of the third camera and the three-dimensional reconstruction coordinates of each common measurement point in the second common measurement point set based on the feature information and the three-dimensional coordinate system. The second common measurement point set includes all the common measurement points of the initial image and the second adjacent image, and the third camera is the camera that captures the second adjacent image.

[0075] In a specific implementation, select any image adjacent to the initial image from the target image set as the second adjacent image, determine the third camera through the corresponding relationship between each measurement point in the second adjacent image and the feature information and the camera, calculate the external parameters of the third camera in the three-dimensional coordinate system through the perspective-n-points (pnp) algorithm, and reconstruct the three-dimensional reconstruction coordinates of each common measurement point in the second common measurement point set based on the triangulation reconstruction method.

[0076] As an example, if there are also common measurement points between the first adjacent image and the second adjacent image, reconstruct the three-dimensional reconstruction coordinates of each common measurement point between the first adjacent image and the second adjacent image based on the triangulation reconstruction method.

[0077] S36. Based on the point cloud data set, the three-dimensional reconstruction coordinates of each common measurement point in the second common measurement point set, the rigid body change parameters, the internal and external parameters of the first camera, and the initial internal and external parameters of the third camera, use a preset objective function to locally optimize the internal and / or external parameters of the first camera and the third camera.

[0078] In a specific implementation, based on the point cloud data set, the three-dimensional reconstruction coordinates of each common measurement point in the second common measurement point set, the rigid body change parameters, the internal and external parameters of the first camera, and the initial internal and external parameters of the third camera, perform local BA by minimizing the preset objective function through the Levenberg-Marquardt method to locally optimize the internal and / or external parameters of the first camera and the third camera. During the optimization process, C in the preset objective function is the index set of the first camera and the third camera, and P is the index set of the second common measurement point set.

[0079] As an example, if there are common measurement points in the initial image, the first adjacent image, and the second adjacent image, based on the point cloud data set, the three-dimensional reconstruction coordinates of the common measurement points in the initial image, the first adjacent image, and the second adjacent image, the rigid body change parameters, the internal and external parameters of the first camera, the internal and external parameters of the second camera, and the initial internal and external parameters of the third camera, perform local BA by minimizing the preset objective function through the Levenberg-Marquardt method to locally optimize the internal and / or external parameters of the first camera, the second camera, and the third camera. During the optimization process, C in the preset objective function is the index set of the first camera, the second camera, and the third camera, and P is the index set of the common measurement points in the initial image, the first adjacent image, and the second adjacent image.

[0080] It should be noted that during the optimization process, the internal parameters and external parameters can be optimized according to the actual situation, or only the external parameters or internal parameters can be optimized. In this case, only the corresponding parameter values in the solver need to be fixed, and the optimization logic is essentially the same.

[0081] S37. Repeat steps S35 - S36 until all images in the target image set are traversed, and the three-dimensional reconstruction coordinates of each measurement point and the internal and external parameters of each camera after local optimization are obtained.

[0082] In a specific implementation, traverse the target image set, replace the three-dimensional reconstruction coordinates with point cloud data, perform alternating BA optimization based on a preset objective function, obtain the internal and external parameters of each camera in the camera cluster after local optimization, and then calculate the three-dimensional reconstruction coordinates of each measurement point using a triangulation reconstruction method based on the internal and external parameters of each camera after local optimization.

[0083] Furthermore, after step S37, it further includes:

[0084] S38. Based on the point cloud data set, the three-dimensional reconstruction coordinates of each measurement point, the rigid body transformation parameters, and the internal and external parameters of each camera after local optimization, use a preset objective function to globally optimize the internal and / or external parameters of each camera.

[0085] In a specific implementation, after completing the local optimization of the internal and external parameters of each camera, based on the point cloud data set, the three-dimensional reconstruction coordinates of each measurement point, the rigid body transformation parameters, and the internal and external parameters of each camera after local optimization, perform global BA optimization by minimizing the preset objective function using the Levenberg - Marquardt method to globally optimize the internal and / or external parameters of each camera.

[0086] Specifically, during the optimization process, C in the preset objective function is the index set of all cameras in the camera cluster, and P is the index set of all measurement points on the workpiece to be measured.

[0087] It should be noted that during the optimization process, the internal parameters and external parameters can be optimized according to the actual situation, or only the external parameters or internal parameters can be optimized. In this case, only the corresponding parameter values in the solver need to be fixed, and the optimization logic is essentially the same.

[0088] S4. Based on the internal and external parameters of each camera, determine the three-dimensional coordinate information of each measurement point.

[0089] In a specific implementation, after locally and globally optimizing the internal and external parameters of each camera based on the preset objective function, use the finally obtained internal and external parameters of each camera, and use the triangulation reconstruction method to calculate the three-dimensional reconstruction coordinates of each measurement point again and determine the three-dimensional coordinate information of each measurement point, improving the accuracy of three-dimensional reconstruction.

[0090] S5. Determine the dimensional information of the workpiece to be measured based on the three-dimensional coordinate information and the point cloud data set of each measurement point.

[0091] In a possible implementation, replace the depth information in the three-dimensional coordinate information with the depth information of each measurement point in the point cloud data set to determine the target three-dimensional coordinate information of each measurement point; determine the dimensional information of the workpiece to be measured based on the target three-dimensional coordinate information of each measurement point.

[0092] In another possible implementation, directly use the three-dimensional coordinate information and the rigid body transformation parameters of each measurement point obtained in step S4 to determine the dimensional information of the workpiece to be measured.

[0093] In specific implementation, on the xy plane of the workpiece coordinate system, when calculating the position degree, the value in the z direction is often not required. At this time, z can be not considered during the BA optimization solution process. Similarly, x can be not considered on the yz plane, and y can be not considered on the xz plane; when the values in the x, y, and z directions need to be provided, the three-dimensional reconstruction information can be used to provide the values in the other two directions except the depth direction, and the point cloud data is used to provide the value in the depth direction, effectively solving the problem of insufficient accuracy in estimating the depth direction during existing multi-view reconstruction.

[0094] The technical solution provided by this application constructs a target image set by collecting images of the workpiece to be measured through a camera cluster and determines the feature information. When collecting images, ensure that each measurement point on the workpiece to be measured is collected by at least two cameras in the camera cluster, which ensures the richness of the target image set and improves the accuracy of subsequent three-dimensional reconstruction and workpiece size detection; obtain the point cloud data of each measurement point and construct a point cloud data set, construct a three-dimensional coordinate system based on the target image set and the feature information, and optimize the internal parameters and / or external parameters of each camera based on the point cloud data set. Based on the optimized internal and external parameters of each camera, determine the three-dimensional coordinate information of each measurement point. Use the point cloud data of each measurement point to replace the values of the three-dimensional reconstruction for optimizing the internal and external parameters of the camera, which can timely correct the three-dimensional reconstruction points with large errors caused by drift effects, improve the optimization effect of the internal and external parameters of the camera, and further improve the accuracy of the three-dimensional coordinate information of each measurement point and the accuracy of subsequent workpiece size measurement; then determine the dimensional information of the workpiece to be measured based on the three-dimensional coordinate information and the point cloud data set of each measurement point. When measuring the workpiece size based on the three-dimensional coordinate information of each measurement point, the point cloud data set is introduced, effectively supplementing the weak constraint in the camera optical axis direction, improving the accuracy in the depth direction during multi-view reconstruction, and further improving the accuracy of workpiece size measurement.

[0095] In summary, the technical solution provided by this application has the following beneficial effects:

[0096] 1) In the traditional incremental SFM method, although robust 3D reconstruction results can be obtained, when a new view is added and local BA is performed each time, only the influence of several adjacent views is considered, that is, the local optimization is accurate, but the global may not be accurate, and the drift error will accumulate, which will cause a large deviation (drift error) between the subsequent reconstructed 3D points and the real points; by using the sampling values of a 3D line laser sensor or the like to replace the 3D reconstruction values for reprojection and performing local alternating BA view by view, it is equivalent to gradually introducing global information, which can timely correct the 3D reconstruction points with large errors caused by drift influence.

[0097] 2) Due to problems such as camera imaging distortion, algorithm errors in edge detection or template matching, etc., the detected hole center coordinates often only have high repeatability accuracy and may not fully represent the real hole center coordinates in absolute scale. At this time, by performing local BA optimization with the sampling values of a 3D line laser sensor or the like, it is equivalent to performing an overfitting operation with the real and specific measurement scenario, which is beneficial to maintaining the consistency and robustness of the measurement results.

[0098] Figure 5 The following is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 5 shown, the electronic device 5 of this embodiment includes: at least one processor 50 ( Figure 5 only one is shown in the figure), a memory 51, and a computer program 52 stored in the memory 51 and executable on at least one processor 50. When the processor 50 executes the computer program 52, it implements the steps in the above Figure 1 method embodiment.

[0099] The electronic device 5 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The electronic device 5 may include but is not limited to the processor 50 and the memory 51. Those skilled in the art can understand that Figure 5 this is only an example of the electronic device 5 and does not constitute a limitation on the electronic device 5. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0100] The processor 50 may be a Central Processing Unit (CPU), and the processor 50 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0101] The memory 51 may be an internal storage unit of the electronic device 5 in some embodiments, such as the hard disk or memory of the electronic device 5. The memory 51 may also be an external storage device of the electronic device 5 in other embodiments, such as a plug-in hard disk equipped on the electronic device 5, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 51 may also include both the internal storage unit and the external storage device of the electronic device 5. The memory 51 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of a computer program. The memory 51 may also be used to temporarily store data that has been output or will be output.

[0102] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments can be implemented.

[0103] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunications signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc.

[0104] A computer-readable storage medium provided by an embodiment of this application has the same beneficial effects as the above-described full-size detection method based on multi-view vision.

[0105] An embodiment of this application provides a computer program product. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above-described method embodiments can be implemented.

[0106] A computer program product provided by an embodiment of this application has the same beneficial effects as the above-described full-size detection method based on multi-view vision.

[0107] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0109] In the embodiments provided in the present application, it should be understood that the disclosed device / equipment and method can be implemented in other ways. For example, the device / equipment embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0110] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A full-size detection method based on multi-view vision, characterized in that, The method includes: Collecting images of the workpiece to be measured through a camera cluster to construct a target image set, and determining feature information. Each measurement point on the workpiece to be measured is collected by at least two cameras in the camera cluster. The feature information includes the correspondence between each measurement point and the camera and the pixel coordinates in the corresponding camera coordinate system; Obtaining the point cloud data of each measurement point and constructing a point cloud data set; Constructing a three-dimensional coordinate system based on the target image set and the feature information, and optimizing the internal parameters and / or external parameters of each camera based on the point cloud data set; Determining the three-dimensional coordinate information of each measurement point based on the optimized internal parameters and external parameters of each camera; Determining the dimensional information of the workpiece to be measured based on the three-dimensional coordinate information of each measurement point and the point cloud data set.

2. The method according to claim 1, wherein The constructing a three-dimensional coordinate system based on the target image set and the feature information, and optimizing the internal parameters and / or external parameters of each camera based on the point cloud data set includes: S31, selecting an initial image and a first adjacent image from the target image set; S32, determining the external parameter of the first camera, the external parameter of the second camera, the three-dimensional coordinate system, and the three-dimensional reconstruction coordinates of each common measurement point in the first common measurement point set based on the initial image, the first adjacent image, and the feature information. The first camera is the camera that collects the initial image, the second camera is the camera that collects the first adjacent image, and the first common measurement point set includes all common measurement points of the initial image and the first adjacent image; S33, registering the three-dimensional coordinate system with the point cloud data set to determine the rigid body transformation parameters between the three-dimensional coordinate system and the workpiece coordinate system; S34, using a preset objective function to locally optimize the internal parameters and / or external parameters of the first camera and the second camera based on the point cloud data set, the three-dimensional reconstruction coordinates of each common measurement point in the first common measurement point set, the rigid body transformation parameters, the feature information, the initial internal parameters and external parameters of the first camera, and the initial internal parameters and external parameters of the second camera; S35, selecting a second adjacent image of the initial image from the target image set, and determining the external parameter of the third camera and the three-dimensional reconstruction coordinates of each common measurement point in the second common measurement point set based on the feature information and the three-dimensional coordinate system. The second common measurement point set includes all common measurement points of the initial image and the second adjacent image, and the third camera is the camera that collects the second adjacent image; S36, using the preset objective function to locally optimize the internal parameters and / or external parameters of the first camera and the third camera based on the point cloud data set, the three-dimensional reconstruction coordinates of each common measurement point in the second common measurement point set, the rigid body transformation parameters, the internal parameters and external parameters of the first camera, and the initial internal parameters and external parameters of the third camera; S37, repeatedly executing steps S35 - S36 until all images in the target image set are traversed, obtaining the three-dimensional reconstruction coordinates of each measurement point and the internally and externally optimized parameters of each camera after local optimization.

3. The method according to claim 2, wherein After the step S37, the method further includes: S38. Based on the point cloud data set, the three-dimensional reconstruction coordinates of each measurement point, the rigid body change parameters, and the internal and external parameters of each camera after local optimization, the internal and / or external parameters of each camera are globally optimized by using the preset objective function.

4. The method according to claim 3, characterized in that, The preset objective function is: min∑ i∈C ∑ j∈p α||K i *[R i │t i *[R m │t m X j -x ij || 2 +(1-α)||K i *[R i │t i *[R m │t m X 3d-j -x ij || 2 , where C is the index set of cameras or images, P is the index set of measurement points, K i is the internal parameter of the i-th camera, [R i │t i is the external parameter of the i-th camera, [R m │t m is the rigid body change parameter, Xj represents the 3D reconstruction coordinates, xij represents the pixel coordinates, X 3d-j represents the point cloud data, and α is the weight coefficient, 0 ≤ α ≤ 1.

5. The method according to claim 2, characterized in that, The selection of the initial image and the first adjacent image from the target image set includes: Selecting two images with the most common measurement points from the target image set as the initial image and the first adjacent image respectively.

6. The method according to claim 2, wherein The determination of the external parameter of the first camera, the external parameter of the second camera, the three-dimensional coordinate system, and the three-dimensional reconstruction coordinates of each common measurement point in the first common measurement point set based on the initial image, the first adjacent image, and the feature information includes: Determining the first camera and the second camera based on the correspondence between each measurement point and the camera in the initial image, the first adjacent image, and the feature information; Constructing the three-dimensional coordinate system based on the first camera and the second camera; Determining the external parameter of the first camera and the external parameter of the second camera by solving the fundamental matrix and the essential matrix between the initial image and the first adjacent image; Determining the three-dimensional reconstruction coordinates of each common measurement point in the first common measurement point set based on the triangulation reconstruction method.

7. The method according to claim 1, wherein The determination of the size information of the workpiece to be measured based on the three-dimensional coordinate information of each measurement point and the point cloud data set includes: Replacing the depth information in the three-dimensional coordinate information with the depth information of each measurement point in the point cloud data set to determine the target three-dimensional coordinate information of each measurement point; Determining the size information of the workpiece to be measured based on the target three-dimensional coordinate information of each measurement point.

8. The method according to any one of claims 1 to 7, characterized in that, The acquisition of the point cloud data of each measurement point and the construction of the point cloud data set include: Scanning and sampling the workpiece to be measured by using a line laser sensor or a structured light sensor to acquire the point cloud data of each measurement point and construct the point cloud data set.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1 to 8 is implemented.

Images