Integrated methods, systems, electronic devices, and storage media for camera calibration and testing

Abstract

This application relates to the field of machine vision technology and discloses a method, system, electronic device, and storage medium for integrated camera calibration and production testing. The method includes: determining second feature information of the test board in the camera coordinate system based on a real speckle image captured by the target camera on a preset test board and first feature information of the test board in the world coordinate system; generating several candidate speckle images based on the second feature information and several different assembly errors, and calculating the matching cost between each candidate speckle image and the real speckle image; designating the candidate speckle image with the smallest matching cost as the reference speckle image of the target camera, and performing production testing on the target camera based on the real speckle image and the reference speckle image. This integrates camera calibration and production testing, reducing the cost of camera calibration and production testing while greatly improving the efficiency of camera calibration and production testing.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a camera calibration and measurement integrated method, system, electronic device and storage medium. Background Technology

[0002] Depth perception is arguably the most dynamic branch of machine vision technology, and speckle structured light technology is a crucial component of it. As the most common active stereo vision technology, speckle structured light has wide applications in facial recognition, autonomous driving, and security monitoring. A speckle structured light camera consists of a speckle projector, an infrared sensor, and other components. The speckle projector projects pseudo-random specks onto the object, the infrared sensor captures the scene image, and the processing chip matches the scene image with a pre-stored reference image to obtain disparity information. Then, depth calculation is performed to acquire the scene's depth information.

[0003] As precision devices, speckle structured light cameras require a series of processes before leaving the factory, including calibration and production testing. Calibration typically includes camera parameter calibration and reference chart calibration. After calibration, production testing is necessary to ensure the camera functions correctly. However, for certain applications, existing calibration and production testing processes contain steps that are unavoidable but practically unnecessary, resulting in low calibration and production testing efficiency and high costs. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, electronic device, and storage medium for integrated camera calibration and production testing, which realizes the integration of camera calibration and production testing, and greatly improves the efficiency of camera calibration and production testing while reducing the cost of camera calibration and production testing.

[0005] To address the aforementioned technical problems, embodiments of this application provide an integrated calibration and production testing method for a camera, comprising the following steps: determining second feature information of the test board in the camera coordinate system based on a real speckle pattern captured by a target camera on a preset test board and first feature information of the test board in the world coordinate system; generating several candidate speckle patterns based on the second feature information and several different assembly errors, and calculating the matching cost between each candidate speckle pattern and the real speckle pattern; designating the candidate speckle pattern with the smallest matching cost as the reference speckle pattern of the target camera, and performing production testing on the target camera based on the real speckle pattern and the reference speckle pattern.

[0006] Embodiments of this application also provide an integrated calibration and production testing system for a camera, comprising: an acquisition module, a transformation module, a matching module, a calibration module, and a production testing module; the acquisition module is used to acquire a real speckle pattern obtained by a target camera capturing a preset test board; the transformation module is used to determine a second feature information of the test board in the camera coordinate system based on the real speckle pattern and a first feature information of the test board in the world coordinate system; the matching module is used to generate a plurality of candidate speckle patterns based on the second feature information and a plurality of different assembly errors, and calculate the matching cost between each candidate speckle pattern and the real speckle pattern; the calibration module is used to designate the candidate speckle pattern with the smallest matching cost as the reference speckle pattern of the target camera; the production testing module is used to perform production testing on the target camera based on the real speckle pattern and the reference speckle pattern.

[0007] Embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described integrated calibration and testing method.

[0008] Embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described integrated calibration and testing method.

[0009] The camera calibration and production testing integrated method, system, electronic device, and storage medium provided in the embodiments of this application first control the target camera to take pictures of a preset test board to obtain a real speckle pattern. Based on the real speckle pattern and the first feature information of the test board in the world coordinate system, the second feature information of the test board in the camera coordinate system of the target camera is determined. Then, based on the second feature information and several different assembly errors, several candidate speckle patterns are generated, and the matching cost between each candidate speckle pattern and the real speckle pattern is calculated. The candidate speckle pattern with the smallest matching cost is then designated as the reference speckle pattern of the target camera. Finally, the target camera is subjected to production testing based on the real speckle pattern and the reference speckle pattern. Considering that for some application scenarios, calibration and production testing involve steps that are unavoidable but not actually needed, resulting in low calibration efficiency and high production testing costs, the embodiments of this application combine the calibration and production testing processes into one. Both processes are based on a preset test board and can be completed using a single workstation, achieving integrated camera calibration and production testing. This saves the production line the cost of one workstation. At the same time, the calibration process does not require additional equipment to calculate assembly errors; instead, it determines assembly errors through simulation matching, greatly improving the efficiency of camera calibration and production testing.

[0010] Furthermore, the first feature information is the shape function of the test board in the world coordinate system, and the second feature information is the shape function of the test board in the camera coordinate system. The step of determining the second feature information of the test board in the camera coordinate system based on the real speckle image obtained by the target camera from the preset test board and the first feature information of the test board in the world coordinate system includes: determining the points corresponding to the feature points of the test board in the real speckle image and obtaining the coordinates of the feature points in the image coordinate system; determining the coordinates of the feature points in the camera coordinate system based on the coordinates of the feature points in the world coordinate system and the coordinates of the feature points in the image coordinate system; determining the rigid transformation relationship between the camera coordinate system and the world coordinate system based on a preset cost function, the coordinates of the feature points in the world coordinate system, and the coordinates of the feature points in the camera coordinate system; and determining the shape function of the test board in the camera coordinate system based on the rigid transformation relationship and the shape function of the test board in the world coordinate system. The shape function of the test board in the world coordinate system is known inherent information about the test board. However, the camera coordinate system, established based on the camera's optical axis, is not entirely consistent with the world coordinate system; there are certain differences. This difference represents the rigid transformation relationship between the two coordinate systems. The test board contains many feature points that are easily found in real speckle maps. By using the coordinates of these feature points in the image coordinate system, their coordinates in the camera coordinate system can be obtained. Furthermore, the rigid transformation relationship between the two coordinate systems can be quickly and accurately determined using the cost function.

[0011] In addition, the number of feature points is at least five. Determining the coordinates of the feature points in the camera coordinate system based on their coordinates in the world coordinate system and their coordinates in the image coordinate system includes: determining the actual distance between each pair of feature points based on their coordinates in the world coordinate system; traversing each actual distance and establishing an equation corresponding to the current actual distance based on the current actual distance and the coordinates of the two feature points corresponding to the current actual distance in the image coordinate system; wherein the unknowns in the equations include the focal length of the target camera, the baseline distance, and the vertical coordinates of the two feature points corresponding to the current actual distance in the camera coordinate system; and simultaneously solving the equations established based on each actual distance to obtain a system of equations, thereby solving for the focal length of the target camera, the baseline distance, and the vertical coordinates of each feature point in the camera coordinate system. Regardless of whether it is in the world coordinate system or the camera coordinate system, the actual distance between two feature points is fixed and will not change with rigid transformation. Therefore, this application constructs a system of equations based on the actual distance. When the number of feature points is at least five, ten equations can be established. At this time, there are nine unknowns, which can be solved to accurately determine the coordinates of each feature point in the camera coordinate system.

[0012] Furthermore, the equation corresponding to the current actual distance is established based on the current actual distance and the coordinates of the two feature points corresponding to the current actual distance in the image coordinate system, which is achieved through the following formula:

[0013]

[0014] Wherein, d is the current actual distance, (u1, v1) is the coordinate of the first feature point in the image coordinate system, (u2, v2) is the coordinate of the second feature point in the image coordinate system, (fx, fy) is the focal length, (cx, cy) is the baseline distance, c1 is the vertical coordinate of the first feature point in the camera coordinate system, and c2 is the vertical coordinate of the second feature point in the camera coordinate system.

[0015] Furthermore, the rigid transformation relationship between the camera coordinate system and the world coordinate system is determined based on a preset cost function, the coordinates of the feature point in the world coordinate system, and the coordinates of the feature point in the camera coordinate system. This is achieved through the following formula:

[0016]

[0017] Where cost represents the preset cost function, (a i ,b i ,c i (x) represents the coordinates of the feature point in the camera coordinate system. i ,y i ,z i ) represents the coordinates of the feature point in the world coordinate system, and [R,t] represents the rigid transformation relationship between the camera coordinate system and the world coordinate system.

[0018] Furthermore, the calculation of the matching cost between each candidate speckle pattern and the real speckle pattern includes: traversing each candidate speckle pattern, arbitrarily selecting a reference speckle and a reference diffraction order in the real speckle pattern, and determining the coordinates of the speckle corresponding to the reference speckle in the current candidate speckle pattern; calculating the matching cost between the current candidate speckle pattern and the real speckle pattern based on the coordinates of the reference speckle, the coordinates of the speckle corresponding to the reference speckle in the current candidate speckle pattern, the boundary coordinate cluster of the reference diffraction order in the real speckle pattern, the boundary coordinate cluster of the reference diffraction order in the current candidate speckle pattern, and a preset bias coefficient. The matching cost considers the coordinate error between speckle patterns of the same name and the matching degree between orders, while the bias coefficient is used to adjust the importance of tilt angle and rotation angle in assembly errors, thus accurately calculating a matching cost that meets actual needs.

[0019] Furthermore, the matching cost between the current candidate speckle map and the real speckle map is calculated based on the coordinates of the reference speckle, the coordinates of the corresponding speckle in the current candidate speckle map, the boundary coordinate cluster of the reference diffraction order in the real speckle map, the boundary coordinate cluster of the reference diffraction order in the current candidate speckle map, and a preset bias coefficient, using the following formula:

[0020] L=∑||(p s ,q s )-(k s ,h s )||+λ∑||m w -b w ||

[0021] Where L is the matching cost, (k s ,h s (p) represents the coordinates of the reference speckle, (p) s ,q s ) represents the coordinates of the reference speckle in the current candidate speckle map, n w The m represents the boundary coordinate family of the reference diffraction order in the true speckle pattern. w The reference diffraction order represents the boundary coordinate cluster in the current candidate speckle map, and λ is the preset bias coefficient.

[0022] In addition, the assembly error includes the tilt angle and rotation angle between the VCSEL and DOE in the speckle projector of the camera. The several different assembly errors are obtained through the following steps: generating several different tilt angles based on a first preset step size, the largest tilt angle being less than a first preset threshold; generating several different rotation angles based on a second preset step size, the largest rotation angle being less than a second preset threshold; and arbitrarily combining the several different tilt angles and the several different rotation angles to obtain several different assembly errors.

[0023] Furthermore, the step of generating several candidate speckle maps based on the second feature information and several different assembly errors includes: traversing the several different assembly errors; establishing a first mapping relationship between the VCSEL coordinate system and the camera coordinate system based on the current assembly error, the shape function of the test board in the camera coordinate system, and the pose relationship between the speckle projector and the infrared lens; establishing a second mapping relationship between the camera coordinate system and the image coordinate system based on the shape function of the test board in the camera coordinate system; mapping each transmitter on the VCSEL to diffraction speckle in the image coordinate system according to the first mapping relationship and the second mapping relationship; and generating a candidate speckle map corresponding to the current assembly error based on each diffraction speckle. Attached Figure Description

[0024] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0025] Figure 1 This is a flowchart of a camera calibration and measurement integrated method provided in one embodiment of this application;

[0026] Figure 2 This is a schematic diagram of a fusion workstation provided in one embodiment of this application;

[0027] Figure 3 This is a schematic diagram of a test board provided in one embodiment of this application;

[0028] Figure 4 This is a flowchart of an embodiment of the present application, which describes the process of determining the second feature information of the test board in the camera coordinate system based on the real speckle pattern obtained by the target camera from the preset test board and the first feature information of the test board in the world coordinate system.

[0029] Figure 5 This is a flowchart of an embodiment of the present application, which describes how to determine the coordinates of a feature point in the camera coordinate system based on the coordinates of the feature point in the world coordinate system and the coordinates of the feature point in the image coordinate system.

[0030] Figure 6 This is a flowchart illustrating, in one embodiment of the present application, the matching cost between each candidate speckle image and the real speckle image.

[0031] Figure 7 This is a flowchart of an embodiment of the present application, illustrating the generation of several candidate speckle patterns based on second feature information and several different assembly errors;

[0032] Figure 8 This is a schematic diagram of a camera calibration and measurement integrated system provided in another embodiment of this application;

[0033] Figure 9 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0035] One embodiment of this application relates to an integrated calibration and testing method for cameras, applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using a server as an example. The implementation details of the integrated calibration and testing method for cameras in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0036] The specific process of the camera calibration and production testing integration method in this embodiment can be described as follows: Figure 1 As shown, it includes:

[0037] Step 101: Based on the real speckle pattern obtained by the target camera from the preset test board and the first feature information of the test board in the world coordinate system, determine the second feature information of the test board in the camera coordinate system.

[0038] In its implementation, unlike traditional calibration and production testing stations, the calibration and production testing of the target camera in this application are both performed at the same fusion station, with the target camera fixed at a position such as... Figure 2In the fusion station shown, the test board is positioned at a predetermined distance directly opposite the fusion camera, ensuring that the target camera can capture a complete and clear image of the test board. During calibration testing, the target camera is used to photograph the test board to obtain a true speckle pattern. Then, based on the first feature information of the test board in the world coordinate system, the second feature information of the test board in the camera coordinate system is determined. The world coordinate system can be established with a corner point of the test board as its origin; therefore, the first feature information of the test board in the world coordinate system is known.

[0039] In some examples, the first feature information includes the three-dimensional coordinates of each point on the test board in the world coordinate system, while the second feature information is the three-dimensional coordinates of each point on the test board in the camera coordinate system.

[0040] In some examples, the first feature information is the shape function of the test board in the world coordinate system, denoted as z = f(x,y), and the second feature information is the shape function of the test board in the camera coordinate system, denoted as c = g(a,b).

[0041] In some cases, the test board only needs to have sufficiently obvious feature points, such as Harris corners or QR code patterns. Figure 3 A test board is shown, the surface of which is provided with a first image block for detecting color cast, an image strip for detecting sharpness, a second image block for detecting brightness, and a stereo image block for detecting depth map quality. The first image block includes... Figure 3 The image includes red, green, and blue image blocks, and the image stripes include... Figure 3 The image contains multiple black and white bands spaced at intervals. The widths of the black bands are all different, while the widths of the white bands are all equal. The second image block includes... Figure 3 The image blocks consist of black, white, and gray blocks, while the stereoscopic image blocks include multiple color blocks of varying heights.

[0042] Step 102: Generate several candidate speckle maps based on the second feature information and several different assembly errors, and calculate the matching cost between each candidate speckle map and the real speckle map.

[0043] In practical implementation, the core of a speckle structured light camera consists of two components: a speckle projector and an infrared lens. The speckle projector projects a speckle structured light pattern onto the target scene, which is then captured by the infrared lens to obtain the speckle map. The speckle projector is generally composed of a VCSEL (Vertical-Cavity Surface-Emitting Laser), a CL (Collimator), and DOE (Diffractive Optical Elements). When designing the speckle projector, the positional coordinates of each emitter on the VCSEL and the corresponding coordinate points after diffraction by the DOE are known. Based on this, the speckle GT map can be obtained. If the components of the speckle projector can be perfectly assembled, the reference speckle map should be a part of the speckle GT map. Then, by locating a few key points, the clipping position, size, and scaling factor of the speckle GT map can be determined, thus obtaining the reference speckle map from the speckle GT map. However, in actual assembly, assembly errors inevitably occur between the VCSEL and DOE, including the rotation and tilt angles of the VCSEL and DOE. This results in angular discrepancies in the splicing of different diffraction orders, and the speckles within each order cannot perfectly match the speckles in the speckle GT image. Therefore, this application uses a simulated matching method to determine the assembly error, without requiring additional precision equipment to obtain the assembly error. Specifically, based on the second feature information and several different assembly errors, several candidate speckle images are generated, and the matching cost between each candidate speckle image and the real speckle image is calculated. The matching cost method used can be set by those skilled in the art according to actual needs.

[0044] In some examples, assembly errors include the tilt angle and rotation angle between the VCSEL and DOE in the camera's speckle projector. The server can generate several different tilt angles based on a first preset step size, with the largest tilt angle being less than a first preset threshold, and generate several different rotation angles based on a second preset step size, with the largest rotation angle being less than a second preset threshold. Then, the server can arbitrarily combine the several different tilt angles and the several different rotation angles to obtain several different assembly errors.

[0045] Step 103: The candidate speckle image with the lowest matching cost is designated as the reference speckle image for the target camera, and the target camera is subjected to production testing based on the real speckle image and the reference speckle image.

[0046] In a specific implementation, after the server calculates the matching cost values between each candidate speckle pattern and the real speckle pattern, it can find the candidate speckle pattern with the minimum matching cost value, and designate the candidate speckle pattern with the minimum matching cost value as the reference speckle pattern of the target camera. At this time, the server has obtained the reference speckle pattern and the real speckle pattern, and then can perform production testing on the target camera based on the real speckle pattern and the reference speckle pattern.

[0047] In some examples, when the server controls the target camera to take pictures of a test board as Figure 3 shown, it can synchronously take pictures of the color image, infrared image of the test board. The server can also obtain the depth map of the test board based on the real speckle pattern and the reference speckle pattern. During production testing, the server respectively obtains the first quality index for the color image, the second quality index for the infrared image, and the third quality index for the depth map. When the first quality index, the second quality index, and the third quality index all meet the preset conditions, it can be determined that the quality of the target camera meets the standard. The first quality index includes clarity, the second quality index includes clarity, brightness, and distortion degree, and the third quality index includes the void ratio.

[0048] In this embodiment, the server first controls the target camera to take pictures of a preset test board to obtain a real speckle pattern. According to the real speckle pattern and the first feature information of the test board in the world coordinate system, it determines the second feature information of the test board in the camera coordinate system of the target camera. Then, based on the second feature information and several different assembly errors, it generates several candidate speckle patterns, and calculates the matching cost values between each candidate speckle pattern and the real speckle pattern respectively. Then, it designates the candidate speckle pattern with the minimum matching cost value as the reference speckle pattern of the target camera. Finally, it performs production testing on the target camera based on the real speckle pattern and the reference speckle pattern. Considering that for some application scenarios, there are steps that cannot be avoided but are actually unnecessary in the calibration process and the production testing process, the calibration efficiency and the production testing efficiency are low, and the cost is high. However, the embodiment of the present application combines the calibration process and the production testing process into one. Both processes are based on a preset test board and can be completed using one work station, realizing the integration of camera calibration and production testing, saving the cost of one work station for the production line. At the same time, the calibration process does not require additional equipment to calculate the assembly error, but determines the assembly error through a simulation matching method, greatly improving the efficiency of camera calibration and production testing.

[0049] In one embodiment, the first feature information is the shape function of the test board in the world coordinate system, and the second feature information is the shape function of the test board in the camera coordinate system. The server can, through the steps as Figure 4 shown, according to the real speckle pattern obtained by the target camera taking pictures of the preset test board and the first feature information of the test board in the world coordinate system, determine the second feature information of the test board in the camera coordinate system, specifically including:

[0050] Step 201: Determine the points corresponding to the feature points of the test board in the real speckle image and obtain the coordinates of the feature points in the image coordinate system.

[0051] In the specific implementation, several feature points are set on the test board. These feature points can be, for example, Harris corner points, points on QR code patterns, etc. These feature points are easily detected in the real speckle map. After the server obtains the real speckle map, it can determine the points corresponding to each feature point on the test board in the real speckle map and obtain the coordinates of each feature point in the image coordinate system. The coordinates of the i-th feature point in the image coordinate system can be represented as (u i ,v i Correspondingly, the coordinates of the i-th feature point in the image coordinate system can be represented as (x... i ,y i ,z i ).

[0052] Step 202: Determine the coordinates of the feature point in the camera coordinate system based on the coordinates of the feature point in the world coordinate system and the coordinates of the feature point in the image coordinate system.

[0053] In the specific implementation, the server uses the image coordinate system as a bridge to first establish the mapping relationship between the world coordinate system and the image coordinate system, and then establish the mapping relationship between the image coordinate system and the camera coordinate system. This gives us the mapping relationship between the world coordinate system and the camera coordinate system. Then, based on the coordinates of the feature point in the world coordinate system and the coordinates of the feature point in the image coordinate system, we determine the coordinates of the feature point in the camera coordinate system.

[0054] In some examples, the number of feature points on the test board is at least five. The server determines the coordinates of the feature points in the camera coordinate system based on their coordinates in the world coordinate system and their coordinates in the image coordinate system. This can be achieved through methods such as... Figure 5 The implementation of each sub-step shown includes:

[0055] Sub-step 2021: Determine the actual distance between each pair of feature points based on their coordinates in the world coordinate system.

[0056] In the actual implementation, the actual distance between two feature points is fixed regardless of whether it is in the world coordinate system or the camera coordinate system. It will not change with rigid transformation. The server first calculates the actual distance between each pair of feature points, that is, it determines the actual distance between each pair of feature points based on the coordinates of each feature point in the world coordinate system.

[0057] In some examples, the number of feature points is e, where e is an integer greater than 4. The actual distance between each pair of feature points can be calculated as follows: The actual distance.

[0058] Sub-step 2022: Traverse each actual distance, and establish the equation corresponding to the current actual distance based on the current actual distance and the coordinates of the two feature points corresponding to the current actual distance in the image coordinate system.

[0059] In the implementation, after the server calculates the actual distances between each pair of feature points, it can iterate through all the actual distances and establish an equation corresponding to the current actual distance based on the current actual distance and the coordinates of the two feature points corresponding to the current actual distance in the image coordinate system. The unknowns in the equation include the focal length of the target camera, the baseline distance, and the vertical coordinates of the two feature points corresponding to the current actual distance in the camera coordinate system. For five feature points, ten equations can be established.

[0060] In some cases, based on the current actual distance and the coordinates of the two feature points corresponding to the current actual distance in the image coordinate system, an equation corresponding to the current actual distance can be established, which can be achieved through the following formula:

[0061]

[0062] In the formula, d is the current actual distance, (u1,v1) is the coordinate of the first feature point in the image coordinate system, (u2,v2) is the coordinate of the second feature point in the image coordinate system, (fx,fy) is the focal length of the target camera, (cx,cy) is the baseline distance of the target camera, c1 is the vertical coordinate of the first feature point in the camera coordinate system, and c2 is the vertical coordinate of the second feature point in the camera coordinate system.

[0063] Sub-step 2023: Combine the equations established based on each actual distance to obtain a system of equations and solve them to obtain the focal length of the target camera, the baseline distance, and the vertical coordinates of each feature point in the camera coordinate system.

[0064] In practical implementation, after the server establishes an equation for each actual distance, it can simultaneously solve the system of equations based on these actual distances to obtain the focal length of the target camera, the baseline distance, and the vertical coordinates of each feature point in the camera coordinate system. For five feature points, ten equations can be established, resulting in nine unknowns. The established equations constitute an overdetermined system of equations, which the server can solve using the least squares method.

[0065] Step 203: Determine the rigid transformation relationship between the camera coordinate system and the world coordinate system based on the preset cost function, the coordinates of the feature points in the world coordinate system, and the coordinates of the feature points in the camera coordinate system.

[0066] In the specific implementation, after the server determines the coordinates of each feature point in the camera coordinate system, it can determine the rigid transformation relationship between the camera coordinate system and the world coordinate system based on the preset cost function, the coordinates of the feature points in the world coordinate system, and the coordinates of the feature points in the camera coordinate system.

[0067] In some examples, the server determines the rigid transformation relationship between the camera coordinate system and the world coordinate system based on a preset cost function, the coordinates of the feature points in the world coordinate system, and the coordinates of the feature points in the camera coordinate system. This can be achieved using the following formula:

[0068]

[0069] In the formula, cost represents the preset cost function, (a i ,b i ,c i (x) represents the coordinates of the feature point in the camera coordinate system. i ,y i ,z i ) represents the coordinates of the feature point in the world coordinate system, and [R,t] represents the rigid transformation relationship between the camera coordinate system and the world coordinate system.

[0070] Step 204: Determine the shape function of the test board in the camera coordinate system based on the rigid transformation relationship and the shape function of the test board in the world coordinate system.

[0071] In the specific implementation, the server performs corresponding transformations on the shape function of the test board in the world coordinate system according to the rigid transformation relationship, thereby obtaining the shape function of the test board in the camera coordinate system.

[0072] In this embodiment, considering that the shape function of the test board in the world coordinate system is known inherent information of the test board, but the camera coordinate system established based on the camera optical axis is not completely consistent with the world coordinate system and there are certain differences, this difference is the rigid transformation relationship between the two coordinate systems. The test board has many feature points that are easily found in the real speckle image. By using the coordinates of the feature points in the image coordinate system, the coordinates of the feature points in the camera coordinate system can be obtained. Therefore, the rigid transformation relationship between the two coordinate systems can be quickly and accurately determined using the cost function.

[0073] In one embodiment, the server calculates the matching cost between each candidate speckle map and the true speckle map, which can be achieved through methods such as... Figure 6 The steps shown are implemented as follows:

[0074] Step 301: Traverse each candidate speckle pattern, arbitrarily select a reference speckle and a reference diffraction order in the real speckle pattern, and determine the coordinates of the speckle corresponding to the reference speckle in the current candidate speckle pattern.

[0075] In the specific implementation, after the server generates several candidate speckle patterns based on the second feature information and several different assembly errors, it can traverse each candidate speckle pattern, arbitrarily select a reference speckle and a reference diffraction order in the real speckle pattern, and determine the coordinates of the speckle corresponding to the reference speckle in the current candidate speckle pattern, that is, find the speckle with the same name as the reference speckle in the current candidate speckle pattern.

[0076] Step 302: Calculate the matching cost between the current candidate speckle map and the real speckle map based on the coordinates of the reference speckle, the coordinates of the corresponding speckle in the current candidate speckle map, the boundary coordinate cluster of the reference diffraction order in the real speckle map, the boundary coordinate cluster of the reference diffraction order in the current candidate speckle map, and the preset bias coefficient.

[0077] In its implementation, the server arbitrarily selects a reference speckle and a reference diffraction order in the real speckle map. After determining the coordinates of the speckle corresponding to the reference speckle in the current candidate speckle map, the server calculates the matching cost between the current candidate speckle map and the real speckle map based on the coordinates of the reference speckle, the coordinates of the corresponding speckle in the current candidate speckle map, the boundary coordinate clusters of the reference diffraction order in the real speckle map, the boundary coordinate clusters of the reference diffraction order in the current candidate speckle map, and a preset bias coefficient. Based on the coordinates of the reference speckle and the coordinates of the corresponding speckle in the current candidate speckle map, the coordinate error between speckles of the same name can be measured. Based on the boundary coordinate clusters of the reference diffraction order in the real speckle map and the boundary coordinate clusters of the reference diffraction order in the current candidate speckle map, the matching degree between orders can be measured. The bias coefficient is used to adjust the importance of tilt and rotation angles in assembly errors. If the importance of rotation angle is relatively high, the bias coefficient can be set to a smaller value to accurately calculate the matching cost that meets the actual requirements.

[0078] In some examples, the server calculates the matching cost between the current candidate speckle map and the real speckle map based on the coordinates of the reference speckle, the coordinates of the corresponding speckle in the current candidate speckle map, the boundary coordinate cluster of the reference diffraction order in the real speckle map, the boundary coordinate cluster of the reference diffraction order in the current candidate speckle map, and a preset bias coefficient. This can be achieved using the following formula:

[0079] L=∑||(p s ,q s )-(ks ,h s )||+λΣ||m w -n w ||

[0080] In the formula, L is the matching cost, (k s ,h s (p) represents the coordinates of the reference speckle, s ,q s ) represents the coordinates of the reference speckle in the current candidate speckle map, n w The m represents the boundary coordinate set of the reference diffraction order in the true speckle pattern. w This represents the boundary coordinate cluster of the reference diffraction order in the current candidate speckle pattern, where λ is a preset bias coefficient.

[0081] In one embodiment, the server generates several candidate speckle maps based on the second feature information and several different assembly errors, which can be achieved through methods such as... Figure 7 The steps shown are implemented as follows:

[0082] Step 401: Iterate through several different assembly errors, and establish the first mapping relationship between the VCSEL coordinate system and the camera coordinate system based on the current assembly error, the shape function of the test board in the camera coordinate system, and the pose relationship between the speckle projector and the infrared lens.

[0083] In the specific implementation, after the server generates several different assembly errors, it can establish a mapping relationship between the VCSEL coordinate system and the speckle projector coordinate system based on the current assembly error and the shape function of the test board in the camera coordinate system. Then, combined with the pose relationship between the speckle projector and the infrared lens, the first mapping relationship between the VCSEL coordinate system and the camera coordinate system is established.

[0084] In one example, the DOE could be META-Optic, which is based on hyperplane technology and has a higher degree of integration than traditional DOEs, and also has collimation capabilities.

[0085] Step 402: Based on the shape function of the test board in the camera coordinate system, establish a second mapping relationship between the camera coordinate system and the image coordinate system.

[0086] In practice, speckle patterns are two-dimensional images, while camera coordinates are three-dimensional. Therefore, to generate speckle patterns by simulating diffraction, a second mapping relationship between the camera coordinate system and the image coordinate system must be established. This is actually dimensionality reduction, which is related to the shape function of the test board in the camera coordinate system. That is, based on the shape function of the test board in the camera coordinate system, a second mapping relationship between the camera coordinate system and the image coordinate system is established.

[0087] In one example, the server can use the shape function of the test board in the camera coordinate system to remove the z-values ​​of each point in the camera coordinate system, thereby establishing a second mapping relationship between the camera coordinate system and the image coordinate system.

[0088] Step 403: Based on the first mapping relationship and the second mapping relationship, map each emitter on the VCSEL to a diffraction speckle pattern in the image coordinate system, and generate a candidate speckle pattern corresponding to the current assembly error based on each diffraction speckle pattern.

[0089] In the specific implementation, after the server establishes the first mapping relationship and the second mapping relationship, it finds the mapping relationship between the VCSEL coordinate system and the image coordinate system. Based on this, each transmitter on the VCSEL can be mapped to a diffraction speckle pattern in the image coordinate system, and a candidate speckle pattern corresponding to the current assembly error can be generated according to each diffraction speckle pattern.

[0090] In one example, the server determines the diffracted coordinates based on the transmitter coordinates on the VCSEL, the first mapping relationship, and the second mapping relationship, and assigns the transmitter's brightness value to the diffracted coordinates. Based on the assigned brightness values, the server generates a candidate speckle map corresponding to the current assembly error.

[0091] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0092] Another embodiment of this application relates to an integrated calibration and testing system for cameras. The implementation details of this integrated calibration and testing system are described below. The following content is for ease of understanding and is not essential for implementing this solution. A schematic diagram of the integrated calibration and testing system for cameras in this embodiment can be seen as follows: Figure 8 As shown, it includes: an acquisition module 501, a transformation module 502, a matching module 503, a calibration module 504, and a production testing module 505.

[0093] The acquisition module 501 is used to acquire the real speckle image obtained by the target camera from the preset test board.

[0094] The transformation module 502 is used to determine the second feature information of the test board in the camera coordinate system based on the real speckle pattern and the first feature information of the test board in the world coordinate system.

[0095] The matching module 503 is used to generate several candidate speckle maps based on the second feature information and several different assembly errors, and to calculate the matching cost between each candidate speckle map and the real speckle map.

[0096] The calibration module 504 is used to designate the candidate speckle pattern with the lowest matching cost as the reference speckle pattern for the target camera.

[0097] The production test module 505 is used to perform production tests on the target camera based on the real speckle pattern and the reference speckle pattern.

[0098] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0099] Another embodiment of this application relates to an electronic device, such as... Figure 9 As shown, it includes: at least one processor 601; and a memory 602 communicatively connected to the at least one processor 601; wherein the memory 602 stores instructions executable by the at least one processor 601, the instructions being executed by the at least one processor 601 to enable the at least one processor 601 to execute the integrated calibration and measurement method for cameras in the above embodiments.

[0100] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0101] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0102] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0103] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for integrating camera calibration and production testing, characterized in that, include: Based on the real speckle pattern obtained by the target camera from the preset test board and the first feature information of the test board in the world coordinate system, the second feature information of the test board in the camera coordinate system is determined. Based on the second feature information and several different assembly errors, several candidate speckle maps are generated, and the matching cost between each candidate speckle map and the real speckle map is calculated respectively. The candidate speckle pattern with the lowest matching cost is designated as the reference speckle pattern for the target camera, and the target camera is subjected to production testing based on the real speckle pattern and the reference speckle pattern. The step of calculating the matching cost between each candidate speckle map and the real speckle map includes: Traverse each of the candidate speckle patterns, arbitrarily select a reference speckle and a reference diffraction order in the real speckle pattern, and determine the coordinates of the speckle corresponding to the reference speckle in the current candidate speckle pattern; The matching cost between the current candidate speckle map and the real speckle map is calculated based on the coordinates of the reference speckle, the coordinates of the corresponding speckle in the current candidate speckle map, the boundary coordinate cluster of the reference diffraction order in the real speckle map, the boundary coordinate cluster of the reference diffraction order in the current candidate speckle map, and a preset bias coefficient.

2. The integrated calibration and testing method for cameras according to claim 1, characterized in that, The first feature information is the shape function of the test board in the world coordinate system, and the second feature information is the shape function of the test board in the camera coordinate system. Determining the second feature information of the test board in the camera coordinate system based on the real speckle pattern obtained by the target camera from the preset test board and the first feature information of the test board in the world coordinate system includes: In the real speckle image, the points corresponding to the feature points of the test board are determined, and the coordinates of the feature points in the image coordinate system are obtained. The coordinates of the feature point in the camera coordinate system are determined based on the coordinates of the feature point in the world coordinate system and the coordinates of the feature point in the image coordinate system. Based on the preset cost function, the coordinates of the feature point in the world coordinate system, and the coordinates of the feature point in the camera coordinate system, the rigid transformation relationship between the camera coordinate system and the world coordinate system is determined. Based on the rigid transformation relationship and the shape function of the test board in the world coordinate system, the shape function of the test board in the camera coordinate system is determined.

3. The integrated calibration and testing method for cameras according to claim 2, characterized in that, The number of feature points is at least five. Determining the coordinates of the feature points in the camera coordinate system based on their coordinates in the world coordinate system and their coordinates in the image coordinate system includes: Determine the actual distance between each pair of the aforementioned feature points based on their coordinates in the world coordinate system. Iterate through each of the actual distances, and establish an equation corresponding to the current actual distance based on the current actual distance and the coordinates of the two feature points corresponding to the current actual distance in the image coordinate system; wherein, the unknowns in the equation include the focal length of the target camera, the baseline distance, and the vertical coordinates of the two feature points corresponding to the current actual distance in the camera coordinate system; The equations established based on the actual distances are combined to obtain a system of equations, which are then solved to obtain the focal length of the target camera, the baseline distance, and the vertical coordinates of each feature point in the camera coordinate system.

4. The integrated calibration and testing method for cameras according to claim 3, characterized in that, The equation corresponding to the current actual distance is established based on the current actual distance and the coordinates of the two feature points corresponding to the current actual distance in the image coordinate system, which is achieved through the following formula: Wherein, d is the current actual distance, (u1, v1) is the coordinate of the first feature point in the image coordinate system, (u2, v2) is the coordinate of the second feature point in the image coordinate system, (fx, fy) is the focal length, (cx, cy) is the baseline distance, c1 is the vertical coordinate of the first feature point in the camera coordinate system, and c2 is the vertical coordinate of the second feature point in the camera coordinate system.

5. The integrated calibration and testing method for a camera according to any one of claims 2 to 4, characterized in that, The rigid transformation relationship between the camera coordinate system and the world coordinate system is determined based on a preset cost function, the coordinates of the feature point in the world coordinate system, and the coordinates of the feature point in the camera coordinate system. This is achieved through the following formula: Where cost represents the preset cost function, (a i ,b i ,c i (x) represents the coordinates of the feature point in the camera coordinate system. i ,y i ,z i ) represents the coordinates of the feature point in the world coordinate system, and [R,t] represents the rigid transformation relationship between the camera coordinate system and the world coordinate system.

6. The integrated calibration and testing method for cameras according to claim 1, characterized in that, The matching cost between the current candidate speckle map and the real speckle map is calculated based on the coordinates of the reference speckle, the coordinates of the corresponding speckle in the current candidate speckle map, the boundary coordinate cluster of the reference diffraction order in the real speckle map, the boundary coordinate cluster of the reference diffraction order in the current candidate speckle map, and a preset bias coefficient. This is achieved through the following formula: Where L is the matching cost, (k s ,h s (p) represents the coordinates of the reference speckle, (p) s ,q s ) represents the coordinates of the reference speckle in the current candidate speckle map, n w The m represents the boundary coordinate family of the reference diffraction order in the true speckle pattern. w The reference diffraction order represents the boundary coordinate cluster in the current candidate speckle map, and λ is the preset bias coefficient.

7. The integrated calibration and testing method for a camera according to any one of claims 1 to 4, characterized in that, The assembly errors include the tilt angle and rotation angle between the VCSEL and DOE in the speckle projector of the camera. The various assembly errors are obtained through the following steps: Several different tilt angles are generated based on the first preset step size, and the largest tilt angle is less than the first preset threshold. Several different rotation angles are generated based on the second preset step size, and the largest rotation angle is less than the second preset threshold. By arbitrarily combining the several different tilt angles and the several different rotation angles, several different assembly errors can be obtained.

8. The integrated calibration and testing method for a camera according to any one of claims 2 to 4, characterized in that, The generation of several candidate speckle maps based on the second feature information and several different assembly errors includes: By iterating through the various assembly errors, and based on the current assembly error, the shape function of the test board in the camera coordinate system, and the pose relationship between the speckle projector and the infrared lens, a first mapping relationship between the VCSEL coordinate system and the camera coordinate system is established. Based on the shape function of the test board in the camera coordinate system, a second mapping relationship is established between the camera coordinate system and the image coordinate system; Based on the first mapping relationship and the second mapping relationship, each emitter on the VCSEL is mapped to a diffraction speckle pattern in the image coordinate system, and a candidate speckle pattern corresponding to the current assembly error is generated based on each diffraction speckle pattern.

9. A camera calibration and measurement integrated system, characterized in that, include: The module includes an acquisition module, a transformation module, a matching module, a calibration module, and a production testing module. The acquisition module is used to acquire the real speckle image obtained by the target camera from the preset test board; The transformation module is used to determine the second feature information of the test board in the camera coordinate system based on the real speckle map and the first feature information of the test board in the world coordinate system; The matching module is used to generate several candidate speckle maps based on the second feature information and several different assembly errors, and to calculate the matching cost between each candidate speckle map and the real speckle map respectively. The matching module is further specifically used to traverse each of the candidate speckle patterns, arbitrarily select a reference speckle and a reference diffraction order in the real speckle pattern, determine the coordinates of the speckle corresponding to the reference speckle in the current candidate speckle pattern, and calculate the matching cost between the current candidate speckle pattern and the real speckle pattern based on the coordinates of the reference speckle, the coordinates of the speckle corresponding to the reference speckle in the current candidate speckle pattern, the boundary coordinate cluster of the reference diffraction order in the real speckle pattern, the boundary coordinate cluster of the reference diffraction order in the current candidate speckle pattern, and a preset bias coefficient. The calibration module is used to designate the candidate speckle pattern with the lowest matching cost as the reference speckle pattern for the target camera. The production testing module is used to perform production testing on the target camera based on the real speckle pattern and the reference speckle pattern.

10. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the integrated calibration and measurement method for a camera as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the integrated calibration and measurement method for the camera as described in any one of claims 1 to 8.

Images