Evaluation apparatus and method for a binocular structured light camera module

Abstract

The application provides an evaluation device and method for a binocular structured light camera module. The evaluation device comprises a first fixing device, a left infrared camera module, a speckle projection module and a right infrared camera module which are installed on a substrate support according to a preset baseline, and a reflecting device which is horizontally arranged at a specific distance above the vertical direction of the first fixing device and is used for reflecting the light beam projected by the speckle projection module. Through the evaluation device, whether the speckle projection module, the left infrared camera module and the right infrared camera module of the binocular structured light camera after use are damaged and whether the corresponding calibration external parameters are changed can be quickly evaluated, so that the reason for the decrease in the depth precision of the binocular structured light camera can be quickly found out, and a reasonable maintenance scheme can be formulated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical technology, in particular to an evaluation device and method of a binocular structured light camera module. BACKGROUND

[0002] In recent years, 3D structured light camera modules are increasingly applied in the field of consumer electronics, such as in the fields of robot obstacle avoidance, face payment and scene modeling. The 3D structured light technology can not only image a target object, but also obtain the depth information of the target object, and is the most widely used 3D imaging device at present.

[0003] At present, there is a big pain point in the industry for 3D binocular structured light depth cameras. After the end customers use them for a period of time, the depth accuracy of some depth cameras may decrease, that is, there may be partial holes or random noise points in the depth map. According to the industry positioning confirmation, the main reasons for this problem generally include two aspects: one is that the speckle projector module or the left and right infrared camera modules are damaged, resulting in deterioration of the optical system; the other is that the calibration external parameters of the depth camera have changed during use, that is, the position of the baseline module has changed, resulting in deterioration of the depth performance.

[0004] Therefore, how to quickly judge whether each camera module of the binocular structured light camera is damaged and whether the calibration external parameters are changed during use has become an important problem that needs to be solved in the industry. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose an evaluation device and method of a binocular structured light camera module. The purpose is to quickly evaluate whether the corresponding speckle projection module, left infrared camera module and right infrared camera module of the binocular structured light camera after use are damaged and whether the corresponding calibration external parameters are changed based on the evaluation device of the binocular structured light camera module, so as to quickly find out the reason for the decrease in the depth accuracy of the binocular structured light camera and help to develop a reasonable maintenance plan.

[0006] To achieve the above purpose, a first aspect of the embodiments of the present application proposes an evaluation device of a binocular structured light camera module, which comprises:

[0007] A first fixing device comprising a substrate support and a left infrared camera module, a speckle projection module and a right infrared camera module mounted on the substrate support according to a pre-set baseline, wherein the speckle projection module is located at the center position of the left infrared camera module and the right infrared camera module;

[0008] A reflection device horizontally arranged at a specific distance vertically above the first fixing device, used for reflecting the light beam projected by the speckle projection module.

[0009] To achieve the above object, the second aspect of the embodiment of the present application provides an evaluation method of a binocular structured light camera module, which is executed based on the evaluation device of the binocular structured light camera module in the first aspect. The evaluation method comprises:

[0010] The used binocular structured light camera module is installed at the corresponding position of the evaluation device. The binocular structured light camera module comprises a speckle projection module, a left infrared camera module and a right infrared camera module.

[0011] The speckle image projected by the speckle projection module is collected by the camera module, and the zero-level area speckle points are extracted. The zero-level area speckle points are all the speckle points in the central region of the speckle image. The camera module comprises the left infrared camera module and the right infrared camera module.

[0012] The centroid coordinate set, the speckle area mean value and the speckle ellipticity mean value of the zero-level area speckle points are calculated respectively.

[0013] Whether the binocular structured light camera module is damaged and whether the camera calibration external parameter is changed are judged according to the centroid coordinate set, the speckle area mean value and the speckle ellipticity mean value of the zero-level area speckle points and the corresponding preset specifications. The preset specifications are standard data of the binocular structured light camera module when it is shipped out, which are obtained based on the evaluation device. The preset specifications comprise a centroid standard coordinate set, a speckle area threshold value and a speckle ellipticity threshold value.

[0014] In some embodiments, the speckle image projected by the speckle projection module is collected by the camera module, and the zero-level area speckle points are extracted, which comprises:

[0015] The first speckle image projected by the speckle projection module is collected by the left infrared camera module, and the first zero-level area speckle points are extracted. The first zero-level area speckle points are all the speckle points in the central region of the speckle image.

[0016] The second speckle image projected by the speckle projection module is collected by the right infrared camera module, and the second zero-level area speckle points are extracted. The second zero-level area speckle points are all the speckle points in the central region of the speckle image.

[0017] In some embodiments, the centroid coordinates, the speckle area mean value and the speckle ellipticity mean value of all the zero-level area speckle points are calculated respectively, which comprises:

[0018] A first array is calculated. The first array comprises a first centroid coordinate set, a first speckle area mean value and a first speckle ellipticity mean value of the first zero-level area speckle points.

[0019] calculating a second array including a second centroid coordinate set of second speckle points in the second zero-level region, a second speckle area average, and a second speckle ellipticity average.

[0020] In some embodiments, the calculating the first array includes:

[0021] processing the first zero-level region to identify all speckle points in the first zero-level region;

[0022] obtaining centroid coordinates of each speckle point in the first zero-level region to obtain the first centroid coordinate set;

[0023] calculating a speckle area and a speckle ellipticity of each speckle point in the first zero-level region;

[0024] calculating the first speckle area average according to the speckle area of each speckle point in the first zero-level region;

[0025] calculating the first speckle ellipticity average according to the speckle ellipticity of each speckle point in the first zero-level region.

[0026] In some embodiments, the calculating the second array includes:

[0027] processing the second zero-level region to identify all speckle points in the second zero-level region;

[0028] obtaining centroid coordinates of each speckle point in the second zero-level region to obtain the second centroid coordinate set;

[0029] calculating a speckle area and a speckle ellipticity of each speckle point in the second zero-level region;

[0030] calculating the second speckle area average according to the speckle area of each speckle point in the second zero-level region;

[0031] calculating the second speckle ellipticity average according to the speckle ellipticity of each speckle point in the second zero-level region.

[0032] In some embodiments, the judging whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters are changed according to the centroid coordinate set of the speckle points in the zero-level region, the speckle area average, the speckle ellipticity average, and the corresponding preset specifications includes:

[0033] obtaining a first preset specification, the first preset specification being standard data of the left infrared camera module when the left infrared camera module is shipped, the first preset specification including a first centroid standard coordinate set, a first speckle area threshold, and a first speckle ellipticity threshold;

[0034] obtaining a second preset specification, the second preset specification being standard data of the right infrared camera module when leaving factory, obtained by the evaluation device, the second preset specification comprising a second centroid standard coordinate set, a second speckle area threshold and a second speckle ellipticity threshold;

[0035] comparing the first array with the first preset specification and comparing the second array with the second preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameter is changed.

[0036] In some embodiments, comparing the first array with the first preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameter is changed comprises:

[0037] determining whether the first centroid coordinate set is consistent with the first centroid standard coordinate set;

[0038] if the first centroid coordinate set is consistent with the first centroid standard coordinate set, determining that the left infrared camera module and the speckle projection module are not damaged, and that the first extrinsic parameter is not changed, the first extrinsic parameter being a calibration extrinsic parameter corresponding to the left infrared camera module;

[0039] if the first centroid coordinate set is not consistent with the first centroid standard coordinate set, determining that the first extrinsic parameter is changed;

[0040] determining whether the first speckle area mean value is greater than the first speckle area threshold and whether the first speckle ellipticity mean value is less than the first speckle ellipticity threshold;

[0041] if the first speckle area mean value is greater than the first speckle area threshold or the first speckle ellipticity mean value is less than the first speckle ellipticity threshold, determining that the left infrared camera module or the speckle projection module is damaged;

[0042] if the first speckle area mean value is not greater than the first speckle area threshold and the first speckle ellipticity mean value is not less than the first speckle ellipticity threshold, determining that the left infrared camera module and the speckle projection module are not damaged.

[0043] In some embodiments, comparing the second array with the second preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameter is changed comprises:

[0044] determining whether the second centroid coordinate set is consistent with the second centroid standard coordinate set;

[0045] if the second centroid coordinate set is consistent with the second centroid standard coordinate set, it is determined that the right infrared camera module and the speckle projection module are not damaged, and the second external parameter, which is a calibration external parameter corresponding to the right infrared camera module, has not changed;

[0046] if the second centroid coordinate set is not consistent with the second centroid standard coordinate set, it is determined that the second external parameter has changed;

[0047] it is determined whether the second speckle area average is greater than the second speckle area threshold and whether the second speckle ellipticity average is less than the second speckle ellipticity threshold;

[0048] if the second speckle area average is greater than the second speckle area threshold or the second speckle ellipticity average is less than the second speckle ellipticity threshold, it is determined that the right infrared camera module or the speckle projection module is damaged;

[0049] if the second speckle area average is not greater than the second speckle area threshold and the second speckle ellipticity average is not less than the second speckle ellipticity threshold, it is determined that the right infrared camera module and the speckle projection module are not damaged.

[0050] In some embodiments, the comparison of the first array with the first preset specification and the comparison of the second array with the second preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration external parameter has changed include:

[0051] comparing the first array with the first preset specification and comparing the second array with the second preset specification;

[0052] when the first centroid coordinate set is consistent with the first centroid standard coordinate set and the second centroid coordinate set is consistent with the second centroid standard coordinate set, it is determined that the binocular structured light camera module is not damaged and the camera calibration external parameter has not changed;

[0053] when the first centroid coordinate set is consistent with the first centroid standard coordinate set, the second centroid coordinate set is not consistent with the second centroid standard coordinate set, and the second speckle area average is not greater than the second speckle area threshold and the second speckle ellipticity average is not less than the second speckle ellipticity threshold, it is determined that the second external parameter has changed;

[0054] when the first centroid coordinate set is consistent with the first centroid standard coordinate set, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area mean value is greater than the second speckle area threshold or the second speckle ellipticity mean value is less than the second speckle ellipticity threshold, it is determined that the right infrared camera module is damaged and the second extrinsic parameter is changed;

[0055] when the second centroid coordinate set is consistent with the second centroid standard coordinate set, the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is not greater than the first speckle area threshold and the first speckle ellipticity mean value is not less than the first speckle ellipticity threshold, it is determined that the first extrinsic parameter is changed;

[0056] when the second centroid coordinate set is consistent with the second centroid standard coordinate set, the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is greater than the first speckle area threshold or the first speckle ellipticity mean value is less than the first speckle ellipticity threshold, it is determined that the left infrared camera module is damaged and the first extrinsic parameter is changed;

[0057] when the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is not greater than the first speckle area threshold and the first speckle ellipticity mean value is not less than the first speckle ellipticity threshold, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area mean value is not greater than the second speckle area threshold and the second speckle ellipticity mean value is not less than the second speckle ellipticity threshold, it is determined that the first extrinsic parameter and the second extrinsic parameter are both changed;

[0058] when the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is not greater than the first speckle area threshold and the first speckle ellipticity mean value is not less than the first speckle ellipticity threshold, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area mean value is greater than the second speckle area threshold or the second speckle ellipticity mean value is less than the second speckle ellipticity threshold, it is determined that the first extrinsic parameter and the second extrinsic parameter are both changed and the right infrared camera module is damaged;

[0059] When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is greater than the first speckle area threshold value or the first speckle ellipticity mean value is less than the first speckle ellipticity threshold value, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area mean value is not greater than the second speckle area threshold value and the second speckle ellipticity mean value is not less than the second speckle ellipticity threshold value, it is determined that the first extrinsic parameter and the second extrinsic parameter are both changed, and at least two camera modules in the binocular structured light camera module are damaged.

[0060] When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is greater than the first speckle area threshold value or the first speckle ellipticity mean value is less than the first speckle ellipticity threshold value, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area mean value is not greater than the second speckle area threshold value and the second speckle ellipticity mean value is not less than the second speckle ellipticity threshold value, it is determined that the first extrinsic parameter and the second extrinsic parameter are both changed, and the left infrared camera module is damaged.

[0061] The application provides an evaluation device and method for a binocular structured light camera module. The evaluation device comprises a first fixing device, a left infrared camera module, a speckle projection module and a right infrared camera module which are installed on a substrate support according to a preset baseline, and a reflecting device which is horizontally arranged at a specific distance above the vertical direction of the first fixing device and is used for reflecting the light beam projected by the speckle projection module. Through the evaluation device, it can be quickly evaluated whether the corresponding speckle projection module, left infrared camera module and right infrared camera module of the binocular structured light camera are damaged and whether the corresponding calibration extrinsic parameter is changed after use, so that the reason for the decrease in the depth precision of the binocular structured light camera can be quickly found out, and a reasonable maintenance plan can be formulated. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 FIG. 1 is a structural schematic diagram of an evaluation device for a binocular structured light camera module provided in an embodiment of the application;

[0063] Figure 2 FIG. 2 is a step flowchart of an evaluation method executed by the evaluation device shown in FIG. 1; Figure 1

[0064] Figure 3 FIG. 3 is a step flowchart of collecting the speckle image projected by the speckle projection module through the camera module and extracting the speckle points in the zero-order region provided in an embodiment of the application;

[0065] ​Figure 4 is a schematic diagram of the first zero-level region extracted according to an embodiment of the present application;

[0066] Figure 5 is a step flowchart of calculating the first array according to an embodiment of the present application;

[0067] Figure 6 is a schematic diagram of the first zero-level region after processing according to an embodiment of the present application;

[0068] Figure 7 is a schematic diagram of the first zero-level region according to an embodiment of the present application; Figure 6 is a data schematic diagram corresponding to each speckle point in the first zero-level region shown in the figure;

[0069] Figure 8 is a step flowchart of calculating the second array according to an embodiment of the present application;

[0070] Figure 9 is a step flowchart of comparing the first array with the first preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameter is changed according to an embodiment of the present application;

[0071] Figure 10 is a step flowchart of comparing the second array with the second preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameter is changed according to an embodiment of the present application;

[0072] Figure 11 is an evaluation flowchart of the binocular structured light camera module according to an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0074] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0076] With the gradual upgrading of the consumer field, the demand for applying 3D imaging technology to the consumer field is increasingly urgent. In addition to imaging the target object, 3D imaging technology can also obtain the depth information of the target object, and according to the depth information, functions such as 3D face recognition, virtual scene modeling, and human-computer interaction can be further realized. At the same time, it is required that the 3D imaging device can meet the requirements of low power, high performance, and miniaturization, so as to be set in an electronic terminal device that is convenient to carry.

[0077] The 3D binocular structured light depth camera is composed of a speckle projector module and two identical left and right infrared camera modules. When the three camera modules are assembled and calibrated, a complete depth map can be output. The speckle projector projects clear pseudo-random speckle points to increase the spatial texture information, and then the two identical left and right infrared camera modules perform binocular stereo matching, and the two modules form a system to perform three-dimensional measurement on the target object.

[0078] After use, some binocular structured light depth cameras may have a problem of depth precision degradation, that is, the phenomenon of partial holes or random noise points in the depth map. The main reasons for this problem include: (1) the speckle projector module or the left and right infrared camera modules are damaged, resulting in a poor optical system, (2) during use, the calibration external parameters of the depth camera change, that is, the position of the baseline module changes, resulting in poor depth performance.

[0079] Therefore, how to quickly determine whether each camera module of the binocular structured light camera is damaged and whether the calibration external parameters change during use has become an important problem that needs to be solved in the industry.

[0080] Based on this, the embodiment of the present application provides an evaluation device for a binocular structured light camera module. The evaluation device can quickly evaluate whether the corresponding speckle projection module, left infrared camera module and right infrared camera module of the used binocular structured light camera are damaged and whether the corresponding calibration external parameters change, so as to quickly find out the reason for the depth precision degradation of the binocular structured light camera and help to develop a reasonable maintenance plan.

[0081] Reference Figure 1 , Figure 1 is a structural schematic diagram of the evaluation device for the binocular structured light camera module provided in the embodiment of the present application. As shown in Figure 1 , the evaluation device comprises:

[0082] The first fixing device 100 comprises a substrate support 101 and a left infrared camera module 102, a speckle projection module 103 and a right infrared camera module 104 installed on the substrate support 101 according to a preset baseline. The speckle projection module 103 is located at the center position of the left infrared camera module 102 and the right infrared camera module 104. The left infrared camera module 102 collects light passing through an infrared lens Lens into a photosensitive chip COMS to form an image. The right infrared camera module 104 is the same as the left infrared camera module 102.

[0083] The reflecting device 110 is horizontally arranged at a specific distance vertically above the first fixing device 100, and is used for reflecting the light beam projected by the speckle projection module 103.

[0084] Specifically, the speckle projection module 103 comprises a light source, a collimating mirror and a diffractive optical element (DOE). The light source is a VCSEL (Vertical-Cavity Surface-Emitting Laser) installed on a ceramic substrate through a semiconductor packaging process. The VCSEL is a two-dimensional light source composed of a two-dimensional pattern of many sub-sources, which has the advantages of small volume, small divergence angle and energy concentration compared with traditional light sources. The collimating mirror is used to receive the light beam emitted by the VCSEL array light source and collimate the light beam with a certain divergence angle. The diffractive optical element is used to receive the collimated light beam and project the light beam into the target space through light diffraction. The light beam is formed by copying the VCSEL array light source. For example, if the VCSEL has 100 sub-sources and the number of copies of the diffractive optical element is 100, then 10,000 speckle points will be formed in the space.

[0085] The reflecting device 110 is horizontally arranged at a specific distance vertically above the first fixing device 100, and is used for reflecting the light beam projected by the speckle projection module 103. Thus, the projected light beam of the speckle projection module 103 can be reflected by the reflecting device 110, so that the left infrared camera module 102 and the right infrared camera module 104 can obtain the speckle image projected by the speckle projection module 103.

[0086] Exemplarily, the reflecting device 110 can be a glass plate horizontally arranged vertically above the first fixing device 100 and a white paper attached to the glass plate. Alternatively, the reflecting device 110 can be a mirror horizontally arranged vertically above the first fixing device 100 and an optical film coated on the mirror.

[0087] It should be noted that, in this embodiment, the left infrared camera module 102 and the right infrared camera module 104 are the left and right camera modules of a binocular structured light camera, and their structures are identical. This is to ensure that the speckle image acquired by the left infrared camera module 102 is the same as the speckle image acquired by the right infrared camera module 104.

[0088] Reference Figure 2 , Figure 2 The embodiments provided in this application are based on Figure 1 The flowchart of the evaluation method performed by the evaluation device shown includes, but is not limited to, steps S201 to S204.

[0089] Step S201: Install the used binocular structured light camera module at the corresponding position of the evaluation device. The binocular structured light camera module includes a speckle projection module, a left infrared camera module, and a right infrared camera module.

[0090] In this embodiment of the application, the used binocular structured light camera module is first placed in a position such as... Figure 1 In the evaluation apparatus shown, the speckle projection module 103 is ensured to be perpendicular to the reflector 110, and a fixed distance is determined between the first fixing device 100 and the reflector 110. Then, the light source of the speckle projection module 103 is turned on, and the projected beam of the speckle projection module 103 can be reflected back to the first fixing device 100 through the reflector 110, so that both the left infrared camera module 102 and the right infrared camera module 104 can acquire the speckle image projected by the speckle projection module 103.

[0091] Step S202: Acquire speckle images projected by the speckle projection module through the camera module, and extract zero-order region speckles. Zero-order region speckles are all speckles in the central region of the speckle image. The camera module includes a left infrared camera module and a right infrared camera module.

[0092] In this embodiment of the application, after the light source of the speckle projection module is turned on, the left infrared camera module and the right infrared camera module can simultaneously acquire speckle images projected by the speckle projection module, and process the acquired speckle images to extract all speckle spots in the central region of the speckle image.

[0093] Reference Figure 3 , Figure 3 This is a flowchart of the steps for acquiring speckle images projected by a speckle projection module through a camera module and extracting speckle spots in the zero-order region, as provided in the embodiments of this application, including but not limited to steps S301 to S302.

[0094] Step S301: Acquire the first speckle image projected by the speckle projection module through the left infrared camera module, and extract the first zero-order region speckles. The first zero-order region speckles are all speckles in the central region of the speckle image.

[0095] In this embodiment, after the light source of the speckle projection module 103 is turned on, the projected beam of the speckle projection module 103 is reflected back to the first fixing device 100 through the reflection device 110, so that the left infrared camera module 102 can acquire the speckle image projected by the speckle projection module 103. The speckle image projected by the speckle projection module 103 is acquired by the left infrared camera module 102 and recorded as the first speckle image. Then, the central region of the first speckle image is extracted and called the first zero-order region. All speckles in the first zero-order region are called the first zero-order region speckles.

[0096] like Figure 4 As shown, Figure 4 This is a schematic diagram of the first zero-level region extracted according to the embodiments of this application. Figure 4 The first zero-order region contains 33 speckles.

[0097] It should be noted that the central region of a speckle image exhibits the best optical performance, free from light distortion and aberration. Therefore, this embodiment extracts the central region from the speckle image and performs calculations and evaluations using the speckles within the central region. This approach reduces computational load while ensuring the accuracy of the evaluation.

[0098] Step S302: Acquire the second speckle image projected by the speckle projection module through the right infrared camera module, and extract the second zero-order region speckles. The second zero-order region speckles are all speckles in the central region of the speckle image.

[0099] In this embodiment, after the light source of the speckle projection module 103 is turned on, the projected beam of the speckle projection module 103 is reflected back to the first fixing device 100 through the reflecting device 110, so that the right infrared camera module 104 can acquire the speckle image projected by the speckle projection module 103. The speckle image projected by the speckle projection module 103 is acquired by the right infrared camera module 104 and recorded as the second speckle image. Then, the central region of the second speckle image is extracted and called the second zero-order region. All speckles in the second zero-order region are called second zero-order region speckles.

[0100] Step S203: Calculate the centroid coordinate set, mean speckle area, and mean speckle ellipticity of the speckle in the zero-order region.

[0101] In this embodiment, after extracting the speckle pattern of the zero-order region, it is necessary to further calculate the centroid coordinate set, the average speckle area, and the average speckle ellipticity of the zero-order region speckle. Specifically, a first array is calculated, which includes the first centroid coordinate set, the average first speckle area, and the average first speckle ellipticity of the first zero-order region speckle. A second array is calculated, which includes the second centroid coordinate set, the average second speckle area, and the average second speckle ellipticity of the second zero-order region speckle.

[0102] Reference Figure 5 , Figure 5 This is a flowchart of the steps for calculating the first array provided in the embodiments of this application, namely, the flowchart of the steps for calculating the first centroid coordinate set, the average area of ​​the first speckle, and the average ellipticity of the first speckle in the first zero-order region, including but not limited to steps S501 to S505.

[0103] Step S501: Process the first zero-level region to identify all speckles in the first zero-level region;

[0104] Step S502: Obtain the centroid coordinates of each speckle in the first zero-level region to obtain the first centroid coordinate set;

[0105] Step S503: Calculate the speckle area and speckle ellipticity of each speckle in the first zero-order region.

[0106] Step S504: Calculate the average area of ​​the first speckle pattern based on the speckle area of ​​each speckle pattern in the first zero-level region.

[0107] Step S505: Calculate the average value of the first speckle ellipticity based on the speckle ellipticity of each speckle in the first zero-level region.

[0108] In this embodiment of the application, after the left infrared camera module acquires the first speckle image, it can extract the following from the first speckle image: Figure 4 The first zero-level region of interest is shown, then grayscale conversion is performed on this first zero-level region of interest, followed by denoising, then binarization, then 8-connectivity is used to connect the image, and the Canny edge algorithm is used to find the boundaries of the connected components in the binarized image. (Refer to...) Figure 6 , Figure 6This is a schematic diagram illustrating the processing of the first zero-level region according to an embodiment of this application. After processing the first zero-level region, all speckle patterns within the first zero-level region can be identified. Then, the centroid coordinates of each speckle pattern in the first zero-level region are obtained to form a centroid coordinate set. The speckle area and speckle ellipticity of each speckle pattern in the first zero-level region are calculated. The average speckle area of ​​the speckle patterns in the first zero-level region is calculated based on the speckle area of ​​each speckle pattern in the first zero-level region; and the average speckle ellipticity of the speckle patterns in the first zero-level region is calculated based on the speckle ellipticity of each speckle pattern in the first zero-level region. (Refer to...) Figure 7 , Figure 7 This is provided by the embodiments of this application. Figure 6 The diagram shows the data corresponding to each speckle in the first zero-level region.

[0109] It should be noted that, in the embodiments of this application, operations such as extracting the first zero-level region, processing the first zero-level region, calculating the centroid coordinate set, average speckle area, and average speckle ellipticity of the speckles in the first zero-level region can all be implemented through pre-set programming. That is, after the left infrared camera module acquires the first speckle image, it can directly output the centroid coordinate set, average speckle area, and average speckle ellipticity of the speckles in the first zero-level region.

[0110] It should be noted that a qualified speckle projection module projects speckle spots in an image that are approximately circular, achieving a circle size greater than 0.9. Simultaneously, when projecting at a fixed distance, changing the sharpness of the receiving camera module (e.g., the left and right infrared camera modules) will cause the speckle spots to become blurred and larger, and the calculated speckle area will also increase; that is, the speckle area can indirectly characterize sharpness. When the centroid position of all speckle spots changes, it indicates that the baseline position of the module has changed. Furthermore, when the camera module assembly is tilted or the lens and image sensor are tilted, the speckle spots will become elliptical, and all speckle spots will tilt in the same direction. Based on these principles, the damage to the speckle projection module and the left and right infrared camera modules after use, as well as whether the corresponding calibration extrinsic parameters have changed, can be determined by calculating the centroid coordinate set, speckle area, and speckle ellipticity.

[0111] Reference Figure 8 , Figure 8 This is a flowchart of the steps for calculating the second array provided in the embodiments of this application, namely, the flowchart of the steps for calculating the second centroid coordinate set, the average area of ​​the second speckle, and the average ellipticity of the second zero-order region speckles, including but not limited to steps S801 to S805.

[0112] Step S801: Process the second zero-level region to identify all speckles in the second zero-level region;

[0113] Step S802, the centroid coordinates of each speckle point in the second zero-level region are obtained, and a second centroid coordinate set is obtained;

[0114] Step S803, the speckle area and the speckle ellipticity of each speckle point in the second zero-level region are calculated;

[0115] Step S804, the second speckle area average is calculated according to the speckle area of each speckle point in the second zero-level region;

[0116] Step S805, the second speckle ellipticity average is calculated according to the speckle ellipticity of each speckle point in the second zero-level region.

[0117] In the embodiment of the application, the second zero-level region extracted by the right infrared camera module also contains a plurality of speckle points, and the centroid coordinates, speckle areas and speckle ellipticities of the plurality of speckle points need to be calculated respectively, and then the centroid coordinate set, the speckle area average and the speckle ellipticity average of the speckle points in the second zero-level region are calculated. The calculation process and method are the same as the calculation process and method of the centroid coordinate set, the speckle area average and the speckle ellipticity average of the speckle points in the first zero-level region, which will not be described here.

[0118] Step S204, whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters are changed is judged according to the centroid coordinate set, the speckle area average and the speckle ellipticity average of the speckle points in the zero-level region and the corresponding preset specifications. The preset specifications are standard data of the binocular structured light camera module when it is shipped based on the evaluation device, and the preset specifications include a centroid standard coordinate set, a speckle area threshold and a speckle ellipticity threshold.

[0119] In the embodiment of the application, the standard data of the standard binocular structured light camera module when it is shipped is obtained by the evaluation device as shown in the Figure 1 The standard data obtained includes the speckle image projected by the standard speckle module collected based on the standard left infrared camera module and the standard right infrared camera module, and the corresponding preset specifications calculated. Specifically, the preset specifications include a first preset specification and a second preset specification. The first preset specification is the standard data of the left infrared camera module when it is shipped obtained by the evaluation device as shown in the Figure 1 The first preset specification includes a first centroid standard coordinate set, a first speckle area threshold and a first speckle ellipticity threshold. The second preset specification is the standard data of the right infrared camera module when it is shipped obtained by the evaluation device as shown in the Figure 1 The second preset specification includes a second centroid standard coordinate set, a second speckle area threshold and a second speckle ellipticity threshold.

[0120] In the embodiments of the present application, when the standard binocular structured light camera is shipped, the standard left infrared camera module, the standard right infrared camera module and the standard speckle projection module are placed at the corresponding positions of the evaluation device shown in the drawings, and then the speckle image projected by the standard speckle projection module is collected by the standard left infrared camera module and the speckle image projected by the standard speckle projection module is collected by the standard right infrared camera module. Then the speckle images are processed and calculated to obtain the corresponding preset specifications. Figure 1

[0121] Exemplarily, the standard left infrared camera module collects a first speckle image projected by the standard speckle projection module, and extracts first zero-level region speckle points, which are all the speckle points in the central region of the first speckle image. Then the centroid coordinates of all the speckle points in the first zero-level region are calculated to obtain a first centroid standard coordinate set; the speckle area and the speckle ellipticity of all the speckle points in the first zero-level region are calculated, and then the average speckle area of all the speckle points in the first zero-level region is calculated to obtain a first speckle area threshold, and the average speckle ellipticity of all the speckle points in the first zero-level region is calculated to obtain a first speckle ellipticity threshold. Similarly, the standard right infrared camera module collects a second speckle image projected by the standard speckle projection module, and extracts second zero-level region speckle points, which are all the speckle points in the central region of the second speckle image. Then the centroid coordinates of all the speckle points in the second zero-level region are calculated to obtain a second centroid standard coordinate set; the speckle area and the speckle ellipticity of all the speckle points in the second zero-level region are calculated, and then the average speckle area of all the speckle points in the second zero-level region is calculated to obtain a second speckle area threshold, and the average speckle ellipticity of all the speckle points in the second zero-level region is calculated to obtain a second speckle ellipticity threshold.

[0122] In the embodiments of the present application, comparing the first array with the first preset specifications and comparing the second array with the second preset specifications can determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters are changed.

[0123] Referring to Figure 9 , Figure 9 is a step flow chart provided by the embodiments of the present application for comparing the first array with the first preset specifications to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters are changed, including but not limited to steps S901 to S907.

[0124] Step S901, determining whether the first centroid coordinate set is consistent with the first centroid standard coordinate set;

[0125] ​Step S902: If the first centroid coordinate set is consistent with the first centroid standard coordinate set, it is determined that the left infrared camera module and the speckle projection module are not damaged and the first extrinsic parameter has not changed. The first extrinsic parameter is the calibration extrinsic parameter corresponding to the left infrared camera module.

[0126] Step S903: If the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, it is determined that the first external parameter has changed;

[0127] Step S904: Determine whether the average area of ​​the first speckle pattern is greater than the threshold value of the first speckle pattern area;

[0128] Step S905: If the average area of ​​the first speckle is not greater than the first speckle area threshold, determine whether the average ellipticity of the first speckle is less than the first speckle ellipticity threshold.

[0129] Step S906: If the average value of the first speckle ellipticity is not less than the first speckle ellipticity threshold, it is determined that neither the left infrared camera module nor the speckle projection module is damaged.

[0130] Step S907: If the average value of the first speckle area is greater than the first speckle area threshold or the average value of the first speckle ellipticity is less than the first speckle ellipticity threshold, it is determined that the left infrared camera module or the speckle projection module is damaged.

[0131] In this embodiment, if the first centroid coordinate set is consistent with the first centroid standard coordinate set, it indicates that the position of each speckle in the first zero-order region of the speckle image acquired by the used left infrared camera module is consistent with the standard data when the left infrared camera module was manufactured. In this case, it can be determined that neither the left infrared camera module nor the speckle projection module is damaged, and the first extrinsic parameter has not changed. If the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, it can be determined that the first extrinsic parameter has changed. In this case, it is further determined whether the average first speckle area is greater than the first speckle area threshold and whether the average first speckle ellipticity is less than the first speckle ellipticity threshold. If the average first speckle area is greater than the first speckle area threshold or the average first speckle ellipticity is less than the first speckle ellipticity threshold, it is determined that the left infrared camera module or the speckle projection module is damaged; if the average first speckle area is not greater than the first speckle area threshold and the average first speckle ellipticity is not less than the first speckle ellipticity threshold, it is determined that neither the left infrared camera module nor the speckle projection module is damaged.

[0132] Reference Figure 10 , Figure 10 This application provides a flowchart of steps for comparing a second array with a second preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters have changed, including but not limited to steps S1001 to S1007.

[0133] In step S1001, it is judged whether the second centroid coordinate set is consistent with the second centroid standard coordinate set.

[0134] In step S1002, if the second centroid coordinate set is consistent with the second centroid standard coordinate set, it is determined that the right infrared camera module and the speckle projection module are not damaged, and the second external parameter is not changed, and the second external parameter is the calibration external parameter corresponding to the right infrared camera module.

[0135] In step S1003, if the second centroid coordinate set is not consistent with the second centroid standard coordinate set, it is determined that the second external parameter is changed.

[0136] In step S1004, it is judged whether the second speckle area average is greater than the second speckle area threshold.

[0137] In step S1005, if the second speckle area average is not greater than the second speckle area threshold, it is judged whether the second speckle ellipticity average is less than the second speckle ellipticity threshold.

[0138] In step S1006, if the second speckle ellipticity average is not less than the second speckle ellipticity threshold, it is determined that the right infrared camera module and the speckle projection module are not damaged.

[0139] In step S1007, if the second speckle area average is greater than the second speckle area threshold or the second speckle ellipticity average is less than the second speckle ellipticity threshold, it is determined that the right infrared camera module or the speckle projection module is damaged.

[0140] In the embodiment of the application, if the second centroid coordinate set is consistent with the second centroid standard coordinate set, it indicates that the position of each speckle point in the second zero-level region of the speckle image collected by the used right infrared camera module is consistent with the standard data when the right infrared camera module is factory-finished, at this time, it can be judged that the right infrared camera module and the speckle projection module are not damaged, and the second external parameter is not changed. If the second centroid coordinate set is not consistent with the second centroid standard coordinate set, it can be judged that the second external parameter is changed. At this time, it is further judged whether the second speckle area average is greater than the second speckle area threshold and whether the second speckle ellipticity average is less than the second speckle ellipticity threshold. If the second speckle area average is greater than the second speckle area threshold or the second speckle ellipticity average is less than the second speckle ellipticity threshold, it is determined that the right infrared camera module or the speckle projection module is damaged; if the second speckle area average is not greater than the second speckle area threshold and the second speckle ellipticity average is not less than the second speckle ellipticity threshold, it is determined that the right infrared camera module and the speckle projection module are not damaged.

[0141] Referring to Figure 11 , Figure 11 is an evaluation flowchart of the binocular structured light camera module provided by the embodiment of the application, including but not limited to steps S1101 to S1118.

[0142] Step S1101, a first speckle image projected by the speckle projection module is collected by the left infrared camera module, and first zero-level region speckle points are extracted, the first zero-level region speckle points being all speckle points in a central region of the speckle image;

[0143] Step S1102, a first array is calculated, the first array including a first centroid coordinate set of the first zero-level region speckle points, a first speckle area mean value and a first speckle ellipticity mean value;

[0144] Step S1103, whether the first centroid coordinate set is consistent with a first centroid standard coordinate set is judged;

[0145] Step S1104, if the first centroid coordinate set is consistent with the first centroid standard coordinate set, it is determined that the left infrared camera module and the speckle projection module are not damaged, and a first extrinsic parameter is not changed, the first extrinsic parameter being a calibration extrinsic parameter corresponding to the left infrared camera module;

[0146] Step S1105, if the first centroid coordinate set is not consistent with the first centroid standard coordinate set, it is determined that the first extrinsic parameter is changed;

[0147] Step S1106, whether the first speckle area mean value is greater than a first speckle area threshold value is judged;

[0148] Step S1107, if the first speckle area mean value is not greater than the first speckle area threshold value, whether the first speckle ellipticity mean value is less than a first speckle ellipticity threshold value is judged;

[0149] Step S1108, if the first speckle ellipticity mean value is not less than the first speckle ellipticity threshold value, it is determined that the left infrared camera module and the speckle projection module are not damaged;

[0150] Step S1109, if the first speckle area mean value is greater than the first speckle area threshold value or the first speckle ellipticity mean value is less than the first speckle ellipticity threshold value, it is determined that the left infrared camera module or the speckle projection module is damaged;

[0151] Step S1110, a second speckle image projected by the speckle projection module is collected by the right infrared camera module, and second zero-level region speckle points are extracted, the second zero-level region speckle points being all speckle points in a central region of the speckle image;

[0152] Step S1111, a second array is calculated, the second array including a second centroid coordinate set of the second zero-level region speckle points, a second speckle area mean value and a second speckle ellipticity mean value;

[0153] Step S1112, whether the second centroid coordinate set is consistent with a second centroid standard coordinate set is judged;

[0154] Step S1113, if the second centroid coordinate set is consistent with the second centroid standard coordinate set, it is determined that the right infrared camera module and the speckle projection module are not damaged, and the second extrinsic parameter is not changed, and the second extrinsic parameter is the calibration extrinsic parameter corresponding to the right infrared camera module.

[0155] Step S1114, if the second centroid coordinate set is not consistent with the second centroid standard coordinate set, it is determined that the second extrinsic parameter is changed.

[0156] Step S1115, it is judged whether the second speckle area average is greater than the second speckle area threshold value.

[0157] Step S1116, if the second speckle area average is not greater than the second speckle area threshold value, it is judged whether the second speckle ellipticity average is less than the second speckle ellipticity threshold value.

[0158] Step S1117, if the second speckle ellipticity average is not less than the second speckle ellipticity threshold value, it is determined that the right infrared camera module and the speckle projection module are not damaged.

[0159] Step S1118, if the second speckle area average is greater than the second speckle area threshold value or the second speckle ellipticity average is less than the second speckle ellipticity threshold value, it is determined that the right infrared camera module or the speckle projection module is damaged.

[0160] In the embodiment of the application, if the centroid coordinate set is consistent with the centroid standard coordinate set, the speckle area threshold value and the speckle ellipticity are also certainly consistent with the preset specifications, at this time, the camera module can be directly determined to be not damaged, and the calibration extrinsic parameter is not changed. If the centroid coordinate set is not consistent with the centroid standard coordinate set, it can be first determined that the calibration extrinsic parameter is changed. At this time, it is also necessary to further judge whether the speckle area threshold value and the speckle ellipticity are consistent with the preset specifications. If the speckle area or the speckle ellipticity is not consistent with the preset specifications, it is determined that the corresponding camera module is also damaged. If the speckle area and the speckle ellipticity are both consistent with the preset specifications, it is determined that the corresponding module is not damaged. That is, by Figure 1 The evaluation device shown in the figure and the evaluation method provided by the embodiment of the application can evaluate the reason for the decrease in the depth map precision. The reason is that the calibration extrinsic parameter (the first extrinsic parameter or the second extrinsic parameter) is changed, or the calibration extrinsic parameter is changed and the camera module is also damaged.

[0161] It should be noted that since the left infrared camera module and the right infrared camera module both collect the speckle image emitted by the same speckle projection module. Therefore, the judgment results of the left infrared camera module and the right infrared camera module are combined to determine the final reason for the decrease in the depth map precision.

[0162] Specifically, in the embodiment of the application, each module of a used binocular structured light camera is evaluated by Figure 1The evaluation device shown performs evaluation, and the combination of the determination result of the left infrared camera module and the determination result of the right infrared camera module can obtain the following eight reasons for the reduced depth map accuracy:

[0163] Reason 1: The first extrinsic parameter (the calibration extrinsic parameter corresponding to the left infrared camera module) changes:

[0164] When the second centroid coordinate set is consistent with the second centroid standard coordinate set, the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area average is not greater than the first speckle area threshold and the first speckle ellipticity average is not less than the first speckle ellipticity threshold, it is determined that the first extrinsic parameter changes.

[0165] Reason 2: The first extrinsic parameter changes and the left infrared camera module is damaged:

[0166] When the second centroid coordinate set is consistent with the second centroid standard coordinate set, the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area average is greater than the first speckle area threshold or the first speckle ellipticity average is less than the first speckle ellipticity threshold, it is determined that the first extrinsic parameter changes and the left infrared camera module is damaged.

[0167] Reason 3: The second extrinsic parameter (the calibration extrinsic parameter corresponding to the right infrared camera module) changes:

[0168] When the first centroid coordinate set is consistent with the first centroid standard coordinate set, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area average is not greater than the second speckle area threshold and the second speckle ellipticity average is not less than the second speckle ellipticity threshold, it is determined that the second extrinsic parameter changes.

[0169] Reason 4: The second extrinsic parameter changes and the right infrared camera module is damaged:

[0170] When the first centroid coordinate set is consistent with the first centroid standard coordinate set, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area average is greater than the second speckle area threshold or the second speckle ellipticity average is less than the second speckle ellipticity threshold, it is determined that the second extrinsic parameter changes and the right infrared camera module is damaged.

[0171] Reason 5: The first extrinsic parameter and the second extrinsic parameter both change:

[0172] When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area average is not greater than the first speckle area threshold and the first speckle ellipticity average is not less than the first speckle ellipticity threshold, and the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area average is not greater than the second speckle area threshold and the second speckle ellipticity average is not less than the second speckle ellipticity threshold, it is determined that the first extrinsic parameter and the second extrinsic parameter both change.

[0173] Reason 6: both the first extrinsic parameter and the second extrinsic parameter are changed and the left infrared camera module is damaged:

[0174] When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is greater than the first speckle area threshold value or the first speckle ellipticity mean value is less than the first speckle ellipticity threshold value, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area mean value is not greater than the second speckle area threshold value and the second speckle ellipticity mean value is not less than the second speckle ellipticity threshold value, it is determined that both the first extrinsic parameter and the second extrinsic parameter are changed and the left infrared camera module is damaged.

[0175] Reason 7: both the first extrinsic parameter and the second extrinsic parameter are changed and the right infrared camera module is damaged:

[0176] When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is not greater than the first speckle area threshold value and the first speckle ellipticity mean value is not less than the first speckle ellipticity threshold value, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area mean value is greater than the second speckle area threshold value or the second speckle ellipticity mean value is less than the second speckle ellipticity threshold value, it is determined that both the first extrinsic parameter and the second extrinsic parameter are changed and the right infrared camera module is damaged.

[0177] Reason 8: both the first extrinsic parameter and the second extrinsic parameter are changed and there are at least two camera modules damaged:

[0178] When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the first speckle area mean value is greater than the first speckle area threshold value or the first speckle ellipticity mean value is less than the first speckle ellipticity threshold value, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the second speckle area mean value is not greater than the second speckle area threshold value and the second speckle ellipticity mean value is not less than the second speckle ellipticity threshold value, it is determined that both the first extrinsic parameter and the second extrinsic parameter are changed and there are at least two camera modules damaged in the binocular structured light camera module.

[0179] In the embodiments of the present application, if the first speckle area mean value is greater than the first speckle area threshold value or the first speckle ellipticity mean value is less than the first speckle ellipticity threshold value, it can be determined that the left infrared camera module is damaged or the speckle projection module is damaged; if the second speckle area mean value is not greater than the second speckle area threshold value and the second speckle ellipticity mean value is not less than the second speckle ellipticity threshold value, it can be determined that the right infrared camera module is damaged or the speckle projection module is damaged. Therefore, after the combination of the two, it can be determined that at least two camera modules are damaged. That is, it is possible that the right infrared camera module is damaged and the speckle projection module is damaged, or the left infrared camera module is damaged and the speckle projection module is damaged, or the left infrared camera module is damaged and the right infrared camera module is damaged, or the right infrared camera module, the speckle projection module and the left infrared camera module are all damaged.

[0180] In the embodiments of the present application, for reason 8, that is, at least two camera modules are damaged, it can be considered that the camera module is severely damaged and the maintenance cost is high, at this time, the binocular structured light camera can be directly disassembled and recycled without maintenance on the original basis.

[0181] It can be understood that for reason 8, in order to reuse the undamaged modules, the first centroid coordinate set of the first zero-level area speckle point, the first speckle area mean value and the first speckle ellipticity mean value, and the second centroid coordinate set of the second zero-level area speckle point, the second speckle area mean value and the second speckle ellipticity mean value are compared. If both have the same regularity, such as the consistent elliptical direction of the speckle point ellipse, it can be further determined that the speckle projection module is not damaged, and the left infrared camera module and the right infrared camera module are both damaged.

[0182] It can be understood that for reasons 1 to 7, since the specific reason for the decrease in depth map accuracy is evaluated, the corresponding damaged module can be directly guided to be repaired or the corresponding changed calibration parameter can be corrected and modified.

[0183] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0184] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps or different steps.

[0185] The apparatus embodiments described above are merely exemplary, and the units described as separate units can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

[0186] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combinations thereof.

[0187] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, such that the embodiments of the application described herein can be carried out in other than the order discussed herein without departing from the scope of the application. Further, the terms "comprise" and "comprising" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises a list of steps or units can not necessarily comprise only those steps or units but can include other not expressly listed or inherent steps or units. In other words, reference to a "comprising" step or unit does not exclude the presence of other steps or units.

[0188] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0189] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. The division of the units is merely logical function division. There can be another division manner for the actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0190] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0191] The preferred embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the present application. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and spirit of the present application should be within the scope of the present application.

Claims

1. An evaluation method for a binocular structured light camera module, performed using an evaluation device, the evaluation device comprising: The first fixing device includes a substrate support and a left infrared camera module, a speckle projection module and a right infrared camera module mounted on the substrate support according to a preset baseline. The speckle projection module is located at the center of the left infrared camera module and the right infrared camera module. A reflecting device is horizontally positioned at a specific distance above the first fixed device to reflect the light beam projected by the speckle projection module. The evaluation method includes: The used binocular structured light camera module is installed in the corresponding position of the evaluation device. The binocular structured light camera module includes a speckle projection module, a left infrared camera module, and a right infrared camera module. The camera module acquires the speckle image projected by the speckle projection module and extracts the zero-order region speckles, which are all speckles in the central region of the speckle image. The camera module includes the left infrared camera module and the right infrared camera module. The centroid coordinate set, mean speckle area, and mean speckle ellipticity of the zero-order region speckles were calculated respectively. Based on the centroid coordinate set, average speckle area, and average speckle ellipticity of the zero-order region speckles, and the corresponding preset specifications, it is determined whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters have changed. The preset specifications are the standard data of the binocular structured light camera module at the time of manufacture obtained by the evaluation device, and the preset specifications include the centroid standard coordinate set, speckle area threshold, and speckle ellipticity threshold.

2. The method according to claim 1, characterized in that, The step of acquiring the speckle image projected by the speckle projection module through the camera module and extracting speckle points in the zero-order region includes: The left infrared camera module acquires the first speckle image projected by the speckle projection module, and extracts the first zero-order region speckle, which is all the speckle in the central region of the speckle image. The second speckle image projected by the speckle projection module is acquired by the right infrared camera module, and the second zero-order region speckle spots are extracted. The second zero-order region speckle spots are all the speckle spots in the central region of the speckle image.

3. The method according to claim 2, characterized in that, The calculation of the centroid coordinates, mean speckle area, and mean speckle ellipticity of all speckles in the zero-order region includes: Calculate the first array, which includes the first centroid coordinate set of the speckle in the first zero-order region, the mean area of ​​the first speckle, and the mean ellipticity of the first speckle; Calculate the second array, which includes the second centroid coordinate set of the speckle in the second zero-order region, the mean area of ​​the second speckle, and the mean ellipticity of the second speckle.

4. The method according to claim 3, characterized in that, The calculation of the first array includes: The first zero-level region is processed to identify all speckles within the first zero-level region; Obtain the centroid coordinates of each speckle in the first zero-level region to obtain the first centroid coordinate set; Calculate the speckle area and speckle ellipticity of each speckle in the first zero-order region; The average value of the first speckle area is calculated based on the speckle area of ​​each speckle in the first zero-level region. The mean value of the first speckle ellipticity is calculated based on the speckle ellipticity of each speckle in the first zero-level region.

5. The method according to claim 3, characterized in that, The calculation of the second array includes: The second zero-level region is processed to identify all speckles within the second zero-level region; Obtain the centroid coordinates of each speckle in the second zero-level region to obtain the second centroid coordinate set; Calculate the speckle area and speckle ellipticity of each speckle in the second zero-order region; The average value of the second speckle area is calculated based on the speckle area of ​​each speckle in the second zero-level region; The mean value of the second speckle ellipticity is calculated based on the speckle ellipticity of each speckle in the second zero-level region.

6. The method according to claim 3, characterized in that, The step of determining whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters have changed based on the centroid coordinate set, average speckle area, average speckle ellipticity, and corresponding preset specifications of the zero-order region speckles includes: Obtain a first preset specification, which is the standard data of the left infrared camera module at the time of its manufacture obtained by the evaluation device. The first preset specification includes a first centroid standard coordinate set, a first speckle area threshold, and a first speckle ellipticity threshold. Obtain a second preset specification, which is the standard data of the right infrared camera module at the time of its manufacture obtained by the evaluation device. The second preset specification includes a second centroid standard coordinate set, a second speckle area threshold, and a second speckle ellipticity threshold. The first array is compared with the first preset specification and the second array is compared with the second preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters have changed.

7. The method according to claim 6, characterized in that, Comparing the first array with the first preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters have changed includes: Determine whether the first centroid coordinate set is consistent with the first centroid standard coordinate set; If the first centroid coordinate set is consistent with the first centroid standard coordinate set, it is determined that the left infrared camera module and the speckle projection module are not damaged, and the first extrinsic parameter has not changed. The first extrinsic parameter is the calibration extrinsic parameter corresponding to the left infrared camera module. If the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, it is determined that the first external parameter has changed; Determine whether the average area of ​​the first speckle pattern is greater than the first speckle area threshold and whether the average ellipticity of the first speckle pattern is less than the first speckle ellipticity threshold; If the average first speckle area is greater than the first speckle area threshold or the average first speckle ellipticity is less than the first speckle ellipticity threshold, it is determined that the left infrared camera module or the speckle projection module is damaged. If the average area of ​​the first speckle is not greater than the first speckle area threshold and the average ellipticity of the first speckle is not less than the first speckle ellipticity threshold, it is determined that neither the left infrared camera module nor the speckle projection module is damaged.

8. The method according to claim 7, characterized in that, The second array is compared with the second preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters have changed, including: Determine whether the second centroid coordinate set is consistent with the second centroid standard coordinate set; If the second centroid coordinate set is consistent with the second centroid standard coordinate set, it is determined that the right infrared camera module and the speckle projection module are not damaged, and the second extrinsic parameter has not changed. The second extrinsic parameter is the calibration extrinsic parameter corresponding to the right infrared camera module. If the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, it is determined that the second external parameter has changed; Determine whether the average area of ​​the second speckle pattern is greater than the second speckle area threshold and whether the average ellipticity of the second speckle pattern is less than the second speckle ellipticity threshold; If the average value of the second speckle area is greater than the second speckle area threshold or the average value of the second speckle ellipticity is less than the second speckle ellipticity threshold, it is determined that the right infrared camera module or the speckle projection module is damaged. If the average area of ​​the second speckle is not greater than the second speckle area threshold and the average ellipticity of the second speckle is not less than the second speckle ellipticity threshold, it is determined that neither the right infrared camera module nor the speckle projection module is damaged.

9. The method according to claim 8, characterized in that, The step of comparing the first array with the first preset specification and comparing the second array with the second preset specification to determine whether the binocular structured light camera module is damaged and whether the camera calibration extrinsic parameters have changed includes: Compare the first array with the first preset specification and compare the second array with the second preset specification; When the first centroid coordinate set is consistent with the first centroid standard coordinate set and the second centroid coordinate set is consistent with the second centroid standard coordinate set, it is determined that the binocular structured light camera module is not damaged and the calibration extrinsic parameters have not changed. When the first centroid coordinate set is consistent with the first centroid standard coordinate set, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the average value of the second speckle area is not greater than the second speckle area threshold and the average value of the second speckle ellipticity is not less than the second speckle ellipticity threshold, it is determined that the second extrinsic parameter has changed. When the first centroid coordinate set is consistent with the first centroid standard coordinate set, the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the average value of the second speckle area is greater than the second speckle area threshold or the average value of the second speckle ellipticity is less than the second speckle ellipticity threshold, it is determined that the right infrared camera module is damaged and the second external parameter has changed. When the second centroid coordinate set is consistent with the second centroid standard coordinate set, the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the average value of the first speckle area is not greater than the first speckle area threshold and the average value of the first speckle ellipticity is not less than the first speckle ellipticity threshold, it is determined that the first extrinsic parameter has changed. When the second centroid coordinate set is consistent with the second centroid standard coordinate set, the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the average value of the first speckle area is greater than the first speckle area threshold or the average value of the first speckle ellipticity is less than the first speckle ellipticity threshold, it is determined that the left infrared camera module is damaged and the first external parameter has changed. When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the average value of the first speckle area is not greater than the first speckle area threshold and the average value of the first speckle ellipticity is not less than the first speckle ellipticity threshold, and the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the average value of the second speckle area is not greater than the second speckle area threshold and the average value of the second speckle ellipticity is not less than the second speckle ellipticity threshold, it is determined that both the first extrinsic parameter and the second extrinsic parameter have changed. When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the average value of the first speckle area is not greater than the first speckle area threshold and the average value of the first speckle ellipticity is not less than the first speckle ellipticity threshold, and the second centroid coordinate set is inconsistent with the second centroid standard coordinate set and the average value of the second speckle area is greater than the second speckle area threshold or the average value of the second speckle ellipticity is less than the second speckle ellipticity threshold, it is determined that both the first extrinsic parameter and the second extrinsic parameter have changed and the right infrared camera module has been damaged. When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the average value of the first speckle area is greater than the first speckle area threshold or the average value of the first speckle ellipticity is less than the first speckle ellipticity threshold, and the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the average value of the second speckle area is not greater than the second speckle area threshold and the average value of the second speckle ellipticity is not less than the second speckle ellipticity threshold, it is determined that both the first extrinsic parameter and the second extrinsic parameter have changed and at least two camera modules in the binocular structured light camera module have been damaged. When the first centroid coordinate set is inconsistent with the first centroid standard coordinate set, and the average value of the first speckle area is greater than the first speckle area threshold or the average value of the first speckle ellipticity is less than the first speckle ellipticity threshold, and the second centroid coordinate set is inconsistent with the second centroid standard coordinate set, and the average value of the second speckle area is not greater than the second speckle area threshold and the average value of the second speckle ellipticity is not less than the second speckle ellipticity threshold, it is determined that both the first extrinsic parameter and the second extrinsic parameter have changed and the left infrared camera module has been damaged.

Images