A Structured Light 3D Camera Calibration Method and Device

Abstract

The embodiments of the present application provide a structured light 3D camera calibration method and apparatus, which relate to the field of camera imaging technology. The above method includes: obtaining a first image, a second image, a third image, a fourth image, and three-dimensional coordinates; detecting the calibration point coordinates in the first image and the second image as the first coordinate and the second coordinate, performing structured light decoding on the third image and the fourth image to obtain the third coordinate and the fourth coordinate; calculating the main camera parameters, the auxiliary camera parameters, and the projector parameters based on the conversion relationship; calculating the calculated coordinates by using the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, and the feed distance; calculating the difference between the calculated coordinates and the measured coordinates, and optimizing the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle to obtain the optimized parameters. By applying the structured light 3D camera calibration solution provided by the embodiments of the present application, the parameters of the main camera and the projector in the structured light 3D camera can be accurately obtained.

Description

Technical Field

[0001] The present application relates to the field of camera imaging technology, and particularly to a structured light 3D camera calibration method and device. Background Art

[0002] A structured light 3D camera can be used to reconstruct a three-dimensional model of a target object. Before applying the structured light 3D camera for model reconstruction, it is usually necessary to calculate the internal parameters and external parameters of the main camera and the projector included in the structured light 3D camera, so as to achieve the above-mentioned purpose of model reconstruction based on the internal parameters and external parameters of the main camera and the projector and the information of the structured light projected on the surface of the target object by the projector.

[0003] In order to more accurately reconstruct the three-dimensional model of the target object, a structured light 3D camera calibration scheme is required to obtain relatively accurate parameters of the main camera and the projector in the structured light 3D camera. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a structured light 3D camera calibration scheme to obtain relatively accurate parameters of the main camera and the projector in the structured light 3D camera. The specific technical solutions are as follows:

[0005] In a first aspect, the embodiments of the present application provide a structured light 3D camera calibration method, and the method includes:

[0006] For calibration plates that are in different parallel positions and move translationally along a fixed feed direction, first images and second images of the calibration plates at the parallel positions collected by the main camera and a preset auxiliary camera in the structured light 3D camera are respectively obtained, and third images and fourth images of the calibration plates at the parallel positions collected by the main camera and the auxiliary camera are respectively obtained when the projector in the structured light 3D camera projects structured light onto the calibration plates, and three-dimensional coordinates of each calibration point in the calibration plates at the parallel positions in a preset three-dimensional coordinate system are obtained, where the calibration plates at different parallel positions are parallel to each other and have different heights;

[0007] The coordinates of each calibration point in each of the first images and the second images are respectively detected as the first coordinates and the second coordinates of each calibration point in the fields of view of the main camera and the auxiliary camera, and structured light decoding is respectively performed on each of the third images and the fourth images to obtain the third coordinates and the fourth coordinates of each calibration point in the field of view of the projector;

[0008] Based on the conversion relationships between the first coordinates, the second coordinates, the reference coordinates and the three-dimensional coordinates respectively, the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera and the projector parameters of the projector are calculated, where the reference coordinates include: the third coordinates and / or the fourth coordinates;

[0009] Using the main camera parameters, auxiliary camera parameters, projector parameters, feed angle, and feed distance respectively, calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, auxiliary camera, and projector as the calculated coordinates. Herein, the feed angle reflects the angle between the feed direction and the normal direction of the calibration plate, and the feed distance is the distance between each parallel position and the lowermost parallel position.

[0010] Calculate the differences between the calculated coordinates and the measured coordinates, and optimize the main camera parameters, auxiliary camera parameters, projector parameters, and feed angle based on the principle of reducing the calculated differences to obtain the optimized parameters of the main camera and projector. Herein, the measured coordinates include the first coordinate, second coordinate, third coordinate, and fourth coordinate obtained when the calibration plate is at different parallel positions.

[0011] In one embodiment of the present application, calculating the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera, and the projector parameters of the projector respectively based on the conversion relationships between the first coordinate, the second coordinate, the reference coordinate, and the three-dimensional coordinate includes:

[0012] Calculating the main camera parameters of the main camera based on the conversion relationship between the first coordinate and the three-dimensional coordinate;

[0013] Calculating the auxiliary camera parameters of the auxiliary camera based on the conversion relationship between the second coordinate and the three-dimensional coordinate;

[0014] Calculating the projector parameters of the projector based on the conversion relationship between the reference coordinate and the three-dimensional coordinate.

[0015] In one embodiment of the present application, calculating the main camera parameters of the main camera based on the conversion relationship between the first coordinate and the three-dimensional coordinate includes:

[0016] Using the obtained first coordinates and three-dimensional coordinates, calculate the first transformation matrix between the first coordinate and the three-dimensional coordinate, and decompose the first transformation matrix to obtain the internal parameter matrix and external parameter matrix of the main camera as the main camera parameters of the main camera.

[0017] In one embodiment of the present application, using the main camera parameters, auxiliary camera parameters, projector parameters, feed angle, and feed distance respectively to calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, auxiliary camera, and projector as the calculated coordinates includes:

[0018] Using the main camera parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the main camera;

[0019] Using the auxiliary camera parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the auxiliary camera;

[0020] Using the projector parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the projector, and use the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the main camera, the auxiliary camera, and the projector as the calculation coordinates.

[0021] In an embodiment of the present application, the calculating the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively by using the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, and the feed distance as the calculation coordinates includes:

[0022] Using the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, the feed distance, the first distortion parameter, the second distortion parameter, and the third distortion parameter respectively to calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively as the calculation coordinates, where the first distortion parameter, the second distortion parameter, and the third distortion parameter are the distortion parameters of the main camera, the auxiliary camera, and the projector respectively;

[0023] Calculating the differences between each calculated coordinate and the measured coordinate, and optimizing the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle based on the principle of reducing the calculated differences to obtain the optimized parameters of the main camera and the projector, includes:

[0024] Calculating the differences between each calculated coordinate and the measured coordinate, and optimizing the main camera parameters, the auxiliary camera parameters, the projector parameters, the first distortion parameter, the second distortion parameter, the third distortion parameter, and the feed angle based on the principle of reducing the calculated differences to obtain the optimized parameters of the main camera and the projector.

[0025] In an embodiment of the present application, the calculating the differences between each calculated coordinate and the measured coordinate includes:

[0026] Calculate the difference e between each calculated coordinate and the measured coordinate according to the following formula:

[0027]

[0028] Wherein, the θ represents the first included angle, and the represents the second included angle, the i represents the i-th parallel position, and the N img represents the total number of parallel positions, the j represents the j-th calibration point in the calibration board, and the N pt represents the total number of calibration points in the calibration board, the d i represents the moving distance between the calibration board at the i-th parallel position and the lowermost parallel position. When n = 1, the p nij represents the first coordinate of the j-th calibration point in the calibration board at the i-th parallel position, and the K n represents the internal parameter matrix of the main camera, the D n represents the distortion coefficient of the main camera, and the T n represents the external parameter matrix of the main camera. When n = 2, the p nij represents the second coordinate of the j-th calibration point in the calibration board at the i-th parallel position, and the K n represents the internal parameter matrix of the auxiliary camera, the D n represents the distortion coefficient of the auxiliary camera, and the T n represents the external parameter matrix of the auxiliary camera. When n = 3 or n = 4, the K n represents the internal parameter matrix of the projector, the D n represents the distortion coefficient of the projector, and the T n represents the external parameter matrix of the projector. When n = 3, the p nij represents the third coordinate of the j-th calibration point in the calibration board at the i-th parallel position. When n = 4, the p nij represents the fourth coordinate of the j-th calibration point in the calibration board at the i-th parallel position, and the X ij and Y ij respectively represent the X-axis coordinate and Y-axis coordinate of the j-th calibration point in the three-dimensional coordinate system when the calibration board is at the i-th parallel position.

[0029] In an embodiment of the present application, the calibration board is located on a lifting table, and the lifting table is used to adjust the parallel position of the calibration board by moving along the feeding direction.

[0030] In an embodiment of the present application, the steps of respectively obtaining the first image and the second image of the calibration board at this parallel position collected by the main camera and the preset auxiliary camera in the structured light 3D camera include:

[0031] When the projector in the structured light 3D camera projects white light onto the calibration board, the first image and the second image of the calibration board at this parallel position collected by the main camera and a preset auxiliary camera in the structured light 3D camera are obtained respectively.

[0032] In an embodiment of the present application, the preset three-dimensional coordinate system is established based on the position where the calibration board is located.

[0033] In a second aspect, an embodiment of the present application further provides a calibration device for a structured light 3D camera, and the device includes:

[0034] An image acquisition module, configured to, for a calibration board that moves translationally along a fixed feeding direction and is in different parallel positions, respectively obtain the first image and the second image of the calibration board at this parallel position collected by the main camera and a preset auxiliary camera in the structured light 3D camera, and when the projector in the structured light 3D camera projects structured light onto the calibration board, respectively obtain the third image and the fourth image of the calibration board at this parallel position collected by the main camera and the auxiliary camera, and obtain the three-dimensional coordinates of each calibration point on the calibration board at this parallel position in a preset three-dimensional coordinate system, wherein the calibration boards in different parallel positions are parallel to each other and have different heights;

[0035] An image processing module, configured to respectively detect the coordinates of each calibration point in each of the first image and the second image as the first coordinate and the second coordinate of each calibration point in the fields of view of the main camera and the auxiliary camera, and respectively perform structured light decoding on each of the third image and the fourth image to obtain the third coordinate and the fourth coordinate of each calibration point in the field of view of the projector;

[0036] A parameter calculation module, configured to respectively calculate the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera, and the projector parameters of the projector based on the conversion relationships between the first coordinate, the second coordinate, the reference coordinate and the three-dimensional coordinate, wherein the reference coordinate includes: the third coordinate and / or the fourth coordinate;

[0037] A coordinate calculation module, configured to respectively use the main camera parameters, the auxiliary camera parameters, the projector parameters, the feeding angle and the feeding distance to calculate the coordinates of the calibration points on the calibration board in different parallel positions in the fields of view of the main camera, the auxiliary camera and the projector as the calculated coordinates, wherein the feeding angle reflects: the included angle between the feeding direction and the normal direction of the calibration board, and the feeding distance is: the distance between each parallel position and the lowermost parallel position;

[0038] A parameter optimization module, which is used to calculate the differences between the respective calculated coordinates and the measured coordinates, and optimize the main camera parameters, auxiliary camera parameters, projector parameters, and feed angle based on the principle of reducing the calculated differences, so as to obtain the optimized parameters of the main camera and the projector. Among them, the measured coordinates include: the first coordinate, the second coordinate, the third coordinate, and the fourth coordinate obtained when the calibration plate is in different parallel positions.

[0039] In an embodiment of the present application, the parameter calculation module includes:

[0040] A first parameter calculation sub-module, which is used to calculate the main camera parameters of the main camera based on the conversion relationship between the first coordinate and the three-dimensional coordinate;

[0041] A second parameter calculation sub-module, which is used to calculate the auxiliary camera parameters of the auxiliary camera based on the conversion relationship between the second coordinate and the three-dimensional coordinate;

[0042] A third parameter calculation sub-module, which is used to calculate the projector parameters of the projector based on the conversion relationship between the reference coordinate and the three-dimensional coordinate.

[0043] In an embodiment of the present application, the first parameter calculation sub-module is specifically used for:

[0044] Using the obtained respective first coordinates and three-dimensional coordinates, calculate the first transformation matrix between the first coordinate and the three-dimensional coordinate, and decompose the first transformation matrix to obtain the internal parameter matrix and external parameter matrix of the main camera as the main camera parameters of the main camera.

[0045] In an embodiment of the present application, the coordinate calculation module is specifically used for:

[0046] Using the main camera parameters, feed angle, and feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the main camera;

[0047] Using the auxiliary camera parameters, feed angle, and feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the auxiliary camera;

[0048] Using the projector parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point in the calibration plate at different parallel positions to obtain the coordinates of each calibration point in the calibration plate at different parallel positions in the field of view of the projector. Use the coordinates of each calibration point in the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector as the calculated coordinates.

[0049] In one embodiment of the present application, the coordinate calculation module is specifically configured to:

[0050] Respectively use the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, the feed distance, the first distortion parameter, the second distortion parameter, and the third distortion parameter to calculate the coordinates of the calibration points in the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively as the calculated coordinates, where the first distortion parameter, the second distortion parameter, and the third distortion parameter are the distortion parameters of the main camera, the auxiliary camera, and the projector respectively;

[0051] The parameter optimization module is specifically configured to:

[0052] Calculate the difference between each calculated coordinate and the measured coordinate, and optimize the main camera parameters, the auxiliary camera parameters, the projector parameters, the first distortion parameter, the second distortion parameter, the third distortion parameter, and the feed angle based on the principle of reducing the calculated difference to obtain the optimized parameters of the main camera and the projector.

[0053] In one embodiment of the present application, calculating the difference between each calculated coordinate and the measured coordinate includes:

[0054] Calculate the difference e between each calculated coordinate and the measured coordinate according to the following formula:

[0055]

[0056] where, the θ represents the first included angle, the represents the second included angle, the i represents the i-th parallel position, the N img represents the total number of parallel positions, the j represents the j-th calibration point in the calibration plate, the N pt represents the total number of calibration points in the calibration plate, the d i represents the moving distance between the calibration plate at the i-th parallel position and the lowermost parallel position. When n = 1, the p nij represents the first coordinate of the j-th calibration point in the calibration plate at the i-th parallel position, the K n represents the internal parameter matrix of the main camera, the D n represents the distortion coefficient of the main camera, the T nrepresents the external parameter matrix of the main camera. When n = 2, the p nij represents the second coordinate of the j-th calibration point on the calibration board at the i-th parallel position. The K n represents the internal parameter matrix of the auxiliary camera. The D n represents the distortion coefficient of the auxiliary camera. The T n represents the external parameter matrix of the auxiliary camera. When n = 3 or n = 4, the K n represents the internal parameter matrix of the projector. The D n represents the distortion coefficient of the projector. The T n represents the external parameter matrix of the projector. When n = 3, the p nij represents the third coordinate of the j-th calibration point on the calibration board at the i-th parallel position. When n = 4, the p nij represents the fourth coordinate of the j-th calibration point on the calibration board at the i-th parallel position. The X ij and Y ij respectively represent the X-axis coordinate and Y-axis coordinate of the j-th calibration point in the three-dimensional coordinate system when the calibration board is at the i-th parallel position.

[0057] In an embodiment of the present application, the calibration board is located on a lifting table, and the lifting table is used to adjust the parallel position of the calibration board by moving along the feed direction.

[0058] In an embodiment of the present application, the image acquisition module is specifically configured to:

[0059] For the calibration board in different parallel positions during translational motion along a fixed feed direction, when the projector in the structured light 3D camera projects white light onto the calibration board, the first image and the second image of the calibration board at this parallel position collected by the main camera and the preset auxiliary camera in the structured light 3D camera are respectively obtained. When the projector in the structured light 3D camera projects structured light onto the calibration board, the third image and the fourth image of the calibration board at this parallel position collected by the main camera and the auxiliary camera are respectively obtained, and the three-dimensional coordinates of each calibration point on the calibration board at this parallel position in the preset three-dimensional coordinate system are obtained.

[0060] In an embodiment of the present application, the preset three-dimensional coordinate system is established based on the position where the calibration board is located.

[0061] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus;

[0062] The memory is used to store a computer program;

[0063] A processor, when executing a program stored in a memory, implements the method steps described in any one of the first aspects.

[0064] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0065] In a fifth aspect, an embodiment of the present application further provides a computer program product containing instructions, which when running on a computer, causes the computer to execute the method steps described in any one of the above first aspects.

[0066] Beneficial effects of the embodiments of the present application:

[0067] As can be seen from the above, when calibrating a structured light 3D camera by applying the solution provided by the embodiment of the present application, the coordinates of each calibration point in the first image are used as the first coordinates of each calibration point in the field of view of the main camera, the coordinates of each calibration point in the second image are used as the second coordinates of each calibration point in the field of view of the auxiliary camera, and the structured light in the third image and the fourth image is decoded to obtain the third coordinates and the fourth coordinates of each calibration point in the field of view of the projector. Based on the conversion relationship between the first coordinates, the second coordinates, the third coordinates, and / or the fourth coordinates and the three-dimensional coordinates in a preset three-dimensional coordinate system, the main camera parameters, the auxiliary camera parameters, and the projector parameters are calculated. Furthermore, based on these three parameters, the coordinates of each calibration point in the fields of view of the main camera, the auxiliary camera, and the projector are determined as the calculated coordinates, and the difference between each calculated coordinate and the measured coordinate is calculated, and the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle are optimized on the principle of reducing this difference. Since the above calculated coordinates are determined according to the above three target parameters, if the above difference is smaller, it means that the calculated coordinates determined according to the above three target parameters are closer to the coordinates actually measured, and the above three target parameters are more accurate. Therefore, optimizing the above three target parameters and the feed angle on the principle of reducing the above difference can make the parameters of the optimized main camera and projector as accurate as possible. Description of the Drawings

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0069] Figure 1a It is a schematic flowchart of the first structured light 3D camera calibration method provided by the embodiment of the present application;

[0070] Figure 1b A Gray code phase shift pattern provided by an embodiment of the present application;

[0071] Figure 1c A multi-frequency phase shift pattern provided by an embodiment of the present application;

[0072] Figure 2 A schematic flowchart of a second structured light 3D camera calibration method provided by an embodiment of the present application;

[0073] Figure 3 A schematic flowchart of a third structured light 3D camera calibration method provided by an embodiment of the present application;

[0074] Figure 4 A schematic structural diagram of a first structured light 3D camera calibration device provided by an embodiment of the present application;

[0075] Figure 5 A schematic structural diagram of a second structured light 3D camera calibration device provided by an embodiment of the present application;

[0076] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0077] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0078] In order to obtain relatively accurate parameters of the main camera and the projector in the structured light camera, an embodiment of the present application provides a structured light 3D camera calibration method, and this method can be applied to a data processing device for processing structured light 3D camera calibration tasks.

[0079] Next, the structured light 3D camera calibration method provided by the embodiments of the present application will be described through specific embodiments.

[0080] Refer to Figure 1a , a schematic flowchart of a first structured light 3D camera calibration method is provided, and the above method includes the following steps S101-S105.

[0081] Step S101: For calibration plates that are moving translationally along a fixed feed direction and are in different parallel positions, respectively obtain the first image and the second image of the calibration plate at this parallel position collected by the main camera and a preset auxiliary camera in the structured light 3D camera. And when the projector in the structured light 3D camera projects structured light onto the calibration plate, respectively obtain the third image and the fourth image of the calibration plate at this parallel position collected by the main camera and the auxiliary camera, and obtain the three-dimensional coordinates of each calibration point on the calibration plate in a preset three-dimensional coordinate system.

[0082] Among them, the calibration plates in different parallel positions are parallel to each other and have different heights.

[0083] The above-mentioned main camera and projector are components in the structured light 3D camera.

[0084] The above-mentioned auxiliary camera is a camera introduced during the calibration of the structured light 3D camera and does not belong to the components in the structured light 3D camera.

[0085] The above-mentioned feed direction can be the moving direction of the calibration plate preset artificially.

[0086] The above-mentioned structured light is a projection light ray that can identify its own spatial information according to the light intensity characteristics of itself or adjacent structured light.

[0087] The above-mentioned first image and third image are images collected by the main camera, and the above-mentioned second image and fourth image are images collected by the auxiliary camera.

[0088] In an embodiment of the present application, the above-mentioned three-dimensional coordinate system is established based on the position where the calibration plate is located.

[0089] Specifically, a vertex of the calibration plate can be determined as the coordinate origin, the two sides of the calibration plate adjacent to this vertex and the direction away from this vertex are respectively used as the X-axis direction and the Y-axis direction, and the direction perpendicular to the calibration plate is used as the Z-axis direction.

[0090] When establishing a three-dimensional coordinate system based on the position where the calibration plate is located, the position of the calibration point in the calibration plate can be matched with the coordinates of the calibration point on the X-axis and Y-axis, so as to speed up the acquisition of the three-dimensional coordinates of the calibration point, and further improve the calibration efficiency of the structured light 3D camera.

[0091] Specifically, first, the first image, second image, third image, fourth image of the calibration board and the three-dimensional coordinates of each calibration point can be obtained when the calibration board is in a parallel position. Then, manually or using an external device, the calibration board is adjusted to another parallel position, and the first image, second image, third image, fourth image of the calibration board and the three-dimensional coordinates of each calibration point at this other parallel position are obtained again. This cycle continues until the first image, second image, third image, fourth image of the calibration board and the three-dimensional coordinates of each calibration point at all different parallel positions are obtained.

[0092] When the main camera and the auxiliary camera collect the first image and the second image, the projector may not project light onto the calibration board, or may project other light onto the calibration board except for structured light.

[0093] In an embodiment of the present application, when the projector in the structured light 3D camera projects white light onto the calibration board, the first image and the second image of the calibration board collected by the main camera and the preset auxiliary camera in the structured light 3D camera can be obtained respectively.

[0094] Specifically, the above projector can project white light onto the calibration board by projecting a full-white pattern onto the calibration board. The above main camera can collect the image of the calibration board after the projector projects the full-white pattern onto the calibration board as the first image, and the auxiliary camera can collect the image of the calibration board after the projector projects the full-white pattern onto the calibration board as the second image.

[0095] Since the depth of field and the field of view of the projector can be obtained by projecting white light, it can ensure that the calibration board is always within the field of view of the projector, ensuring the normal calibration of the structured light 3D camera. Moreover, when the projector projects white light onto the calibration board, the images collected by the main camera and the auxiliary camera can be clearer.

[0096] When the main camera and the auxiliary camera collect the third image and the fourth image, the projector projects structured light onto the calibration board.

[0097] In an embodiment of the present application, the projector can project a structured light pattern onto the calibration board. The main camera can collect the image of the calibration board after the projector projects the structured light pattern onto the calibration board as the third image, and the auxiliary camera can collect the image of the calibration board after the projector projects the structured light pattern onto the calibration board as the fourth image.

[0098] The above structured light pattern can be, for example, Figure 1b the Gray code phase-shift pattern shown, or can be, for example, Figure 1c the multi-frequency phase-shift pattern shown, etc.

[0099] In an embodiment of the present application, after the main camera and the auxiliary camera collect the images of the calibration board, they can send the collected images to the above data processing device.

[0100] In another embodiment of the present application, the above data processing device may read the images of the calibration board stored in the main camera and the auxiliary camera.

[0101] Step S102: Detect the coordinates of each calibration point in each of the first image and the second image, and use them as the first coordinates and the second coordinates of each calibration point in the fields of view of the main camera and the auxiliary camera respectively. Perform structured light decoding on each of the third image and the fourth image to obtain the third coordinates and the fourth coordinates of each calibration point in the field of view of the projector.

[0102] Among them, the coordinates of the above calibration point in the first image may be the position of the pixel point where the calibration point is located in the first image in the first image.

[0103] For example, if the pixel point where the above calibration point is located in the first image is the pixel point in the 4th pixel row and the 7th pixel column in the first image, then the coordinates of the calibration point in the first image are (7, 4).

[0104] Similarly, the coordinates of the above calibration board in the second image may be the position of the pixel point where the calibration point is located in the second image in the second image.

[0105] The above first coordinates can be understood as the position of the calibration point within the field of view of the main camera, the above second coordinates can be understood as the position of the calibration point within the field of view of the auxiliary camera, and the above third coordinates and fourth coordinates can be understood as the position of the calibration point within the field of view of the projector.

[0106] Specifically, each of the four types of images, namely the first image, the second image, the third image, and the fourth image, is a plurality of images. For each of the multiple first images, the coordinates of each calibration point in the first image can be detected and used as a set of first coordinates of each calibration point in the field of view of the main camera; for each of the multiple second images, the coordinates of each calibration point in the second image can be detected and used as a set of second coordinates of each calibration point in the field of view of the auxiliary camera; for each of the multiple third images, the third image can be subjected to structured light decoding to obtain a set of third coordinates of each calibration point in the field of view of the projector; for each of the multiple fourth images, the fourth image can be subjected to structured light decoding to obtain another set of fourth coordinates of each calibration point in the field of view of the projector.

[0107] For example, in the above step S101, the main camera may capture three first images. For each of these three first images, the coordinates of each calibration point in the first image can be detected and used as a set of first coordinates of each calibration point in the field of view of the main camera.

[0108] In one embodiment of the present application, feature extraction can be performed on the first image to determine the positions of the respective calibration points in the first image as the first coordinates of the respective calibration points in the field of view of the main camera, and feature extraction can be performed on the second image to determine the positions of the respective calibration points in the second image as the second coordinates of the respective calibration points in the field of view of the auxiliary camera.

[0109] Feature extraction of the image can be implemented through a feature extraction network or model in the prior art, which will not be elaborated here.

[0110] In another embodiment of the present application, since the shape of the calibration points in the calibration plate is usually circular, the circle-finding method in the prior art can be used to determine the circular regions in the first image and the second image, thereby determining the above-mentioned first coordinates and second coordinates.

[0111] In addition to the feature extraction methods mentioned in the above embodiments, the coordinates of the respective calibration points in the first image and the second image can also be determined manually.

[0112] In another embodiment of the present application, structured light decoding of the third image and the fourth image can be implemented by using the structured light decoding technology in the prior art.

[0113] Step S103: Calculate the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera, and the projector parameters of the projector respectively based on the conversion relationships between the first coordinates, the second coordinates, the reference coordinates, and the three-dimensional coordinates.

[0114] Among them, the reference coordinates include: the third coordinates and / or the fourth coordinates.

[0115] The above-mentioned reference coordinates can be the third coordinates, and the conversion relationship between the reference coordinates and the three-dimensional coordinates can be the conversion relationship between the third coordinates and the three-dimensional coordinates; the above-mentioned reference coordinates can be the fourth coordinates, and the conversion relationship between the reference coordinates and the three-dimensional coordinates can be the conversion relationship between the fourth coordinates and the three-dimensional coordinates; the above-mentioned reference coordinates can be the third coordinates and the fourth coordinates, and the conversion relationship between the reference coordinates and the three-dimensional coordinates can be the conversion relationship between the third coordinates, the fourth coordinates, and the three-dimensional coordinates.

[0116] There is a conversion relationship between the above-mentioned first coordinates, second coordinates, third coordinates, and / or fourth coordinates and the three-dimensional coordinates. That is to say, based on this conversion relationship, the coordinates of the calibration points in the fields of view of the main camera, the auxiliary camera, and the projector can be converted to the coordinates of the calibration points in the above-mentioned three-dimensional coordinate system. Since the fields of view of the main camera, the auxiliary camera, and the projector may be different, accordingly, the conversion relationships between the coordinates of the calibration points in the fields of view of each device and the three-dimensional coordinates may also be different.

[0117] In one embodiment of the present application, the main camera parameters of the main camera can be calculated based on the conversion relationship between the first coordinate and the three-dimensional coordinate; the auxiliary camera parameters of the auxiliary camera can be calculated based on the conversion relationship between the second coordinate and the three-dimensional coordinate; and the projector parameters of the projector can be calculated based on the conversion relationship between the reference coordinate and the three-dimensional coordinate.

[0118] Since the above-mentioned multiple conversion relationships are similar, the following takes the calculation of the main camera parameters of the main camera based on the conversion relationship between the first coordinate and the three-dimensional coordinate as an example for illustration.

[0119] When calculating the main camera parameters of the main camera, the multiple first coordinates and three-dimensional coordinates obtained in the above steps can be used to calculate the first conversion matrix between the first coordinate and the three-dimensional coordinate, and the first conversion matrix is decomposed to obtain the internal parameter matrix and external parameter matrix of the main camera, which are used as the main camera parameters of the main camera.

[0120] The above first conversion matrix can be calculated through the following expression:

[0121]

[0122] where u and v respectively represent the coordinates of the two coordinate axes of the first coordinate, X w , Y w , Z w respectively represent the coordinates of the three coordinate axes of the three-dimensional coordinate, s represents the scale factor, and P represents the above first conversion matrix.

[0123] The decomposition of the first conversion matrix can be realized by using the decomposition technology in the prior art, such as the QR decomposition technology, etc.

[0124] It should be noted that since both the third coordinate and the fourth coordinate are the coordinates of the calibration points in the field of view of the projector, when calculating the above projector parameters, only the third coordinate can be used for calculation, or only the fourth coordinate can be used for calculation, or both the third coordinate and the fourth coordinate can be used for calculation.

[0125] When using the third coordinate and the fourth coordinate to calculate the above projector parameters, the third coordinate and the fourth coordinate can be first fused to obtain a fused coordinate, and then the above projector parameters are calculated based on the fused coordinate.

[0126] For example, the average coordinate of the above third coordinate and the fourth coordinate is calculated as the fused coordinate, and the above projector parameters are calculated based on this average coordinate.

[0127] In addition, the third coordinate and the fourth coordinate can also be used for calculation respectively to obtain two target parameters, and these two target parameters are fused to obtain the fused projector parameters.

[0128] Since the first coordinate is the coordinate of the calibration point in the field of view of the main camera, based on the conversion relationship between the first coordinate and the three-dimensional coordinate, the parameters of the main camera can be accurately calculated. Similarly, based on the conversion relationship between the second coordinate and the three-dimensional coordinate, the parameters of the auxiliary camera can be accurately calculated. Based on the conversion relationship between the reference coordinate and the three-dimensional coordinate, the parameters of the projector can be accurately calculated.

[0129] Step S104: Calculate the coordinates of the calibration points on the calibration board at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively by using the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, and the feed distance, and use them as the calculated coordinates.

[0130] Among them, the feed angle reflects the angle between the feed direction and the normal direction of the calibration board, and the feed distance is the distance between each parallel position and the lowermost parallel position.

[0131] The normal direction of the above-mentioned calibration board is the direction perpendicular to the plane where the calibration board is located and upward.

[0132] Since the above-mentioned parallel positions can be known by presetting or adjusting the position of the calibration board in real time, the distance between the calibration boards at different parallel positions can be measured.

[0133] Specifically, when the calibration board is at different parallel positions, the three-dimensional coordinates of each calibration point on the calibration board in a preset three-dimensional coordinate system can be calculated, and the feed distance when the calibration board is at this parallel position can be measured. Therefore, based on the obtained three-dimensional coordinates, the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, and the obtained feed distance can be used to perform coordinate conversion on the three-dimensional coordinates to obtain the coordinates of the calibration points on the calibration board at this parallel position in the fields of view of the main camera, the auxiliary camera, and the projector respectively. When the calibration board is at one parallel position, a set of coordinates of the calibration points on the calibration board can be calculated, and this set of coordinates includes the coordinates of the calibration points in the fields of view of the main camera, the auxiliary camera, and the projector respectively. In this way, when the calibration board is at multiple different parallel positions, multiple sets of coordinates of the calibration points on the calibration board can be calculated, and the above-mentioned calculated coordinates include multiple sets of coordinates calculated when the calibration board is at each parallel position.

[0134] The above-mentioned coordinate conversion can be realized based on the reprojection technology in the prior art, which will not be elaborated here.

[0135] In an embodiment of the present application, when the lowermost parallel position is in the plane where the X-axis and the Y-axis of the preset three-dimensional coordinate system are located, the three-dimensional coordinates of the calibration points on the calibration board can be calculated according to the following expression:

[0136]

[0137]

[0138] Z1 = d·cosθ

[0139] Wherein, the above X1, Y1, and Z1 respectively represent the three coordinate values of the three-dimensional coordinates of the calibration point, X0 and Y0 respectively represent the X-axis coordinate and Y-axis coordinate of the calibration point when the calibration board where the calibration point is located is in the lowermost parallel position, θ represents the first included angle, represents the second included angle, and d represents the feed distance.

[0140] In the above solution, the feed angle and feed distance are considered when calculating the above reference coordinates, so that the obtained reference coordinates are more accurate, thereby improving the accuracy of camera optimization.

[0141] Step S105: Calculate the difference between each calculated coordinate and the measured coordinate, and optimize the main camera parameters, auxiliary camera parameters, projector parameters, and feed angle based on the principle of reducing the calculated difference, so as to obtain the optimized parameters of the main camera and the projector.

[0142] Wherein, the measured coordinates include: the first coordinate, the second coordinate, the third coordinate, and the fourth coordinate obtained when the calibration board is in different parallel positions.

[0143] For ease of understanding, the coordinates of the determined calibration point in the field of view of the main camera are referred to as the first calculated coordinate, the coordinates of the determined calibration point in the field of view of the auxiliary camera are referred to as the second calculated coordinate, and the coordinates of the determined calibration point in the field of view of the projector are referred to as the third calculated coordinate.

[0144] The differences between the above calculated coordinates and the measured coordinates include the differences between the first coordinate and the first calculated coordinate, the second coordinate and the second calculated coordinate, the third coordinate and the third calculated coordinate, and the fourth coordinate and the third calculated coordinate when the calibration board is in different parallel positions.

[0145] For example, if there are two parallel positions m and n, when the calibration board is in the parallel position m, the first coordinate of the measured coordinates of the calibration points on the calibration board is the coordinate pm1, the second coordinate is the coordinate pm2, the third coordinate is the coordinate pm3, the fourth coordinate is the coordinate pm4, and the first reference coordinate of the reference coordinates of the calibration points on the calibration board is the coordinate qm1, the second reference coordinate is the coordinate qm2, the third reference coordinate is the coordinate qm3, and the fourth reference coordinate is the coordinate qm4; when the calibration board is in the parallel position n, the first coordinate of the measured coordinates of the calibration points on the calibration board is the coordinate pn1, the second coordinate is the coordinate pn2, the third coordinate is the coordinate pn3, the fourth coordinate is the coordinate pn4, and the first reference coordinate of the reference coordinates of the calibration points on the calibration board is the coordinate qn1, the second reference coordinate is the coordinate qn2, the third reference coordinate is the coordinate qn3, and the fourth reference coordinate is the coordinate qn4.

[0146] At this time, the differences between the above-mentioned respective reference coordinates and the measured coordinates include eight differences: the difference between the coordinate pm1 and the coordinate qm1, the difference between the coordinate pm2 and the coordinate qm2, the difference between the coordinate pm3 and the coordinate qm3, the difference between the coordinate pm4 and the coordinate qm4, the difference between the coordinate pn1 and the coordinate qn1, the difference between the coordinate pn2 and the coordinate qn2, the difference between the coordinate pn3 and the coordinate qn3, and the difference between the coordinate pn4 and the coordinate qn4.

[0147] There are various situations for the differences between the above-mentioned respective calculated coordinates and the measured coordinates.

[0148] In one case, for each set of coordinates and the calculated coordinates corresponding to the coordinates, the difference between the coordinate and the calculated coordinate can be expressed as the distance between the two coordinates. Therefore, the differences between the above-mentioned respective calculated coordinates and the measured coordinates can be expressed as the sum of the distances of the above-mentioned respective coordinates.

[0149] In another case, the differences between the above-mentioned respective calculated coordinates and the measured coordinates can be expressed as the difference e in the formula mentioned in the embodiments of the subsequent application, which will not be elaborated here for the time being.

[0150] After calculating the differences between the respective calculated coordinates and the measured coordinates, the parameters of the main camera, the auxiliary camera, and the projector can be optimized based on the principle of reducing the calculated differences to obtain the optimized parameters of the main camera and the projector.

[0151] If the differences between the above-mentioned calculated coordinates and the measured coordinates are expressed as the sum of the distances of the above four groups of coordinates, then the optimization of the main camera parameters, the auxiliary camera parameters, and the projector parameters based on the principle of reducing the calculated differences can be understood as reducing the sum of the distances of the above four groups of coordinates by adjusting the above-mentioned main camera parameters, the auxiliary camera parameters, and the projector parameters until the sum of the distances reaches the minimum. At this time, the adjusted main camera parameters are the parameters of the optimized main camera, and the adjusted projector parameters are the parameters of the optimized projector.

[0152] If the differences between the above-mentioned calculated coordinates and the measured coordinates are expressed by the formulas in the following embodiments, then the parameters of the main camera and the projector when the minimum value of the difference e is obtained can be obtained by the existing formula solving method.

[0153] As can be seen from the above, when calibrating the structured light 3D camera by applying the solution provided in the embodiment of the present application, the coordinates of each calibration point in the first image obtained are used as the first coordinates of each calibration point in the field of view of the main camera, the coordinates of each calibration point in the second image are used as the second coordinates of each calibration point in the field of view of the auxiliary camera, and the structured light of the third image and the fourth image is decoded to obtain the third coordinates and the fourth coordinates of each calibration point in the field of view of the projector. Based on the conversion relationship between the first coordinate, the second coordinate, the third coordinate, and / or the fourth coordinate and the three-dimensional coordinates in the preset three-dimensional coordinate system, the main camera parameters, the auxiliary camera parameters, and the projector parameters are calculated. Furthermore, based on these three parameters, the coordinates of each calibration point in the fields of view of the main camera, the auxiliary camera, and the projector are determined as the calculated coordinates, and the differences between each calculated coordinate and the measured coordinate are calculated. The main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle are optimized based on the principle of reducing the difference. Since the above-mentioned calculated coordinates are determined according to the above three target parameters, if the above difference is smaller, it means that the calculated coordinates determined according to the above three target parameters are closer to the coordinates actually measured, and the above three target parameters are more accurate. Therefore, optimizing the above three target parameters and the feed angle based on the principle of reducing the above difference can make the parameters of the optimized main camera and projector as accurate as possible.

[0154] In addition, when calibrating the structured light 3D camera by applying the solution provided in the embodiment of the present application, the calibration plates in different parallel positions are considered. For each calibration plate in a parallel position, the images of the calibration plate in this parallel position and the three-dimensional coordinates of the calibration points on the calibration plate in this parallel position are obtained, and the coordinates of the calibration points on the calibration plate in this parallel position in the fields of view of the main camera, the auxiliary camera, and the projector are determined. Thus, the differences between the calculated reference coordinates and the measured coordinates comprehensively consider the situations when the calibration plates are in different parallel positions. Optimizing the above target parameters based on the principle of reducing the difference can improve the reliability of the optimized target parameters.

[0155] Since in the above step S101, different images of the calibration board at different parallel positions can be obtained by changing the parallel position of the calibration board. Therefore, in one embodiment of the present application, the calibration board can be located on the lifting table, and the lifting table is used to adjust the parallel position of the calibration board by moving along the feeding direction.

[0156] Since in the process of calibrating a structured light 3D camera, the requirements for the position of the calibration board are usually relatively high. Therefore, if the calibration board is directly moved, it may lead to a large deviation in the process of calibrating the structured light 3D camera, resulting in a decrease in the accuracy of the parameters of the main camera and the projector finally obtained.

[0157] In the embodiment of the present application, the lifting table can be moved along the feeding direction manually or by computer control, so as to move the calibration board, which avoids directly operating on the calibration board and reduces the deviation in the process of calibrating the structured light 3D camera. Moreover, the operation of controlling the lifting table to move up and down is relatively simple, which improves the operability of calibrating the structured light 3D camera. At the same time, controlling the movement of the lifting table can also realize the equidistant movement of the calibration board, reduce the calculation amount of calculating the coordinates of the calibration points in the calibration board, and thus improve the efficiency of calibrating the structured light 3D camera.

[0158] The following describes the specific implementation method of calculating the coordinates of the calibration points in the fields of view of the main camera, the auxiliary camera, and the projector in the above step S104.

[0159] In one embodiment of the present application, refer to Figure 2 , a flowchart of a second structured light 3D camera calibration method is provided. In this embodiment, the above step S104 can be implemented through the following steps S104A - S104C.

[0160] Step S104A: Using the main camera parameters, the feeding angle, and the feeding distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point in the calibration board at different parallel positions to obtain the coordinates of each calibration point in the calibration board at different parallel positions in the field of view of the main camera.

[0161] Specifically, when the calibration board is in different parallel positions, the three-dimensional coordinates of each calibration point in the calibration board in the preset three-dimensional coordinate system can be obtained, and the feed distance when the calibration board is in this parallel position can be measured. Therefore, based on the obtained three-dimensional coordinates, using the main camera parameters, the feed angle, and the obtained feed distance, the coordinates of the calibration points in the calibration board at this parallel position in the field of view of the main camera can be calculated. When the calibration board is in one parallel position, a set of coordinates of the calibration points in the calibration board in the field of view of the main camera can be calculated. In this way, when the calibration board is in multiple different parallel positions, multiple sets of coordinates of the calibration points in the calibration board can be calculated. The above-mentioned calculated coordinates include multiple sets of coordinates calculated when the calibration board is in each parallel position.

[0162] Since the coordinates of the calibration points in the field of view of the main camera are usually two-dimensional coordinates, therefore, the conversion of the above coordinates is from three-dimensional coordinates to two-dimensional coordinates.

[0163] For example, if the above main camera parameters are a 3x4 matrix, then the above three-dimensional coordinates can be homogenized, the above three-dimensional coordinates can be transformed into a 4x1 vector, and then multiplied by the above main camera parameters to obtain a 3x1 vector, and the obtained vector is subjected to reverse homogenization to obtain the coordinates of the calibration points in the field of view of the main camera.

[0164] Step S104B: Using the auxiliary camera parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point in the calibration board at different parallel positions to obtain the coordinates of each calibration point in the calibration board at different parallel positions in the field of view of the auxiliary camera.

[0165] Similar to the above step S104A, the three-dimensional coordinates of each calibration point in the calibration board in the preset three-dimensional coordinate system and the feed distance when the calibration board is in this parallel position can be obtained. Based on the obtained three-dimensional coordinates, using the auxiliary camera parameters, the feed angle, and the obtained feed distance, calculate the coordinates of the calibration points in the calibration board at this parallel position in the field of view of the auxiliary camera.

[0166] Step S104C: Using the projector parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point in the calibration board at different parallel positions to obtain the coordinates of each calibration point in the calibration board at different parallel positions in the field of view of the projector, and use the coordinates of each calibration point in the calibration board at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector as the calculated coordinates.

[0167] Similar to the above step S104A, the three-dimensional coordinates of each calibration point in the calibration plate in the preset three-dimensional coordinate system and the feed distance when the calibration plate is in this parallel position can be obtained. Based on the obtained three-dimensional coordinates, using the projector parameters, the feed angle, and the obtained feed distance, calculate the coordinates of the calibration points in the calibration plate at this parallel position in the field of view of the projector.

[0168] As can be seen from the above, when calibrating the structured light 3D camera using the solution provided in the embodiments of the present application, based on the main camera parameters, the auxiliary camera parameters, and the projector parameters respectively, three coordinate conversions are performed on the three-dimensional coordinates of each calibration point in the calibration plate at different parallel positions. In this way, the coordinates of each calibration point in the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector can be accurately calculated, and these three coordinate conversions can be performed simultaneously, improving the efficiency of calibrating the structured light 3D camera.

[0169] When obtaining the above calculated coordinates, calculating the differences between the calculated coordinates and the measured coordinates, and optimizing the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, and the feed distance, in addition to Figure 2 the methods listed in steps S104 and S105 in the embodiments shown, the following can also be adopted Figure 3 In the embodiments shown, step S104D is used to obtain the above calculated coordinates, and step S105A is used to optimize the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, and the feed distance.

[0170] In an embodiment of the present application, referring to Figure 3 , a flowchart of a third method for calibrating a structured light 3D camera is provided. Compared with the foregoing Figure 2 embodiments shown, in this embodiment, the above step S104 can be implemented by using the following step S104D, and the above step S105 can be implemented by using the following step S105A.

[0171] Step S104D: Use the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, the feed distance, the first distortion parameter, the second distortion parameter, and the third distortion parameter to determine the coordinates of the calibration points in the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively, as the calculated coordinates.

[0172] Among them, the first distortion parameter, the second distortion parameter, and the third distortion parameter are the distortion parameters of the main camera, the auxiliary camera, and the projector respectively.

[0173] Specifically, for the calibration board at each parallel position, after obtaining the three-dimensional coordinates of each calibration point in the calibration board in a preset three-dimensional coordinate system, the three-dimensional coordinates of each calibration point can be converted using the main camera parameters and the first distortion parameter to obtain the coordinates of each calibration point in the field of view of the main camera; the three-dimensional coordinates of each calibration point can be converted using the auxiliary camera parameters and the second distortion parameter to obtain the coordinates of each calibration point in the field of view of the auxiliary camera; the three-dimensional coordinates of each calibration point can be converted using the projector parameters and the third distortion parameter to obtain the coordinates of each calibration point in the field of view of the projector.

[0174] The specific implementation method of the above coordinate conversion can be referred to the description in the foregoing embodiments, and will not be elaborated here.

[0175] After obtaining the above calculated coordinates, the optimized parameters of the main camera and the projector can be obtained through the following step S105A.

[0176] Step S105A: Calculate the difference between each calculated coordinate and the measured coordinate, and optimize the main camera parameters, auxiliary camera parameters, projector parameters, and the first distortion parameter, second distortion parameter, third distortion parameter, and feed angle based on the principle of reducing the calculated difference to obtain the optimized parameters of the main camera and the projector.

[0177] After obtaining the above calculated coordinates, the difference between each calculated coordinate and the measured coordinate can be calculated. At this time, the difference includes the difference between the coordinates determined by the target parameters considering the first distortion parameter, second distortion parameter, third distortion parameter, and feed angle and the measured coordinates.

[0178] The method for calculating the difference between each calculated coordinate and the measured coordinate and the method for optimizing the main camera parameters, auxiliary camera parameters, and projector parameters can be referred to step S105 in the above Figure 1a illustrated embodiment, and will not be elaborated here.

[0179] As can be seen from the above, when applying the solution provided by the embodiments of the present application for structured light 3D camera calibration, the first distortion parameter, second distortion parameter, third distortion parameter, feed angle, and feed distance are considered when calculating the above calculated coordinates, so that the obtained calculated coordinates are more accurate. Moreover, when optimizing the main camera parameters, auxiliary camera parameters, and projector parameters, the first distortion parameter, second distortion parameter, third distortion parameter, and feed angle are also optimized. More parameters are involved in the optimization process, making the factors considered in the optimization process more comprehensive, thereby improving the accuracy of the optimized parameters of the main camera and the projector.

[0180] In an embodiment of the present application, considering the above-mentioned feed angle, feed distance, first distortion parameter, second distortion parameter, and third distortion parameter, the difference e between each calculated coordinate and the measured coordinate can be calculated according to the following formula:

[0181]

[0182] Where θ represents the first included angle, represents the second included angle, i represents the i-th parallel position, and N img represents the total number of parallel positions, j represents the j-th calibration point on the calibration board, and N pt represents the total number of calibration points on the calibration board, d i represents the moving distance between the calibration board at the i-th parallel position and the lowermost parallel position. When n = 1, p nij represents the first coordinate of the j-th calibration point on the calibration board at the i-th parallel position, K n represents the internal parameter matrix of the main camera, D n represents the distortion coefficient of the main camera, T n represents the external parameter matrix of the main camera. When n = 2, p nij represents the second coordinate of the j-th calibration point on the calibration board at the i-th parallel position, K n represents the internal parameter matrix of the auxiliary camera, D n represents the distortion coefficient of the auxiliary camera, T n represents the external parameter matrix of the auxiliary camera. When n = 3 or n = 4, K n represents the internal parameter matrix of the projector, D n represents the distortion coefficient of the projector, T n represents the external parameter matrix of the projector. When n = 3, p nij represents the third coordinate of the j-th calibration point on the calibration board at the i-th parallel position. When n = 4, p nij represents the fourth coordinate of the j-th calibration point on the calibration board at the i-th parallel position, X ij 、Y ij respectively represent the X-axis coordinate and Y-axis coordinate of the j-th calibration point in the three-dimensional coordinate system when the calibration board is at the i-th parallel position.

[0183] The above p nij can be understood as the measured coordinates of each calibration point on the calibration board obtained by measuring the image collected by the main camera or the auxiliary camera, can be understood as the calculated coordinates of each calibration point on the calibration board calculated according to the above target parameters, feed angle, feed distance, and each distortion parameter. Subtracting the above two coordinates can obtain the relatively accurate difference e between each calculated coordinate and the measured coordinate.

[0184] As can be seen from the above, when calibrating a structured light 3D camera using the solution provided in the embodiments of the present application, calculating the differences between the respective calculated coordinates and the measured coordinates according to the above formula takes into account the effects of various target parameters, feed angles, feed distances, and various distortion parameters on the above calculated coordinates. Since many parameters are considered, therefore, according to the above formula, the differences between the respective calculated coordinates and the measured coordinates can be obtained more accurately, thereby improving the accuracy of calibrating the structured light 3D camera.

[0185] Corresponding to the above structured light 3D camera calibration method, an embodiment of the present application further provides a structured light 3D camera calibration device.

[0186] See Figure 4 , which provides a schematic structural diagram of a first structured light 3D camera calibration device. The device includes:

[0187] An image acquisition module 401, configured to respectively acquire a first image and a second image of a calibration board at a parallel position collected by a main camera and a preset auxiliary camera in the structured light 3D camera for calibration boards that are translated along a fixed feed direction and are in different parallel positions, and when the projector in the structured light 3D camera projects structured light onto the calibration board, respectively acquire a third image and a fourth image of the calibration board at this parallel position collected by the main camera and the auxiliary camera, and acquire the three-dimensional coordinates of each calibration point on the calibration board at this parallel position in a preset three-dimensional coordinate system, where the calibration boards at different parallel positions are parallel to each other and have different heights;

[0188] An image processing module 402, configured to respectively detect the coordinates of each calibration point in each first image and second image as the first coordinate and the second coordinate of each calibration point in the fields of view of the main camera and the auxiliary camera, and respectively perform structured light decoding on each third image and fourth image to obtain the third coordinate and the fourth coordinate of each calibration point in the field of view of the projector;

[0189] A parameter calculation module 403, configured to respectively calculate the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera, and the projector parameters of the projector based on the conversion relationships between the first coordinate, the second coordinate, the reference coordinate, and the three-dimensional coordinate, where the reference coordinate includes: the third coordinate and / or the fourth coordinate;

[0190] A coordinate calculation module 404 is configured to calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively by using the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, and the feed distance as calculation coordinates. Wherein, the feed angle reflects the included angle between the feed direction and the normal direction of the calibration plate, and the feed distance is the distance between each parallel position and the lowermost parallel position;

[0191] A parameter optimization module 405 is configured to calculate the differences between the respective calculated coordinates and the measured coordinates, and optimize the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle based on the principle of reducing the calculated differences, so as to obtain the optimized parameters of the main camera and the projector. Wherein, the measured coordinates include the first coordinate, the second coordinate, the third coordinate, and the fourth coordinate obtained when the calibration plate is at different parallel positions;

[0192] As can be seen from the above, when calibrating the structured light 3D camera by applying the solution provided in the embodiment of the present application, the coordinates of each calibration point in the first image obtained are used as the first coordinates of each calibration point in the field of view of the main camera, the coordinates of each calibration point in the second image are used as the second coordinates of each calibration point in the field of view of the auxiliary camera, and the structured light decoding is performed on the third image and the fourth image to obtain the third coordinates and the fourth coordinates of each calibration point in the field of view of the projector. Based on the conversion relationship between the first coordinate, the second coordinate, the third coordinate, and / or the fourth coordinate and the three-dimensional coordinates in the preset three-dimensional coordinate system, the main camera parameters, the auxiliary camera parameters, and the projector parameters are calculated. Furthermore, based on these three parameters, the coordinates of each calibration point in the fields of view of the main camera, the auxiliary camera, and the projector are determined as calculation coordinates, and the differences between each calculation coordinate and the measured coordinate are calculated, and the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle are optimized based on the principle of reducing the differences. Since the above calculation coordinates are determined according to the above three target parameters, if the above differences are smaller, it means that the calculation coordinates determined according to the above three target parameters are closer to the coordinates actually measured, and the above three target parameters are more accurate. Therefore, optimizing the above three target parameters and the feed angle based on the principle of reducing the above differences can make the parameters of the optimized main camera and projector as accurate as possible.

[0193] In addition, when calibrating the structured light 3D camera by applying the solution provided in the embodiments of the present application, calibration plates in different parallel positions are considered. For each calibration plate in a parallel position, each image of the calibration plate in this parallel position and the three-dimensional coordinates of the calibration points on the calibration plate in this parallel position are obtained, and the coordinates of the calibration points on the calibration plate in this parallel position in the main camera field of view, the auxiliary camera field of view, and the projector field of view are determined. As a result, the differences between the calculated respective reference coordinates and the measurement coordinates comprehensively consider the situations when the calibration plates are in different parallel positions, and the above-mentioned respective target parameters are optimized based on the principle of reducing such differences, which can improve the reliability of the optimized target parameters.

[0194] In one embodiment of the present application, referring to Figure 5 , a schematic structural diagram of a second structured light 3D camera calibration device is provided. In this embodiment, the parameter calculation module 403 includes:

[0195] A first parameter calculation sub-module 403A, configured to calculate the main camera parameters of the main camera based on the conversion relationship between the first coordinates and the three-dimensional coordinates;

[0196] A second parameter calculation sub-module 403B, configured to calculate the auxiliary camera parameters of the auxiliary camera based on the conversion relationship between the second coordinates and the three-dimensional coordinates;

[0197] A third parameter calculation sub-module 403C, configured to calculate the projector parameters of the projector based on the conversion relationship between the reference coordinates and the three-dimensional coordinates.

[0198] As can be seen from the above, when calibrating the structured light 3D camera by applying the solution provided in the embodiments of the present application, since the first coordinates are the coordinates of the calibration points in the field of view of the main camera, therefore, based on the conversion relationship between the first coordinates and the three-dimensional coordinates, the main camera parameters of the main camera can be accurately calculated. Similarly, based on the conversion relationship between the enemy coordinates and the three-dimensional coordinates, the auxiliary camera parameters of the auxiliary camera can be accurately calculated. Based on the conversion relationship between the third coordinates and / or the fourth coordinates and the three-dimensional coordinates, the projector parameters of the projector can be accurately calculated.

[0199] In one embodiment of the present application, the first parameter calculation sub-module 403A is specifically configured to:

[0200] Using the obtained respective first coordinates and three-dimensional coordinates, calculate a first transformation matrix between the first coordinates and the three-dimensional coordinates, and decompose the first transformation matrix to obtain the internal parameter matrix and the external parameter matrix of the main camera as the main camera parameters of the main camera.

[0201] As can be seen from the above, when calibrating the structured light 3D camera using the solution provided by the embodiments of the present application, since the main camera, the auxiliary camera, and the projector have internal parameters and external parameters, first calculate the above first transformation matrix, and decompose the first transformation matrix, the internal parameter matrix and the external parameter matrix of the main camera can be accurately obtained. Similarly, the internal parameters and external parameters of the auxiliary camera and the projector can be accurately obtained in the same way.

[0202] In one embodiment of the present application, the coordinate calculation module 404 is specifically configured to:

[0203] Use the main camera parameters, the feed angle, and the feed distances at different parallel positions to perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions, so as to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the main camera;

[0204] Use the auxiliary camera parameters, the feed angle, and the feed distances at different parallel positions to perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions, so as to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the auxiliary camera;

[0205] Use the projector parameters, the feed angle, and the feed distances at different parallel positions to perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions, so as to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the projector, and use the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the main camera, the auxiliary camera, and the projector as the calculated coordinates.

[0206] As can be seen from the above, when calibrating the structured light 3D camera using the solution provided by the embodiments of the present application, three coordinate transformations are respectively performed on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions based on the main camera parameters, the auxiliary camera parameters, and the projector parameters. In this way, the coordinates of each calibration point on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector can be accurately calculated, and the three coordinate transformations can be performed simultaneously, improving the efficiency of calibrating the structured light 3D camera.

[0207] In one embodiment of the present application, the coordinate calculation module 404 is specifically configured to:

[0208] Respectively use the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, the feed distance, the first distortion parameter, the second distortion parameter, and the third distortion parameter to calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively as the calculated coordinates, where the first distortion parameter, the second distortion parameter, and the third distortion parameter are the distortion parameters of the main camera, the auxiliary camera, and the projector respectively;

[0209] The parameter optimization module 405 includes:

[0210] Calculate the differences between the respective calculated coordinates and the measured coordinates, and optimize the main camera parameters, auxiliary camera parameters, projector parameters, and the first distortion parameter, second distortion parameter, third distortion parameter, and feed angle based on the principle of reducing the calculated differences, so as to obtain the optimized parameters of the main camera and the projector.

[0211] As can be seen from the above, when calibrating the structured light 3D camera by applying the solution provided in the embodiment of the present application, calculating the differences between the respective calculated coordinates and the measured coordinates according to the above formula takes into account the influences of various target parameters, feed angle, feed distance, and various distortion parameters on the above calculated coordinates. Since more parameters are considered, the differences between the respective calculated coordinates and the measured coordinates can be obtained more accurately according to the above formula, thereby improving the accuracy of the structured light 3D camera calibration.

[0212] In one embodiment of the present application, calculating the differences between the respective calculated coordinates and the measured coordinates includes:

[0213] Calculate the differences e between the respective calculated coordinates and the measured coordinates according to the following formula:

[0214]

[0215] Among them, the θ represents the first included angle, the represents the second included angle, the i represents the i-th parallel position, the N img represents the total number of parallel positions, the j represents the j-th calibration point on the calibration board, the N pt represents the total number of calibration points on the calibration board, the d i represents the moving distance between the calibration board at the i-th parallel position and the lowermost parallel position. When n = 1, the p nij represents the first coordinate of the j-th calibration point on the calibration board at the i-th parallel position, the K n represents the internal parameter matrix of the main camera, the D n represents the distortion coefficient of the main camera, the T n represents the external parameter matrix of the main camera. When n = 2, the p nij represents the second coordinate of the j-th calibration point on the calibration board at the i-th parallel position, the K n represents the internal parameter matrix of the auxiliary camera, the D n represents the distortion coefficient of the auxiliary camera, the T n represents the external parameter matrix of the auxiliary camera. When n = 3 or n = 4, the K nrepresents the internal parameter matrix of the projector, and the D n represents the distortion coefficient of the projector, and the T n represents the external parameter matrix of the projector. When n = 3, the p nij represents the third coordinate of the j-th calibration point on the calibration plate at the i-th parallel position. When n = 4, the p nij represents the fourth coordinate of the j-th calibration point on the calibration plate at the i-th parallel position, and the X ij and Y ij respectively represent the X-axis coordinate and Y-axis coordinate of the j-th calibration point in the three-dimensional coordinate system when the calibration plate is at the i-th parallel position.

[0216] As can be seen from the above, when calibrating the structured light 3D camera using the solution provided in the embodiments of the present application, calculating the differences between the respective reference coordinates and the measurement coordinates according to the above formula takes into account the effects of various target parameters, feed angles, feed distances, and various distortion parameters on the above reference coordinates. Since more parameters are considered, the differences between the respective reference coordinates and the measurement coordinates can be obtained more accurately according to the above formula, thereby improving the accuracy of the structured light 3D camera calibration.

[0217] In one embodiment of the present application, the calibration plate is located on a lifting table, and the lifting table is used to adjust the parallel position of the calibration plate by moving along the feed direction.

[0218] As can be seen from the above, when calibrating the structured light 3D camera using the solution provided in the embodiments of the present application, the lifting table can be moved along a preset direction manually or by computer control, so as to move the calibration plate. This avoids directly operating on the calibration plate and reduces the deviation during the calibration of the structured light 3D camera. Moreover, the operation of controlling the lifting table to move up and down is relatively simple, which improves the operability of the structured light 3D camera calibration. At the same time, controlling the lifting table to move can also achieve equidistant movement of the calibration plate, reducing the computational amount of calculating the coordinates of the calibration points on the calibration plate, thereby improving the efficiency of the structured light 3D camera calibration.

[0219] In one embodiment of the present application, the image acquisition module 401 is specifically configured to:

[0220] For a calibration board that undergoes translational motion along a fixed feed direction and is in different parallel positions, when the projector in the structured light 3D camera projects white light onto the calibration board, first images and second images of the calibration board at this parallel position collected by the main camera and a preset auxiliary camera in the structured light 3D camera are obtained respectively. When the projector in the structured light 3D camera projects structured light onto the calibration board, third images and fourth images of the calibration board at this parallel position collected by the main camera and the auxiliary camera are obtained respectively, and the three-dimensional coordinates of each calibration point on the calibration board at this parallel position in a preset three-dimensional coordinate system are obtained.

[0221] As can be seen from the above, when calibrating a structured light 3D camera by applying the solution provided in the embodiments of the present application, since the depth of field and the field of view of the projector can be known by projecting white light through the projector, it can be ensured that the calibration board is always within the field of view of the projector, guaranteeing the normal progress of the structured light 3D camera calibration. Moreover, when the projector projects white light onto the calibration board, the images collected by the main camera and the auxiliary camera can be made clearer.

[0222] In an embodiment of the present application, the preset three-dimensional coordinate system is established based on the position where the calibration board is located.

[0223] As can be seen from the above, when calibrating a structured light 3D camera by applying the solution provided in the embodiments of the present application, when establishing a three-dimensional coordinate system based on the position where the calibration board is located, the position of the calibration point on the calibration board can be matched with the coordinates of the calibration point on the X-axis and Y-axis, thereby accelerating the speed of obtaining the three-dimensional coordinates of the calibration point and further improving the efficiency of the structured light 3D camera calibration.

[0224] Embodiments of the present application also provide an electronic device, as Figure 6 shown, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604. Among them, the processor 601, the communication interface 602, and the memory 603 complete communication with each other through the communication bus 604.

[0225] The memory 603 is used to store a computer program.

[0226] The processor 601 is used to implement the following steps when executing the program stored on the memory 603:

[0227] For calibration plates in different parallel positions undergoing translational motion along a fixed feed direction, first images and second images of the calibration plates at these parallel positions captured by the main camera and a preset auxiliary camera in the structured light 3D camera are obtained respectively. When the projector in the structured light 3D camera projects structured light onto the calibration plates, third images and fourth images of the calibration plates at these parallel positions captured by the main camera and the auxiliary camera are obtained respectively, and the three-dimensional coordinates of each calibration point on the calibration plates at these parallel positions in a preset three-dimensional coordinate system are obtained. Among them, the calibration plates in different parallel positions are parallel to each other and have different heights;

[0228] The coordinates of each calibration point in each of the first images and second images are detected respectively as the first coordinates and second coordinates of each calibration point in the fields of view of the main camera and the auxiliary camera. Structured light decoding is performed on each of the third images and fourth images respectively to obtain the third coordinates and fourth coordinates of each calibration point in the field of view of the projector;

[0229] Based on the conversion relationships between the first coordinates, second coordinates, reference coordinates and the three-dimensional coordinates respectively, the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera and the projector parameters of the projector are calculated. Among them, the reference coordinates include: the third coordinates and / or the fourth coordinates;

[0230] Using the main camera parameters, auxiliary camera parameters, projector parameters, feed angle and feed distance respectively, the coordinates of the calibration points on the calibration plates in different parallel positions in the fields of view of the main camera, auxiliary camera and projector are calculated as calculated coordinates. Among them, the feed angle reflects: the angle between the feed direction and the normal direction of the calibration plate, and the feed distance is: the distance between each parallel position and the lowermost parallel position;

[0231] The differences between the calculated coordinates and the measured coordinates are calculated, and the main camera parameters, auxiliary camera parameters, projector parameters and feed angle are optimized based on the principle of reducing the calculated differences, and the optimized parameters of the main camera and projector are obtained. Among them, the measured coordinates include: the first coordinates, second coordinates, third coordinates and fourth coordinates obtained when the calibration plate is in different parallel positions.

[0232] Other solutions for calibrating the structured light 3D camera implemented by the above-mentioned processor 601 executing the program stored in the memory 603 are the same as those mentioned in the foregoing method embodiments and will not be elaborated here.

[0233] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0234] The communication interface is used for communication between the above electronic device and other devices.

[0235] The memory can include a Random Access Memory (RAM), and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located far from the aforementioned processor.

[0236] The above-mentioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0237] In another embodiment provided by the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned structured light 3D camera calibration methods are implemented.

[0238] In another embodiment provided by the present application, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the structured light 3D camera calibration methods in the above embodiments.

[0239] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0240] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements that are not explicitly listed, or elements that are inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0241] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the embodiments of the apparatus, electronic device, computer-readable storage medium, and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0242] The above are only the preferred embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application are all included in the protection scope of the present application.

Claims

1. A structured light 3D camera calibration method, characterized in that The method includes: For calibration plates that perform translational motion along a fixed feed direction and are in different parallel positions, respectively obtain the first image and the second image of the calibration plate at this parallel position collected by the main camera and a preset auxiliary camera in the structured light 3D camera. And when the projector in the structured light 3D camera projects structured light onto the calibration plate, respectively obtain the third image and the fourth image of the calibration plate at this parallel position collected by the main camera and the auxiliary camera, and obtain the three-dimensional coordinates of each calibration point on the calibration plate at this parallel position in a preset three-dimensional coordinate system. Among them, the calibration plates in different parallel positions are parallel to each other and have different heights; Respectively detect the coordinates of each calibration point in each first image and second image as the first coordinate and the second coordinate of each calibration point in the fields of view of the main camera and the auxiliary camera. Respectively perform structured light decoding on each third image and fourth image to obtain the third coordinate and the fourth coordinate of each calibration point in the field of view of the projector; Respectively calculate the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera, and the projector parameters of the projector based on the conversion relationships between the first coordinate, the second coordinate, the reference coordinate and the three-dimensional coordinate. Among them, the reference coordinate includes: the third coordinate and / or the fourth coordinate; Respectively use the main camera parameters, the auxiliary camera parameters, the projector parameters, the feed angle, and the feed distance to calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector as the calculated coordinates. Among them, the feed angle reflects: the angle between the feed direction and the normal direction of the calibration plate, and the feed distance is: the distance between each parallel position and the lowermost parallel position; Calculate the differences between each calculated coordinate and the measured coordinate, and optimize the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle based on the principle of reducing the calculated differences to obtain the optimized parameters of the main camera and the projector. Among them, the measured coordinate includes: the first coordinate, the second coordinate, the third coordinate, and the fourth coordinate obtained when the calibration plate is in different parallel positions.

2. The method according to claim 1, characterized in that, The step of respectively calculating the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera, and the projector parameters of the projector based on the conversion relationships between the first coordinate, the second coordinate, the reference coordinate and the three-dimensional coordinate includes: Calculate the main camera parameters of the main camera based on the conversion relationship between the first coordinate and the three-dimensional coordinate; Calculate the auxiliary camera parameters of the auxiliary camera based on the conversion relationship between the second coordinate and the three-dimensional coordinate; Calculate the projector parameters of the projector based on the conversion relationship between the reference coordinate and the three-dimensional coordinate.

3. The method according to claim 2, wherein The step of calculating the main camera parameters of the main camera based on the conversion relationship between the first coordinate and the three-dimensional coordinate includes: Using each obtained first coordinate and the three-dimensional coordinate, calculate a first transformation matrix between the first coordinate and the three-dimensional coordinate, decompose the first transformation matrix to obtain the internal parameter matrix and the external parameter matrix of the main camera, and use them as the main camera parameters of the main camera.

4. The method according to any one of claims 1 to 3, characterized in that Respectively using the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle and the feed distance, calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively, and use them as the calculated coordinates, including: Using the main camera parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the main camera; Using the auxiliary camera parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the auxiliary camera; Using the projector parameters, the feed angle, and the feed distances at different parallel positions, perform coordinate transformation on the three-dimensional coordinates of each calibration point on the calibration plate at different parallel positions to obtain the coordinates of each calibration point on the calibration plate at different parallel positions in the field of view of the projector, and use the coordinates of each calibration point on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector as the calculated coordinates.

5. The method according to any one of claims 1 to 3, characterized in that Respectively using the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle and the feed distance, calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively, and use them as the calculated coordinates, including: Respectively using the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle, the feed distance, the first distortion parameter, the second distortion parameter, and the third distortion parameter, calculate the coordinates of the calibration points on the calibration plate at different parallel positions in the fields of view of the main camera, the auxiliary camera, and the projector respectively, and use them as the calculated coordinates, where the first distortion parameter, the second distortion parameter, and the third distortion parameter are the distortion parameters of the main camera, the auxiliary camera, and the projector respectively; Calculate the differences between each calculated coordinate and the measured coordinate, and optimize the main camera parameters, the auxiliary camera parameters, the projector parameters, and the feed angle based on the principle of reducing the calculated differences to obtain the optimized parameters of the main camera and the projector, including: Calculate the differences between each calculated coordinate and the measured coordinate, and optimize the main camera parameters, the auxiliary camera parameters, the projector parameters, and the first distortion parameter, the second distortion parameter, the third distortion parameter, and the feed angle based on the principle of reducing the calculated differences to obtain the optimized parameters of the main camera and the projector.

6. The method according to claim 5, wherein The calculation of the differences between each calculated coordinate and the measured coordinate includes: Calculate the difference e between each calculated coordinate and the measured coordinate according to the following formula: Wherein, the θ represents the first included angle, and the represents the second included angle, the i represents the i-th parallel position, and the N img represents the total number of the parallel positions, the j represents the j-th calibration point in the calibration board, and the N pt represents the total number of the calibration points in the calibration board, the d i represents the moving distance between the calibration board at the i-th parallel position and the lowermost parallel position. When n = 1, the p nij represents the first coordinate of the j-th calibration point in the calibration board at the i-th parallel position, and the K n represents the internal parameter matrix of the main camera, the D n represents the distortion coefficient of the main camera, the T n represents the external parameter matrix of the main camera. When n = 2, the p nij represents the second coordinate of the j-th calibration point in the calibration board at the i-th parallel position, and the K n represents the internal parameter matrix of the auxiliary camera, the D n represents the distortion coefficient of the auxiliary camera, the T n represents the external parameter matrix of the auxiliary camera. When n = 3 or n = 4, the K n represents the internal parameter matrix of the projector, the D n represents the distortion coefficient of the projector, the T n represents the external parameter matrix of the projector. When n = 3, the p nij represents the third coordinate of the j-th calibration point in the calibration board at the i-th parallel position. When n = 4, the p nij represents the fourth coordinate of the j-th calibration point in the calibration board at the i-th parallel position, and the X ij 、Y ij respectively represent the X-axis coordinate and the Y-axis coordinate of the j-th calibration point in the three-dimensional coordinate system when the calibration board is at the i-th parallel position, and the represents the calculated coordinate of the j-th calibration point in the calibration board at the i-th parallel position.

7. The method according to any one of claims 1 to 3, characterized in that The calibration plate is located on the lifting table, and the lifting table is used to adjust the parallel position of the calibration plate by moving along the feed direction.

8. The method according to any one of claims 1 to 3, characterized in that The obtaining of the first image and the second image of the calibration board at the parallel position collected by the main camera and the preset auxiliary camera in the structured light 3D camera respectively includes: When the projector in the structured light 3D camera projects white light onto the calibration board, the first image and the second image of the calibration board at the parallel position collected by the main camera and the preset auxiliary camera in the structured light 3D camera are obtained respectively.

9. The method according to any one of claims 1-3, characterized in that, The preset three-dimensional coordinate system is established based on the position where the calibration board is located.

10. A structured light 3D camera calibration device, characterized in that The device includes: An image acquisition module, configured to, for the calibration board that moves translationally along a fixed feeding direction and is at different parallel positions, respectively obtain the first image and the second image of the calibration board at the parallel position collected by the main camera and the preset auxiliary camera in the structured light 3D camera, and when the projector in the structured light 3D camera projects structured light onto the calibration board, respectively obtain the third image and the fourth image of the calibration board at the parallel position collected by the main camera and the auxiliary camera, and obtain the three-dimensional coordinates of each calibration point in the calibration board at the parallel position in the preset three-dimensional coordinate system, where the calibration boards at different parallel positions are parallel to each other and have different heights; An image processing module, configured to respectively detect the coordinates of each calibration point in each of the first image and the second image as the first coordinate and the second coordinate of each calibration point in the fields of view of the main camera and the auxiliary camera, and respectively perform structured light decoding on each of the third image and the fourth image to obtain the third coordinate and the fourth coordinate of each calibration point in the field of view of the projector; A parameter calculation module, configured to respectively calculate the main camera parameters of the main camera, the auxiliary camera parameters of the auxiliary camera, and the projector parameters of the projector based on the conversion relationships between the first coordinate, the second coordinate, the reference coordinate and the three-dimensional coordinate, where the reference coordinate includes: the third coordinate and / or the fourth coordinate; A coordinate calculation module, configured to respectively use the main camera parameters, the auxiliary camera parameters, the projector parameters, the feeding angle and the feeding distance to calculate the coordinates of the calibration points in the calibration board at different parallel positions in the fields of view of the main camera, the auxiliary camera and the projector respectively as the calculated coordinates, where the feeding angle reflects: the included angle between the feeding direction and the normal direction of the calibration board, and the feeding distance is: the distance between each parallel position and the lowermost parallel position; A parameter optimization module, configured to calculate the difference between each calculated coordinate and the measured coordinate, and optimize the main camera parameters, the auxiliary camera parameters, the projector parameters and the feeding angle based on the principle of reducing the calculated difference to obtain the optimized parameters of the main camera and the projector, where the measured coordinate includes: the first coordinate, the second coordinate, the third coordinate and the fourth coordinate obtained when the calibration board is at different parallel positions; 11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory and a communication bus, where the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used for storing computer programs; The processor is configured to, when executing the programs stored on the memory, implement the structured light 3D camera calibration method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the structured light 3D camera calibration method according to any one of claims 1-9.

Images