A joint calibration method, device and equipment for a 2D camera and a 3D camera

Abstract

The present invention discloses a method, device and equipment for joint calibration of a 2D camera and a 3D camera, relating to the technical field of camera calibration. The present invention provides a method for joint calibration of a 2D camera and a 3D camera. First, image data and point cloud data collected by the 2D camera and the 3D camera with a preset calibration board are obtained. The preset calibration board includes a plurality of solid dots with different sizes, no overlap and non-uniform distribution. Then, the dot detection algorithm is used to extract dots from the image data and the point cloud data, so as to further extract dot features. The dot features may include more complex features preset based on the preset calibration board. Thus, subsequent feature matching can be performed according to the dot features corresponding to the image data and the point cloud data respectively to determine the matching dots in the image data and the point cloud data, and the transformation matrix between the image data and the point cloud data can be determined based on multiple groups of matching dots. The present invention improves the stability and applicability of the calibration method.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and particularly relates to a method, device, and equipment for jointly calibrating a 2D camera and a 3D camera. Background Art

[0002] Currently, the method of combined imaging of a 2D camera and a 3D camera has gradually become a research hotspot. The 3D camera can obtain high-density, discretized 3D point clouds on the surface of the target object, but cannot obtain information such as color, brightness, and texture. The 2D camera can obtain high-resolution images with features such as color, brightness, and texture, but cannot obtain the depth information of the images. Therefore, the fusion of 2D images and 3D point clouds can effectively make up for the deficiency of the feature information of a single sensor data, and obtain more comprehensive data of the target object. The joint calibration of the 3D camera and the 2D camera is one of the key technologies for realizing the data fusion and complementary advantages of 3D point clouds and 2D images. The main purpose of the joint calibration is to solve the transformation matrix between the 2D camera and the 3D camera, and convert the 2D and 3D data to the same coordinate system through the solved transformation matrix, so as to realize the data fusion of the two.

[0003] Currently, the existing technical solutions mainly obtain the 2D images and 3D point clouds of a static calibration object through the 2D camera and the 3D camera in the same field of view, then establish the feature point matching relationship between the 2D images and the 3D point clouds through feature extraction and feature matching, and then establish geometric constraint conditions according to the feature point matching relationship, so as to realize the parameter calibration of the 2D camera and the 3D camera. The specific process is generally as follows: first, target objects such as checkerboards, serrated objects, or cylinders are used as calibration objects and placed in the common field of view of the 2D camera and the 3D camera, and then the 2D images and 3D point clouds of the calibration objects are obtained. Then, features such as points, lines, and planes in the 2D images and 3D point clouds are extracted. Feature matching is performed on the 2D images and 3D point clouds according to the features, and geometric constraint calibration is established according to the feature matching relationship between the 2D images and the 3D point clouds.

[0004] However, the existing calibration plates are generally fixed in size and evenly distributed with round dots or checkerboards. The features of the round dot or checkerboard calibration plates are the same. Only in an environment with a simple background and no interference, the 2D camera and the 3D camera can obtain accurate and effective images or point clouds of the calibration plate. However, it is difficult to apply in an environment with more interference and a complex background, and the features that can be extracted based on this calibration plate are also relatively simple. Taking the round dot calibration plate as an example, usually, the entire inspection of the round dot reconstruction grid is required for matching, and the matching failure rate is high. Therefore, the joint calibration method of the 2D camera and the 3D camera based on the existing calibration plate has poor stability and poor applicability. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, device, and equipment for jointly calibrating a 2D camera and a 3D camera for the above technical problems.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a method for joint calibration of a 2D camera and a 3D camera. First, image data and point cloud data collected by the 2D camera and the 3D camera with a preset calibration board are obtained. The preset calibration board includes a plurality of solid dots with different sizes, no overlap, and non-uniform distribution. Then, the first dot data of each dot in the image data and the second dot data of each dot in the point cloud data are respectively extracted through a dot detection algorithm, so as to determine the first dot features of each dot in the image data according to the first dot data, and determine the second dot features of each dot in the point cloud data according to the second dot data. The first dot features and the second dot features can both include some more complex features preset based on the preset calibration board. Thus, subsequent feature matching can be performed according to the first dot features and the second dot features to determine the matching dots in the image data and the point cloud data, and the transformation matrix between the image data and the point cloud data can be determined according to multiple groups of matching dots.

[0008] The present invention provides a device for joint calibration of a 2D camera and a 3D camera, including:

[0009] An acquisition module, configured to acquire image data collected by the 2D camera for the preset calibration board and point cloud data collected by the 3D camera for the preset calibration board; wherein, a plurality of solid dots with different sizes, no overlap, and non-uniform distribution are marked on the preset calibration board;

[0010] An extraction module, configured to extract the first dot data of each dot in the image data and the second dot data of each dot in the point cloud data; wherein, the first dot data and the second dot data include the positions and sizes of the dots;

[0011] A determination module, configured to determine the first dot features according to the first dot data and determine the second dot features according to the second dot data; wherein, the first dot features and the second dot features include at least one of the dot radius of each dot, the distance between each dot and the center of the largest dot, and the angle between the line connecting each dot and the center of the largest dot and the line connecting the largest dot and the center of the second largest dot;

[0012] A calibration module, configured to perform feature matching according to the first dot features and the second dot features, determine the matching dots in the image data and the point cloud data, and determine the transformation matrix between the image data and the point cloud data according to multiple groups of matching dots, so as to realize the joint calibration of the 2D camera and the 3D camera.

[0013] The present invention provides a computer-readable storage medium, and the storage medium stores a computer program, and when the computer program is executed by a processor, the method for joint calibration of the 2D camera and the 3D camera as described above is implemented.

[0014] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned method for jointly calibrating a 2D camera and a 3D camera is implemented.

[0015] The above at least one technical solution adopted by the present invention can achieve the following beneficial effects:

[0016] The present invention collects data on a calibration board including a plurality of solid dots of different sizes, non-overlapping, and non-uniformly distributed by using a 2D camera and a 3D camera, which increases the complexity of the calibration board and makes the feature information contained in the calibration board richer. In an environment with more interference and a complex background, the larger dots can resist interference and ensure the stability of calibration, and the numerous non-uniformly distributed small dots can ensure the accuracy of calibration, providing better support for subsequent feature extraction. In subsequent feature extraction, the features of each dot are personalized features. Therefore, when solving the transformation matrix between image data and point cloud data, only a small number of dot matching pairs are required to obtain a relatively accurate transformation matrix, reducing the need for dot matching pairs, improving the success rate of matching calibration, and improving the stability and applicability of the calibration method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0018] Figure 1 is a schematic flowchart of a method for jointly calibrating a 2D camera and a 3D camera provided by the present invention;

[0019] Figure 2 is a schematic diagram of a preset calibration board provided by the present invention;

[0020] Figure 3 is a schematic diagram of a device for jointly calibrating a 2D camera and a 3D camera provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0022] The following will, with reference to the drawings, elaborate on the technical solutions provided by each embodiment of the present invention.

[0023] Figure 1 This is a schematic flowchart of a method for jointly calibrating a 2D camera and a 3D camera in the present invention, which specifically includes the following steps:

[0024] S101: Obtain the image data collected by the 2D camera for a preset calibration board and the point cloud data collected by the 3D camera for the preset calibration board; wherein, a plurality of solid dots with different sizes, no overlap, and non-uniform distribution are marked on the preset calibration board.

[0025] When jointly calibrating the 2D camera and the 3D camera, the server of the service platform usually needs to first obtain the image data collected by the 2D camera for the preset calibration board and the point cloud data collected by the 3D camera for the preset calibration board.

[0026] Traditional calibration boards are generally circles or checkerboards with a fixed size and uniform distribution. The basic features of the circle or checkerboard calibration board are the same, and accurate and effective calibration board features can only be obtained by 2D and 3D cameras in an environment with a simple background and no interference. However, in actual applications, it is often in an environment with more interference and a complex background. At this time, it is difficult for the 2D camera and the 3D camera to obtain the correct calibration board features, which is likely to lead to problems such as incorrect recognition of 2D features and 3D features subsequently, resulting in a low feature matching accuracy and thus calibration failure.

[0027] For this, in one or more embodiments of the present invention, the server can be based on Figure 2 the preset calibration board shown to collect the image data and the point cloud data. Figure 2 This is a schematic diagram of a preset calibration board in the present invention. As Figure 2 can be seen, the preset calibration board in the present invention includes a plurality of solid dots with different sizes, no overlap, and non-uniform distribution. Preferably, a sufficient number of dots can be set. When calculating the transformation matrix between the image data and the point cloud data subsequently, the more matching dot numbers are used, the more equations can be constructed, and the higher the final calibration accuracy will be (for example, the minimum number is not less than 40). The radius of each dot in the preset calibration board is different and has a significant difference, so that the radius of the dot can be used as a significant feature for subsequent recognition. Of course, different dots in the preset calibration board do not intersect to avoid incorrect dot recognition.

[0028] The server mentioned in the present invention can be a server set up in the service platform, or a device such as a desktop computer or a laptop computer that can execute the solution of the present invention. For the convenience of description, only the server will be used as the execution subject for description below.

[0029] S102: Respectively extract the first dot data of each dot in the image data and the second dot data of each dot in the point cloud data; wherein, the first dot data and the second dot data include the position and size of each dot.

[0030] After obtaining the image data collected by the 2D camera for the preset calibration board and the point cloud data collected by the 3D camera for the preset calibration board as described above, the server can extract the circular dots from the image data and the point cloud data respectively through the circular dot detection algorithm. Currently, there are relatively mature circular dot detection algorithms for extracting circular dots based on image data and circular dot detection algorithms for extracting circular dots based on point cloud data. The present invention will not elaborate on this.

[0031] Furthermore, when directly extracting circular dots based on point cloud data, since it is difficult to obtain the features of the point cloud, it is usually necessary to carefully design the calibration object to make the features of the point cloud easy to identify, which often limits the number of feature points and results in poor calibration accuracy. At the same time, the features of the point cloud are quite different from the image features, and the matching difficulty is relatively high, and it is easy to have the situation of matching failure.

[0032] Based on this, in one or more embodiments of the present invention, the server can first normalize the point cloud intensity of the point cloud data based on the gray value range (0 - 255) to determine the gray scale image corresponding to the point cloud data. Then, according to the gray scale image corresponding to the point cloud data, the server can extract the second circular dot data of each circular dot in the point cloud data through the circular dot detection algorithm. Thus, the extraction of the calibration board features is made simpler (the calibration board can also be designed more complex with more circular dots), the calibration board features of the image data and the point cloud data are more unified, and the subsequent matching is more robust.

[0033] S103: Determine the first circular dot features of each circular dot in the image data according to the first circular dot data, and determine the second circular dot features of each circular dot in the point cloud data according to the second circular dot data; wherein, both the first circular dot features and the second circular dot features include at least one of the circular dot radius of each circular dot, the distance between each circular dot and the center of the largest circular dot, and the angle between the line connecting each circular dot and the center of the largest circular dot and the line connecting the largest circular dot and the center of the second largest circular dot.

[0034] After obtaining the first circular dot features corresponding to the image data and the second circular dot features corresponding to the point cloud data as described above, the server can further perform circular dot feature extraction to determine the transformation matrix between the image data and the point cloud data through feature matching in the subsequent process.

[0035] Let the image data be , and the point cloud data be , and the center coordinates of each circular dot in the point cloud data can be expressed as C 3d = C 3d1 , C 3d2 , …, C 3dn , . In the formula, C3d represents the center coordinates of each dot in the point cloud data, C 3di represents the center coordinates of the i th dot in the point cloud data. n represents the total number of dots in the point cloud data. If the second dot data is directly obtained from the point cloud data, the corresponding center coordinates here are three-dimensional vectors; if the second dot data is obtained from the grayscale image corresponding to the point cloud data, the corresponding center coordinates here are two-dimensional vectors.

[0036] The dot radius of each dot in the point cloud data can be expressed as , , where R 3d represents the dot radius of each dot in the second dot feature, R 3di represents the i th dot radius in the point cloud data.

[0037] The center coordinates of each dot in the image data can be expressed as , . Where C 2d represents the center coordinates of each dot in the image data, C 2di represents the i th dot center coordinates in the image data, m represents the total number of dots in the image data.

[0038] The dot radius of each dot in the image data can be expressed as , , where R 2d represents the dot radius of each dot in the first dot feature, R 2di represents the i th dot radius in the point cloud data.

[0039] Subsequently, a descriptor of the first dot feature can be constructed based on the dot radius and center coordinates Des 2d ( R 2d , dist 2d , θ 2d ), Des 2d = Des 2d1 , Des 2d2 , …, Des2dm , where dist 2d represents the distance between the center of each dot in the image data and the center of the largest dot, θ 2d represents the angle between the line connecting the center of each dot in the image data and the center of the largest dot and the line connecting the center of the largest dot and the center of the second largest dot, Des 2dm represents the descriptor of the first dot feature of the m th dot in the image data. This is only an example, and the present invention does not limit the specific content of the first dot feature.

[0040] For the descriptor of the second dot feature constructed based on it is the same as above and can be expressed as Des 3d ( R 3d , dist 3d , θ 3d ), Des 3d = Des 3d1 , Des 3d2 , …, Des 3dn , where dist 3d represents the distance between the center of each dot in the point cloud data and the center of the largest dot, θ 3d represents the angle between the line connecting the center of each dot in the point cloud data and the center of the largest dot and the line connecting the center of the largest dot and the center of the second largest dot, Des 3dn represents the descriptor of the first dot feature of the n th dot in the point cloud data. This is only an example, and the present invention does not limit the specific content of the second dot feature.

[0041] Further, in one or more embodiments of the present invention, for each dot in the first dot data, the server may normalize the radius of the dot by the radius of the largest dot in the first dot data to determine the normalized dot radius of each dot in the first dot data. For each dot in the second dot data, the server may normalize the radius of the dot by the radius of the largest dot in the second dot data to determine the normalized dot radius of each dot in the second dot data. Thus, all circle radii are on the same scale, that is, using the radius of the largest dot for normalization is to reduce the influence brought by different scales of the 2D camera and the 3D camera, and to ensure that the descriptors of the corresponding dot features of the two are on the same scale. Similarly, the distance between the center of each dot and the center of the largest dot can also be normalized by the maximum distance.

[0042] For the angle between the line connecting each dot and the center of the largest dot and the line connecting the center of the largest dot and the center of the second largest dot, taking the image data as an example, for each dot in the image data, the server may determine the angle between the line connecting the dot and the center of the largest dot and the line connecting the center of the largest dot and the center of the second largest dot through the following formula: , .

[0043] Wherein, θ i represents the angle between the line connecting the i th dot in the image data and the center of the largest dot and the line connecting the center of the largest dot and the center of the second largest dot, represents the vector of the line connecting the center of the largest dot and the center of the second largest dot in the image data, represents the vector of the line connecting the i th dot in the image data and the center of the largest dot, | | represents the modulus operation, C second represents the coordinates of the center of the second largest dot in the image data, C max represents the coordinates of the center of the largest dot in the image data, C i represents the coordinates of the center of the i th dot in the image data. The processing of the point cloud data is the same as that of the image data, and will not be elaborated here one by one.

[0044] S104: Perform feature matching according to the first dot feature and the second dot feature, determine the matching dots in the image data and the point cloud data, and determine the transformation matrix between the image data and the point cloud data according to multiple groups of matching dots, so as to realize the joint calibration of the 2D camera and the 3D camera.

[0045] After obtaining the first dot features of each dot in the image data and the second dot features of each dot in the point cloud data as described above, the server can traverse the second dot features of each dot in the point cloud data for the first dot feature of each dot in the image data to determine the matching error between each dot in the point cloud data and this dot in the image data. Then, according to a preset error threshold, the dots in the point cloud data that match this dot in the image data are determined, as shown in the following formula: err i = pow ( R 2di - R 3di , 2)+ pow ( θ 2di - θ 3di , 2)。

[0046] In the formula, err i represents the matching error between each dot in the point cloud data and the i-th dot in the image data, and pow( ) represents the power operation function, where the first term in the parentheses is the base and the second term is the exponent.

[0047] The dot pair with the smallest matching error and satisfying abs ( dist 2di - dist 3dindex )< feature_dist can be selected as a dot matching pair . Among them, dist 2di is the distance between the center of the i-th dot in the image data and the center of the largest dot, i is the distance between the center of the j-th dot in the point cloud data and the center of the largest dot, dist 3dindex is the distance preset threshold, which can be adjusted according to the visual feature matching in different scenarios to achieve the optimal threshold in different scenarios, index is the i-th dot in the image data, is the j-th dot in the point cloud data. i i index index

[0048]

[0049] Finally, the server can establish a three-dimensional to two-dimensional projection transformation model based on the center coordinates of the matching dots in the image data and the point cloud data. And according to the center coordinates of at least 6 groups of matching dots in the image data and the point cloud data and the three-dimensional to two-dimensional projection transformation model, the transformation matrix between the image data and the point cloud data is determined.

[0049] ​​Suppose at most can be obtained k For the feature matching pairs of the first dot feature and the second dot feature, that is, dot matching pairs, the center coordinates of the corresponding dots can be found according to the dot matching pairs: for the center coordinates of the dots in the image data C 2d = C 2d1 ( u 1, v 1), C 2d2 ( u 2, v 2), …, C 2dk ( u k , v k )], for the center coordinates of the dots in the point cloud data C 3d = C 3d1 ( x 1, y 1, z 1), C 3d2 ( x 2, y 2, z 2),…, C 3dk ( x k , y k , z k )]. In the formula, ([[]] u k , v k ) represents the center coordinates of the dot in the image data that belongs to the k th dot matching pair, ([[]] x k , y k , z k ) represents the center coordinates of the dot in the point cloud data that belongs to the k th dot matching pair.

[0050] Then it can be based on C 2d = C 2d1 ( u 1, v 1), C 2d2 ( u 2, v 2), …,C 2dk ( u k , v k )] and C 3d = C 3d1 ( x 1, y 1, z 1), C 3d2 ( x 2, y 2, z 2), …, C 3dk ( x k , y k , z k )], establish a 3D to 2D projection transformation model: .

[0051] In the formula, is the scale coefficient, related to each pair of matching dots, and when expanded, it can be obtained: , using homogeneous term elimination : .

[0052] Among them, there are a total of 12 unknowns, and at least 6 pairs of matching dots are required to calculate the corresponding transformation matrix. Therefore, k the matching dots in the image data and the point cloud data can be uniformly expanded into matrix form , where: , .

[0053] Solve through Singular Value Decomposition (SVD): , and the transformation matrix between the image data and the point cloud data can be obtained. In the formula, U is the unitary orthogonal matrix obtained by singular value decomposition of matrix F , D is the diagonal matrix obtained by singular value decomposition of matrix F , V T is the transpose of the unitary orthogonal matrix obtained by singular value decomposition of matrix F .

[0054] Then, for any point ( x , y ,z ) Transform to the corresponding point in the image ( u , v ) when: , , .

[0055] Based on the joint calibration method of the 2D camera and the 3D camera shown in Figure 1 , first obtain the image data and point cloud data collected by the 2D camera and the 3D camera with a preset calibration board. The preset calibration board includes a plurality of solid dots with different sizes, no overlap, and non-uniform distribution. Then, respectively extract the first dot data of each dot in the image data and the second dot data of each dot in the point cloud data through a dot detection algorithm, so as to determine the first dot feature of each dot in the image data according to the first dot data, and determine the second dot feature of each dot in the point cloud data according to the second dot data. Both the first dot feature and the second dot feature can include some more complex features preset based on the preset calibration board, so that subsequent feature matching can be performed according to the first dot feature and the second dot feature to determine the matching dots in the image data and the point cloud data, and the transformation matrix between the image data and the point cloud data can be determined according to multiple groups of matching dots.

[0056] The present invention collects data on a calibration board including a plurality of solid dots with different sizes, no overlap, and non-uniform distribution through a 2D camera and a 3D camera, increasing the complexity of the calibration board and making the feature information contained in the calibration board more abundant. In an environment with more interference and complex background, the larger dots can resist interference and ensure the stability of calibration, and the numerous non-uniformly distributed small dots can ensure the accuracy of calibration, providing better support for subsequent feature extraction and improving the stability and applicability of the calibration method.

[0057] In the traditional calibration method, due to the uniform distribution of its calibration board, the features that can be extracted are often relatively simple and not personalized enough. Taking the traditional dot calibration board as an example, usually the entire inspection of dot reconstruction grids is required for matching, and the matching failure rate is high. In the present invention, a calibration board with non-uniform distribution in both size and position is adopted. In subsequent feature extraction, the feature of each dot is a personalized feature. Therefore, when solving the transformation matrix between the image data and the point cloud data, only a small number (at least 6 groups) of dot matching pairs are needed to obtain a relatively accurate transformation matrix, reducing the requirement for dot matching pairs and improving the success rate of matching calibration. More dot matching pairs can further improve the accuracy of the transformation matrix.

[0058] When applying the joint calibration method of the 2D camera and the 3D camera provided by the present invention, it is not necessary to execute according to the Figure 1 sequence of each step shown. The specific execution sequence of each step can be determined according to needs, and the present invention does not limit this.

[0059] The above is the method for jointly calibrating a 2D camera and a 3D camera provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding device for jointly calibrating a 2D camera and a 3D camera, as Figure 3 shown.

[0060] Figure 3 The following is a schematic diagram of a device for jointly calibrating a 2D camera and a 3D camera provided by the present invention, including:

[0061] An acquisition module 201, configured to acquire the image data collected by the 2D camera for a preset calibration board and the point cloud data collected by the 3D camera for the preset calibration board; wherein, a plurality of solid dots with different sizes, non-overlapping and non-uniformly distributed are marked on the preset calibration board;

[0062] An extraction module 202, configured to extract the first dot data of each dot in the image data and the second dot data of each dot in the point cloud data; wherein, the first dot data and the second dot data include the position and size of each dot;

[0063] A determination module 203, configured to determine the first dot feature according to the first dot data and determine the second dot feature according to the second dot data; wherein, the first dot feature and the second dot feature include at least one of the dot radius of each dot, the distance between each dot and the center of the largest dot, and the angle between the line connecting each dot and the center of the largest dot and the line connecting the largest dot and the center of the second largest dot;

[0064] A calibration module 204, configured to perform feature matching according to the first dot feature and the second dot feature, determine the matching dots in the image data and the point cloud data, and determine the transformation matrix between the image data and the point cloud data according to multiple groups of matching dots, so as to realize the joint calibration of the 2D camera and the 3D camera.

[0065] For the specific limitations on the device for jointly calibrating a 2D camera and a 3D camera, reference may be made to the limitations on the method for jointly calibrating a 2D camera and a 3D camera in the above text, which will not be elaborated here. Each module in the above device for jointly calibrating a 2D camera and a 3D camera can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0066] The present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program can be used to execute the above Figure 1 provided method for jointly calibrating a 2D camera and a 3D camera.

[0067] The present invention also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above Figure 1 provided joint calibration method for 2D camera and 3D camera.

[0068] Those of ordinary skill in the art can understand that all or part of the processes in the above-described method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0069] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope recorded by the present invention.

Claims

1. A joint calibration method for a 2D camera and a 3D camera, characterized in that, Comprising: Obtaining image data acquired by a 2D camera for a preset calibration board and point cloud data acquired by a 3D camera for the preset calibration board; wherein, a plurality of solid dots with different sizes, non-overlapping and non-uniformly distributed are marked on the preset calibration board; Extracting first dot data of each dot in the image data through a dot detection algorithm; normalizing based on a gray value interval according to the point cloud intensity of the point cloud data to determine a gray scale image corresponding to the point cloud data; extracting second dot data of each dot in the point cloud data through the dot detection algorithm according to the gray scale image corresponding to the point cloud data; wherein, the first dot data and the second dot data include the position and size of each dot; Determining first dot features of each dot in the image data according to the first dot data, and determining second dot features of each dot in the point cloud data according to the second dot data; wherein, both the first dot features and the second dot features include at least one of the dot radius of each dot, the distance between each dot and the center of the largest dot, and the angle between the line connecting each dot and the center of the largest dot and the line connecting the largest dot and the center of the second largest dot; Performing feature matching according to the first dot features and the second dot features to determine the matching dots in the image data and the point cloud data, and determining a transformation matrix between the image data and the point cloud data according to at least 6 groups of matching dots, thereby realizing the joint calibration of the 2D camera and the 3D camera.

2. The joint calibration method of the 2D camera and the 3D camera according to claim 1, characterized in that, The determining the first dot features according to the first dot data and the determining the second dot features according to the second dot data specifically include: For each dot in the first dot data, normalizing the dot radius of this dot by the radius of the largest dot in the first dot data to determine the normalized dot radius of each dot in the first dot data; For each dot in the second dot data, normalizing the dot radius of this dot by the radius of the largest dot in the second dot data to determine the normalized dot radius of each dot in the second dot data.

3. The joint calibration method of the 2D camera and the 3D camera according to claim 1, characterized in that, The determining the first dot features of each dot in the image data according to the first dot data specifically includes: For each dot in the image data, determining the angle between the line connecting this dot and the center of the largest dot and the line connecting the largest dot and the center of the second largest dot through the following formula: Among them, θ i represents the angle between the line connecting the center of the i-th dot and the center of the largest dot and the line connecting the centers of the largest dot and the second-largest dot in the image data, represents the vector connecting the centers of the largest dot and the second-largest dot in the image data, represents the vector connecting the center of the i-th dot and the center of the largest dot in the image data, || represents the modulus operation, m represents the total number of dots in the image data, C second represents the center coordinates of the second-largest dot in the image data, C max represents the center coordinates of the largest dot in the image data, C i represents the center coordinates of the i-th dot in the image data.

4. The joint calibration method of the 2D camera and the 3D camera according to claim 1, characterized in that The performing feature matching according to the first dot features and the second dot features to determine the matching dots in the image data and the point cloud data specifically includes: For the first dot features of each dot in the image data, traversing the second dot features of each dot in the point cloud data to determine the matching error between each dot in the point cloud data and this dot in the image data; According to a preset error threshold, determining the dot in the point cloud data that matches this dot in the image data.

5. The joint calibration method of the 2D camera and the 3D camera according to claim 1, characterized in that, The determining the transformation matrix between the image data and the point cloud data according to multiple groups of matching dots specifically includes: Establishing a three-dimensional to two-dimensional projection transformation model according to the center coordinates of the matching dots in the image data and the point cloud data; Determining the transformation matrix between the image data and the point cloud data according to the center coordinates of at least 6 groups of matching dots in the image data and the point cloud data and the three-dimensional to two-dimensional projection transformation model.

6. An apparatus for the joint calibration of a 2D camera and a 3D camera, characterized in that, Comprising: An acquisition module, configured to acquire image data collected by a 2D camera for a preset calibration board and point cloud data collected by a 3D camera for the preset calibration board; wherein, a plurality of solid dots with different sizes, non-overlapping and non-uniformly distributed are marked on the preset calibration board; An extraction module, configured to extract first dot data of each dot in the image data through a dot detection algorithm; normalize the point cloud intensity of the point cloud data based on a gray value interval to determine a gray scale image corresponding to the point cloud data; extract second dot data of each dot in the point cloud data through the dot detection algorithm according to the gray scale image corresponding to the point cloud data; wherein, the first dot data and the second dot data include the position and size of each dot; A determination module, configured to determine first dot features according to the first dot data, and determine second dot features according to the second dot data; wherein, the first dot features and the second dot features include at least one of the dot radius of each dot, the distance between each dot and the center of the largest dot, and the angle between the line connecting each dot and the center of the largest dot and the line connecting the largest dot and the center of the second largest dot; A calibration module, configured to perform feature matching according to the first dot features and the second dot features, determine the matching dots in the image data and the point cloud data, and determine a transformation matrix between the image data and the point cloud data according to at least 6 groups of matching dots, so as to realize the joint calibration of the 2D camera and the 3D camera.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images