Multi-channel fusion image generation method based on 2D camera and 3D camera calibration

Abstract

The application discloses a kind of based on 2D camera and 3D camera calibration multi-channel fusion map generation method, comprising: in target area fixed installation 2D\3D camera;From different angles, a plurality of 2D / 3D image data for calibration is collected by 2D\3D camera, 2D camera RGB image and 3D camera texture image are used to 2D camera and 3D camera respectively for single target calibration, then stereo calibration is carried out, and the parameter of calibration is obtained;A group of 2D / 3D image data for fusion is collected, according to the parameter of calibration, 2D camera RGB image, 3D camera space position and 3D camera normal vector, multi-channel fusion map is generated using fusion algorithm.Considering the high resolution advantage of 2D camera is combined with the space position, normal vector advantage of 3D camera, using the related technology in the field of computer vision, multi-channel fusion map with low cost, high precision and high resolution is generated.It can be used for model generation, image preprocessing, industrial detection and other tasks.

Description

Technical Field

[0001] This invention belongs to the field of image fusion processing, and particularly relates to a method for generating multi-channel fusion images based on 2D camera and 3D camera calibration. Background Technology

[0002] While existing 3D cameras can acquire information such as spatial position and normal vectors, the resolution of the images they capture is generally low. Obtaining high-resolution 3D cameras significantly increases the cost of related tasks. Therefore, a trade-off between cost and image quality is necessary when acquiring high-quality 3D images. Image fusion processing is an important research direction in computer vision, its main purpose being to fuse multiple images from different sources to obtain more information and better visual effects. High-resolution 2D cameras are used to capture high-resolution 2D images, and then low-cost 3D cameras are used to acquire the spatial position and normal vector information of objects. By designing fusion algorithms to fuse 2D and 3D images, low-cost, high-precision, and high-resolution 3D images can be generated. This can be achieved by designing specific calibration methods using camera calibration and stereo vision calibration techniques to establish the spatial relationship between the 2D and 3D cameras, which is the foundation and prerequisite for establishing the fusion algorithm. The fusion algorithm mainly uses image processing, stereo vision, and matrix processing techniques. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a method for generating multi-channel fusion images based on 2D and 3D camera calibration, which utilizes relevant technologies in the field of computer vision to generate multi-channel fusion images with low cost, high precision and high resolution.

[0004] Technical solution: The multi-channel fusion image generation method based on 2D and 3D camera calibration of the present invention includes the following steps:

[0005] Step 1: Fix the 2D camera and 3D camera on the same mechanical structure in the target area, and define the RGB image of the 2D camera, the spatial position of the 3D camera, and the normal vector of the 3D camera;

[0006] Step 2: Collect multiple sets of 2D / 3D image data for calibration from different angles using 2D and 3D cameras. Perform single-target calibration on the 2D and 3D cameras respectively using RGB images from the 2D camera and texture images from the 3D camera. Then perform stereo calibration on the results of the single-target calibration to obtain the calibration parameters.

[0007] Step 3: Acquire a set of 2D / 3D image data for fusion. Based on the calibrated parameters, 2D camera RGB image, 3D camera spatial position and 3D camera normal vector, use the fusion algorithm to generate a multi-channel fused image.

[0008] Furthermore, in step 1, the definitions of the 2D camera RGB image, the 3D camera spatial position, and the 3D camera normal vector are as follows: the high-resolution RGB image acquired by the 2D camera is called the 2D camera RGB image, defined as 2D_RGB, and its data structure is defined as a matrix. Where H 2D W 2D The numbers 3 and 3 represent height, width, and color, respectively; the texture image captured by the 3D camera is called the 3D camera texture image, defined as 3D_Texture, and its data structure is defined as a matrix. Where H 3D， W 3D These represent height and width, respectively; the spatial position information acquired by the 3D camera is called the 3D camera spatial position, defined as 3D_XYZ, and the data structure is defined as a matrix. Where H 3D W 3D The numbers 3 and 4 represent the height, width, and spatial coordinates (X, Y, Z) in the 3D camera coordinate system, respectively. The normal vector acquired by the 3D camera is called the 3D camera normal vector, defined as 3D_Normal, and its data structure is defined as a matrix. Where H 3D W 3D , 3 represent the height, width, and normal vector coordinates in the 3D camera coordinate system, respectively (N). X N Y N Z ).

[0009] Furthermore, step 2 specifically includes the following steps:

[0010] Step 2-1: Perform single-target calibration on the input N 2D_RGB images to obtain the 2D camera high-resolution intrinsic parameter matrix cameraHDMatrix2D. 3×3 High-resolution distortion parameters of 2D cameras: distHDCoeffs2D 1×5 ;

[0011] Step 2-2: Crop the input N 2D_RGB images from the top left corner using the origin of the pixel coordinate system. The aspect ratio of the resulting N 2D_RGB images will be consistent with that of the 3D_Texture, i.e., H′. 2D ∶W′ 2D =H 3D ∶W 3D ;

[0012] Step 2-3: Reduce the resolution of the result from step 2-2;

[0013] Steps 2-4: Crop the N 3D_Textures from the top left corner with the origin of the pixel coordinate system. The resulting N 3D_Textures will have the same width and height as the N 2D_RGBs obtained in Step 2-3.

[0014] Steps 2-5 involve performing single-target calibration on the results of steps 2-3 and 2-4 to obtain the intrinsic parameter matrices, distortion parameters, world coordinates, and pixel coordinates of the calibration points for both 2D and 3D cameras. The intrinsic parameter matrices are cameraMatrix2D and cameraMatrix2D, respectively. 3×3 and CameraMatrix3D 3×3 The distortion parameters are distCoeffs2D. 1×5 and distCoeffs3D 1×5 ;

[0015] Steps 2-6 involve performing stereo calibration on the results of Steps 2-5 to obtain the rotation matrix R between the camera coordinate systems of the 2D and 3D cameras. 3×4 and translation vector T 3×1 The positive transformation process of R and T is from the 2D camera coordinate system to the 3D camera coordinate system.

[0016] Furthermore, step 3 specifically includes the following steps:

[0017] Step 3-1, based on the 3D camera intrinsic parameter matrix cameraMarrix3D 3×3 and distortion parameters distCoeffs3D 1×5 For input and The image undergoes distortion correction by connecting the two coordinates at the last dimension, deforming the first two dimensions, and removing outliers (Z==0) to obtain the point set matrix 3D_XYZNor in the 3D camera coordinate system. N×6 Where N and 6 represent the number of points and channel information (X, Y, Z, N) respectively. X N Y N Z X represents the X-coordinate of the 3D camera coordinate system, Y represents the Y-coordinate of the 3D camera coordinate system, Z represents the Z-coordinate of the 3D camera coordinate system, and N represents the N-coordinate of the 3D camera coordinate system. X N represents the X-coordinate of the 3D camera coordinate system normal vector. Y N represents the y-coordinate of the 3D camera coordinate system normal vector. Z The Z-coordinate represents the normal vector of the 3D camera coordinate system.

[0018] Step 3-2, R 3×3 With T 3×1 The matrix obtained by concatenating the last dimension is denoted as RT. 3×4 Transformed into homogeneous coordinate form, i.e., RT 4×4 Its inverse matrix That is, the homogeneous coordinate transformation matrix from the 3D camera coordinate system to the 2D camera coordinate system;

[0019] Step 3-3, from step 3-1, 3D_XYZNor N×6 The first three columns are denoted as 3D_XYZ N×3 Transform it into homogeneous coordinate form, i.e., 3D_XYZ N×1×4 ;

[0020] Steps 3-4 are derived from steps 3-2 and 3-3. Where einsum is the Einstein summation function, i.e., 3D_XYZNor N×6 The spatial position is rotated and translated from the 3D camera coordinate system to the 2D camera coordinate system, and transformed into a non-homogeneous coordinate form, i.e., 2D_XYZ. N×3 ;

[0021] Steps 3-5, derived from steps 3-4, involve converting 2D_XYZ... N×3 The first column is denoted as 2D_X N×1 The second column is denoted as 2D_Y N×1 The third column is denoted as 2D_Z N×1 , in This is the Hadamard product, where the square root is an element-wise operation; that is, calculating the depth 2D_XYZ. N×3 The distance from each point in the coordinate system to the origin of the 2D camera coordinate system;

[0022] Step 3-6, from step 3-1, 3D_XYZNor N×6 The last three columns are denoted as 3D_Nor N×3 , 3D_XYZNor coming soon N×6 The normal vector in the image is rotated from the 3D camera coordinate system to the 2D camera coordinate system.

[0023] Step 3-7, based on steps 3-4, 3-5, and 3-6, will convert 2D_XYZ N×3 2D_Depth N×1 and 2D_Nor N×3 Connecting the last dimension yields the point set matrix 2D_XYZDNor in the 2D camera coordinate system. N×7 Where N and 7 represent the number of points and channel information (X, Y, Z, D, N), respectively. X N Y N Z X represents the X-coordinate of the 2D camera coordinate system, Y represents the Y-coordinate of the 2D camera coordinate system, Z represents the Z-coordinate of the 2D camera coordinate system, D represents the distance to the origin of the 2D camera coordinate system, and N represents the distance to the origin of the 2D camera coordinate system. X N represents the X-coordinate of the 2D camera coordinate system normal vector.Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z Represents the Z-coordinate of the 2D camera coordinate system normal vector;

[0024] Steps 3-8, derived from steps 3-4 and 3-7, 2D_uv N×3 =einsum(camieraMatrix2D) 3×3 ,2D_XYZ N×3 ), that is, the point set matrix 2D_XYZDNor in the 2D camera coordinate system. N×7 The spatial position of each point is projected onto the 2D pixel coordinate system via the 2D camera intrinsic parameter matrix, transforming it into non-homogeneous coordinates, i.e., 2D_uv. N×2 ;

[0025] Steps 3-9, derived from step 3-8, will convert 2D_uv N×2 Each value is scaled up proportionally to increase the resolution, where the increase in resolution is inversely proportional to the decrease in 2D camera resolution during calibration.

[0026] Step 3-10, based on steps 3-8 and 3-9, will convert 2D_uv N×2 and 2D_XYZDNor N×7 Connecting the last dimension yields the point set matrix 2D_uvXYZDNor in the 2D pixel coordinate system. N×9 Where N and 9 represent the number of points and channel information (u, v, X, Y, Z, D, N) respectively. X N Y N Z ), u represents the x-coordinate of the 2D camera pixel coordinate system, v represents the y-coordinate of the 2D camera pixel coordinate system, X represents the x-coordinate of the 2D camera coordinate system, Y represents the y-coordinate of the 2D camera coordinate system, Z represents the z-coordinate of the 2D camera coordinate system, D represents the distance to the origin of the 2D camera coordinate system, N X N represents the X-coordinate of the 2D camera coordinate system normal vector. Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z The Z-coordinate represents the normal vector of the 2D camera coordinate system.

[0027] Step 3-11, from step 3-10, f_map = interpolator(2D_uv N×2 ,2D_XYZDNor N×7 ), where interpolator is the interpolation function and f_map is the mapping function;

[0028] Step 3-12, derived from step 3-11, in H represents the height × width × number of channels in the 2D camera coordinate system.2D ×W 2D A 7x7 image with channel information (X, Y, Z, D, N). X N Y N Z ), where X represents the X coordinate of the 2D camera coordinate system, Y represents the Y coordinate of the 2D camera coordinate system, Z represents the Z coordinate of the 2D camera coordinate system, D represents the distance to the origin of the 2D camera coordinate system, and N represents the distance to the origin of the 2D camera coordinate system. X N represents the X-coordinate of the 2D camera coordinate system normal vector. Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z Represents the Z-coordinate of the 2D camera coordinate system normal vector;

[0029] Steps 3-13, based on the 2D camera high-resolution intrinsic parameter matrix cameraHDMatrix2D 3×3 and distortion parameters distHDCoeffs2D 1×5 For input Image distortion correction;

[0030] Step 3-14, based on steps 3-12 and 3-13, will and Connecting at the last latitude yields... Multi-channel fusion diagram, channel information (R, G, B, X, Y, Z, D, N) X N Y N Z In the coordinate system, R, G, and B represent red, green, and blue, respectively; X represents the X-coordinate of the 2D camera coordinate system; Y represents the Y-coordinate of the 2D camera coordinate system; Z represents the Z-coordinate of the 2D camera coordinate system; D represents the distance to the origin of the 2D camera coordinate system; and N represents the distance to the origin of the 2D camera coordinate system. X N represents the X-coordinate of the 2D camera coordinate system normal vector. Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z The Z-coordinate represents the normal vector of the 2D camera coordinate system.

[0031] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: It combines the high-resolution advantage of 2D cameras with the spatial position and normal vector advantages of 3D cameras, utilizing relevant technologies in the field of computer vision to generate multi-channel fused images with low cost, high precision, and high resolution. The pixels in the multi-channel fused image contain richer information than ordinary images (RGB, grayscale, bitmap, duotone), such as spatial position, depth, and normal vectors, and can be used for tasks such as model generation, image preprocessing, and industrial inspection. Attached Figure Description

[0032] Figure 1 It is a method flowchart;

[0033] Figure 2 This is an example diagram of the camera's fixed mechanical structure; circle 1 represents a 2D camera, and circle 2 represents a 3D camera.

[0034] Figure 3 It is a set of 2D / 3D data visualizations; where (a) is a 2D camera RGB image - 2D_RGB; (b) is a 3D camera texture image - 3D_Texture; (c) is the 3D camera spatial position - 3D_XYZ and the 3D camera normal vector - 3D_Normal;

[0035] Figure 4 It is a calibration flowchart;

[0036] Figure 5 This is a flowchart of the fusion algorithm; Detailed Implementation

[0037] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0038] Embodiments of the present invention propose a method for generating multi-channel fusion maps based on 2D / 3D camera calibration, such as... Figure 1 The process includes: fixing 2D and 3D cameras to a mechanical structure; acquiring multiple sets of 2D / 3D data for calibration from different angles; performing single-target calibration on the 2D and 3D cameras using RGB images from the 2D camera and texture images from the 3D camera, followed by stereo calibration; acquiring a set of 2D / 3D data for fusion; and generating a multi-channel fused image using a fusion algorithm based on the calibration parameters, RGB images from the 2D camera, the spatial position of the 3D camera, and the normal vector of the 3D camera.

[0039] In embodiments of the present invention, 2D and 3D cameras are fixed to a mechanical structure, for example as follows: Figure 2 .

[0040] In embodiments of the present invention, a set of 2D / 3D data visualizations, such as Figure 3 High-resolution RGB images captured by a 2D camera are simply referred to as 2D camera RGB images, abbreviated as 2D_RGB, and their data structure is defined as a matrix. Where H 2D W 2D The numbers 3 and 3 represent the height, width, and color, respectively. The texture image captured by a 3D camera is simply called a 3D camera texture image, abbreviated as 3D_Texture, and its data structure is defined as a matrix. Where H 3D W 3D These represent height and width, respectively. The spatial position information acquired by the 3D camera is simply referred to as 3D camera spatial position, abbreviated as 3D_XYZ, and its data structure is defined as a matrix. Where H 3D W3D 3 represents the height, width, and spatial coordinates (X, Y, Z) in the 3D camera coordinate system, respectively. The normal vector acquired by the 3D camera is simply called the 3D camera normal vector, abbreviated as 3D_Normal, and its data structure is defined as a matrix. Where H 3D W 3D , 3 represent the height, width, and normal vector coordinates in the 3D camera coordinate system, respectively (N). X N Y N Z ).

[0041] In this embodiment of the invention, multiple sets of 2D / 3D data for calibration are acquired from different angles. Single-target calibration of the 2D and 3D cameras is performed using RGB images from the 2D camera and texture images from the 3D camera, followed by stereo calibration. Figure 4 The specific process is as follows:

[0042] Step 1: Perform single-target calibration on the input N 2D_RGB images to obtain the 2D camera high-resolution intrinsic parameter matrix cameraHDMatrix2D. 3×3 High-resolution distortion parameters of 2D cameras: distHDCoeffs2D 1×5 ;

[0043] Step 2: Crop the input N 2D_RGB images from the top left corner using the origin of the pixel coordinate system. The aspect ratio of the resulting N 2D_RGB images will be consistent with that of the 3D_Texture, i.e., H′. 2D ∶W′ 2D =H 3D ∶W 3D ;

[0044] Step 3: Reduce the resolution of the result from Step 2;

[0045] Step 4: Crop the N 3D_Textures from the top left corner with the origin of the pixel coordinate system. The resulting N 3D_Textures will have the same width and height as the N 2D_RGBs obtained in Step 3.

[0046] Step 5: Perform single-target calibration on the results of Steps 3 and 4 to obtain the intrinsic parameter matrices, distortion parameters, world coordinates (objectPoints), and pixel coordinates (imagePoints2D, imagePoints3D) of the calibration points for the 2D and 3D cameras, where the intrinsic parameter matrices are cameraMatrix2D and cameraMatrix2D, respectively. 3×3 and CameraMatrix3D 3×3 The distortion parameters are distCoeffs2D. 1×5 and distCoeffs3D 1×5 .

[0047] Step 6: Perform stereo calibration on the results of Step 5 to obtain the rotation matrix R between the camera coordinate systems of the 2D and 3D cameras. 3×4 and translation vector T 3×1 The positive transformation process of R and T is from the 2D camera coordinate system to the 3D camera coordinate system.

[0048] In this embodiment of the invention, a set of 2D / 3D data for fusion is acquired. Based on the calibrated parameters, the RGB image from the 2D camera, the spatial position of the 3D camera, and the normal vector of the 3D camera, a multi-channel fused image is generated using a fusion algorithm, such as... Figure 5 The specific process of the fusion algorithm:

[0049] Step 1, using the 3D camera intrinsic parameter matrix cameraMotrix3D 3×3 and distortion parameters distCoeffs3D 1×5 For input and The image undergoes distortion correction, and the two coordinates are further connected at the last dimension, deformed in the first two dimensions, and outliers are removed (Z == 0) to obtain the point set matrix 3D_XYZNor in the 3D camera coordinate system. N×6 Where N and 6 represent the number of points and channel information (X, Y, Z, N) respectively. X N Y N Z X represents the X-coordinate of the 3D camera coordinate system, Y represents the Y-coordinate of the 3D camera coordinate system, Z represents the Z-coordinate of the 3D camera coordinate system, and N represents the N-coordinate of the 3D camera coordinate system. X N represents the X-coordinate of the 3D camera coordinate system normal vector. Y N represents the y-coordinate of the 3D camera coordinate system normal vector. Z Represents the Z-coordinate of the 3D camera coordinate system normal vector;

[0050] Step 2, R 3×3 With T 3×1 The matrix obtained by concatenating the last dimension is denoted as RT. 3×4 This is further transformed into homogeneous coordinate form, i.e., RT. 4×4 Its inverse matrix That is, the homogeneous coordinate transformation matrix from the 3D camera coordinate system to the 2D camera coordinate system;

[0051] Step 3, from Step 1, 3D_XYZNor N×6 The first three columns are denoted as 3D_XYZ N×3 This is further transformed into homogeneous coordinate form, i.e., 3D_XYZ. N×1×4 ;

[0052] Step 4, derived from steps 2 and 3, Where einsum is the Einstein summation function, i.e., 3D_XYZNor N×6 The spatial position is rotated and translated from the 3D camera coordinate system to the 2D camera coordinate system, and further transformed into a non-homogeneous coordinate form, i.e., 2D_XYZ. N×3 ;

[0053] Step 5, from step 4, 2D_XYZ N×3 The first column is denoted as 2D_X N×1 The second column is denoted as 2D_Y N×1 The third column is denoted as 2D-Z. N×1 ,

[0054] in This is the Hadamard product, where the square root sign represents an element-wise operation. It calculates the depth – 2D_XYZ. N×3 The distance from each point in the coordinate system to the origin of the 2D camera coordinate system;

[0055] Step 6, from step 1, 3D_XYZNor N×6 The last three columns are denoted as 3D_Nor N×3 ,

[0056] 3D_XYZNor coming soon N×6 The normal vector in the image is rotated from the 3D camera coordinate system to the 2D camera coordinate system.

[0057] Step 7, based on steps 4, 5, and 6, transforms 2D_XYZ N×3 2D_Depth N×1 and 2D_Nor N×3 Connecting the last dimension yields the point set matrix 2D_XYZDNor in the 2D camera coordinate system. N×7 Where N and 7 represent the number of points and channel information (X, Y, Z, D, N), respectively. X N Y N Z X represents the X-coordinate of the 2D camera coordinate system, Y represents the Y-coordinate of the 2D camera coordinate system, Z represents the Z-coordinate of the 2D camera coordinate system, D represents the distance to the origin of the 2D camera coordinate system, and N represents the distance to the origin of the 2D camera coordinate system. X N represents the X-coordinate of the 2D camera coordinate system normal vector. Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z Represents the Z-coordinate of the 2D camera coordinate system normal vector;

[0058] Step 8, derived from steps 4 and 7, 2D_uv N×3 =einsum(cameraMatrix2D) 3×3 ,2D_XYZ N×3), that is, the point set matrix 2D_XYZDNor in the 2D camera coordinate system. N×7 The spatial position of each point is projected onto the 2D pixel coordinate system via the 2D camera intrinsic parameter matrix, and further transformed into non-homogeneous coordinates, i.e., 2D_uv. N×2 ;

[0059] Step 9, from step 8, 2D_uv N×2 Each value is scaled up proportionally to increase the resolution, where the increase in resolution is inversely proportional to the decrease in 2D camera resolution during calibration.

[0060] Step 10, based on steps 8 and 9, will convert 2D_uv N×2 and 2D_XYZDNor N×7 Connecting the last dimension yields the point set matrix 2D_uvXYZDNor in the 2D pixel coordinate system. N×9 Where N and 9 represent the number of points and channel information (u, v, X, Y, Z, D, N) respectively. X N Y N Z ), u represents the x-coordinate of the 2D camera pixel coordinate system, v represents the y-coordinate of the 2D camera pixel coordinate system, X represents the x-coordinate of the 2D camera coordinate system, Y represents the y-coordinate of the 2D camera coordinate system, Z represents the z-coordinate of the 2D camera coordinate system, D represents the distance to the origin of the 2D camera coordinate system, N X N represents the X-coordinate of the 2D camera coordinate system normal vector. Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z Represents the Z-coordinate of the 2D camera coordinate system normal vector;

[0061] Step 11, from step 10, f_map = interpolator(2D_uv N×2 ,2D_XYZDNor N×7 ), where interpolator is the interpolation function and f_map is the mapping function;

[0062] Step 12, from step 11, in H represents the height × width × number of channels in the 2D camera coordinate system. 2D ×W 2D A 7x7 image with channel information (X, Y, Z, D, N). X N Y N Z X represents the X-coordinate of the 2D camera coordinate system, Y represents the Y-coordinate of the 2D camera coordinate system, Z represents the Z-coordinate of the 2D camera coordinate system, D represents the distance to the origin of the 2D camera coordinate system, and N represents the distance to the origin of the 2D camera coordinate system. X N represents the X-coordinate of the 2D camera coordinate system normal vector.Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z Represents the Z-coordinate of the 2D camera coordinate system normal vector;

[0063] Step 13, using the 2D camera high-resolution intrinsic parameter matrix cameraHDMatrix2D 3×3 and distortion parameters distHDCoeffs2D 1×5 For input Image distortion correction;

[0064] Step 14, based on steps 12 and 13, will and Connecting at the last latitude yields... Multi-channel fusion diagram, channel information (R, G, B, X, Y, Z, D, N) X N Y N Z In the coordinate system, R, G, and B represent red, green, and blue, respectively; X represents the X-coordinate of the 2D camera coordinate system; Y represents the Y-coordinate of the 2D camera coordinate system; Z represents the Z-coordinate of the 2D camera coordinate system; D represents the distance to the origin of the 2D camera coordinate system; and N represents the distance to the origin of the 2D camera coordinate system. X N represents the X-coordinate of the 2D camera coordinate system normal vector. Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z The Z-coordinate represents the normal vector of the 2D camera coordinate system.

Claims

1. A method for generating multi-channel fusion images based on 2D and 3D camera calibration, characterized in that, Includes the following steps: Step 1: Fix the 2D camera and 3D camera on the same mechanical structure in the target area, and define the RGB image of the 2D camera, the spatial position of the 3D camera, and the normal vector of the 3D camera; Step 2: Collect multiple sets of 2D / 3D image data for calibration from different angles using 2D and 3D cameras. Perform single-target calibration on the 2D and 3D cameras respectively using RGB images from the 2D camera and texture images from the 3D camera. Then perform stereo calibration on the results of the single-target calibration to obtain the calibration parameters. Step 3: Acquire a set of 2D / 3D image data for fusion. Based on the calibrated parameters, 2D camera RGB image, 3D camera spatial position and 3D camera normal vector, use the fusion algorithm to generate a multi-channel fused image. Steps 3-7: Convert the non-homogeneous coordinates into their original form. Calculation depth and 2D camera coordinate system normal vector Connecting at the last dimension yields the point set matrix in the 2D camera coordinate system. Where N and 7 represent the number of points and channel information (X, Y, Z, D, N), respectively. X N Y N Z ); Steps 3-8: Convert the point set matrix in the 2D camera coordinate system. The spatial position of each point is projected onto the 2D pixel coordinate system via the 2D camera intrinsic parameter matrix, thus transforming it into non-homogeneous coordinates. ; Steps 3-9, will Each value is scaled up proportionally to increase the resolution, where the increase in resolution is inversely proportional to the decrease in 2D camera resolution during calibration. Steps 3-10, will and Connecting at the last dimension yields a point set matrix in a 2D pixel coordinate system. Where N and 9 represent the number of points and channel information (u,v,X,Y,Z,D,N) respectively. X N Y N Z ), u represents the x-coordinate of the 2D camera pixel coordinate system, and v represents the y-coordinate of the 2D camera pixel coordinate system. Steps 3-11 Where interpolator is the interpolation function and f_map is the mapping function; Steps 3-12 ,in This represents the height × width × number of channels in the 2D camera coordinate system. Image, channel information (X,Y,Z,D,N) X N Y N Z ); Step 3-13, based on the high-resolution intrinsic parameter matrix of the 2D camera. and distortion parameters For input Image distortion correction; Step 3-14, based on steps 3-12 and 3-13, will and Connecting at the last latitude yields... Multi-channel fusion diagram, channel information (R, G, B, X, Y, Z, D, N) X N Y N Z In the coordinate system, R, G, and B represent red, green, and blue, respectively; X represents the X-coordinate of the 2D camera coordinate system, Y represents the Y-coordinate of the 2D camera coordinate system, Z represents the Z-coordinate of the 2D camera coordinate system, D represents the distance to the origin of the 2D camera coordinate system, and N represents the distance to the origin of the 2D camera coordinate system. X N represents the X-coordinate of the 2D camera coordinate system normal vector. Y N represents the y-coordinate of the 2D camera coordinate system normal vector. Z The Z-coordinate represents the normal vector of the 2D camera coordinate system.

2. The method for generating a multi-channel fusion image based on 2D and 3D camera calibration according to claim 1, characterized in that, In step 1, the 2D camera RGB image, 3D camera spatial position, and 3D camera normal vector are defined as follows: The high-resolution RGB image acquired by the 2D camera is called the 2D camera RGB image, defined as 2D_RGB, and its data structure is defined as a matrix. ,in These represent height, width, and color, respectively; the texture image captured by the 3D camera is called the 3D camera texture image, defined as 3D_Texture, and its data structure is defined as a matrix. ,in These represent the height and width, respectively. The spatial position information acquired by the 3D camera is called the 3D camera spatial position, defined as 3D_XYZ, and the data structure is defined as a matrix. ,in These represent the height, width, and spatial coordinates (X, Y, Z) in the 3D camera coordinate system, respectively. The normal vector acquired by the 3D camera is called the 3D camera normal vector, defined as 3D_Normal, and its data structure is defined as a matrix. ,in These represent the height, width, and normal vector coordinates in the 3D camera coordinate system, respectively (N). X, N Y N Z ).

3. The method for generating a multi-channel fusion image based on 2D and 3D camera calibration according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2-1: Perform single-target calibration on the input N 2D_RGB images to obtain the 2D camera high-resolution intrinsic parameter matrix. High-resolution distortion parameters of 2D cameras ; Step 2-2: Crop the input N 2D_RGB images from the top left corner using the origin of the pixel coordinate system. The resulting N 2D_RGB images will have the same aspect ratio as the 3D_Texture. ; Step 2-3: Reduce the resolution of the result from step 2-2; Steps 2-4: Crop the N 3D_Textures from the top left corner with the origin of the pixel coordinate system. The resulting N 3D_Textures will have the same width and height as the N 2D_RGBs obtained in Step 2-3. Steps 2-5 involve performing single-target calibration on the results of steps 2-3 and 2-4 to obtain the intrinsic parameter matrices, distortion parameters, world coordinates, and pixel coordinates of the calibration points for both the 2D and 3D cameras. The intrinsic parameter matrices are as follows: and The distortion parameters are respectively and ; Steps 2-6 involve stereo calibration of the results from Steps 2-5 to obtain the rotation matrix between the camera coordinate systems of the 2D and 3D cameras. and translation vector The positive transformation process of R and T is from the 2D camera coordinate system to the 3D camera coordinate system.

4. The method for generating a multi-channel fusion image based on 2D and 3D camera calibration according to claim 3, characterized in that, Step 3 specifically includes the following steps: Step 3-1, based on the 3D camera intrinsic parameter matrix and distortion parameters For input and Image distortion correction involves connecting the two images in the last dimension, deforming the first two dimensions, and removing outliers. The point set matrix in the 3D camera coordinate system is obtained. Where N and 6 represent the number of points and channel information (X, Y, Z, N), respectively. X N Y N Z X represents the X-coordinate of the 3D camera coordinate system, Y represents the Y-coordinate of the 3D camera coordinate system, Z represents the Z-coordinate of the 3D camera coordinate system, and N represents the N-coordinate of the 3D camera coordinate system. X N represents the X-coordinate of the 3D camera coordinate system normal vector. Y N represents the y-coordinate of the 3D camera coordinate system normal vector. Z The Z-coordinate represents the normal vector of the 3D camera coordinate system. Step 3-2, and The matrix obtained by splicing the last dimension is denoted as Transforming into homogeneous coordinate form, i.e. Its inverse matrix That is, the homogeneous coordinate transformation matrix from the 3D camera coordinate system to the 2D camera coordinate system; Step 3-3, from step 3-1, will The first 3 columns are denoted as Transforming into homogeneous coordinate form, i.e. ; Steps 3-4 are derived from steps 3-2 and 3-3. Where einsum is Einstein's summation function, which is... The spatial position is rotated and translated from the 3D camera coordinate system to the 2D camera coordinate system, and transformed into a non-homogeneous coordinate form. ; Steps 3-5, derived from steps 3-4, will The first column is denoted as The second column is denoted as The third column is denoted as , ,in This is the Hadamard product, where the square root sign represents an element-wise operation; that is, calculating the depth. The distance from each point in the coordinate system to the origin of the 2D camera coordinate system; Step 3-6, from step 3-1, will The last three columns are denoted as , Soon The normal vector in the image is rotated from the 3D camera coordinate system to the 2D camera coordinate system.

Images