A dense point cloud generation method based on multi-view infrared

Abstract

The present invention relates to a dense point cloud generation method based on multi-view infrared, comprising: obtaining the internal and external parameters of an infrared camera; using a multi-view low-resolution infrared camera and a high-resolution infrared camera to obtain a multi-view first low-resolution infrared image and a first high-resolution infrared image; constructing a data set; inputting the data set into a dense point cloud generation network to be trained until the loss function converges, thereby obtaining a trained dense point cloud generation network; inputting the multi-view low-resolution infrared image to be generated into the trained dense point cloud generation network, thereby obtaining a disparity map and a high-resolution infrared image of the multi-view low-resolution infrared image, thereby generating a dense point cloud. The present invention adopts a multi-view pure vision solution, which has a stronger anti-interference ability, can avoid the limitations that laser radars interfere with each other, active infrared and other ranging solutions cannot be applied to strong light environments such as outdoor environments, so that the present invention improves the robustness of the constructed system.

Description

Technical Field

[0001] The invention belongs to the technical field of infrared image processing and relates to a dense point cloud generation method based on multi-view infrared. Background Art

[0002] Point cloud data is used to describe the three-dimensional appearance of a scene. By using the data of all points in a certain three-dimensional coordinate system, the data includes the three-dimensional coordinates X, Y, Z and intensity of each point. There are many ways to collect point cloud data, which can be obtained through methods such as laser radar, RGB-D camera, and binocular vision. Point cloud data contains a large amount of three-dimensional structural information, so it has a wide range of applications in modeling, mapping, autonomous driving, and medical treatment. Point cloud data can be divided into sparse point cloud and dense point cloud according to the density of points in the point cloud data. The density of points in dense point cloud is high, so it is more conducive to improving the effect of downstream task processing. In recent years, artificial intelligence has developed rapidly, and deep learning algorithms have a trend of gradually replacing traditional algorithms in all aspects. Deep learning algorithms have excellent performance in stereo matching and binocular image super-resolution reconstruction tasks. Multi-view images contain the three-dimensional relationship of the scene, and the sampling density is higher than that of monocular images, so they contain richer information, which can improve the effect of generating point clouds. However, due to the differences in parallax, color difference, resolution, etc. in the input multi-view images, how to solve the problem of alignment and registration between multi-view images is a challenge faced when using deep learning neural network algorithms. Solving the registration of multi-view images is beneficial to fully explore the information in multi-view images and improve the effect of point cloud generation.

[0003] Chengdu Hangwei Zhixin Technology Co., Ltd. proposed an airborne lidar point cloud generation method in its applied patent document "Airborne LiDAR Point Cloud Generation Method and System" (patent application number: 2021113368882, application publication number: CN113933861A). The steps implemented by the method are: 1. Acquire GNSS data and IMU data in the current calculation cycle as the first data; 2. Acquire the first pose information of the carrier in the current calculation cycle based on the first data and the optimized first pose information in the previous calculation cycle; 3. Acquire the lidar data in the current scanning cycle as the second data, match the second data with the latest local map, and obtain the second pose information of the carrier in the current scanning cycle; 4. Fusion filter the latest first pose information and the latest second pose information in the current filtering cycle to obtain the third pose information and pose information error corresponding to the current filtering cycle; 5. Optimize the first pose information of the current calculation cycle based on the latest pose information error to obtain the optimized first pose information in the current calculation cycle; 6. Perform spatial transformation on the third pose information and the second data to obtain point cloud data in the current filtering cycle, and generate a local map based on the point cloud data in multiple filtering cycles.

[0004] However, the method has the disadvantage that, since it uses lidar as the acquisition device for point cloud generation, on the one hand, the equipment cost is relatively high and it is easy to be interfered with by other devices; on the other hand, if other point cloud information such as infrared intensity information is needed, it needs to be aligned before it can be used and the upper limit of the resolution is limited by the maximum resolution of the device. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method for generating dense point clouds based on multi-view infrared. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] An embodiment of the present invention provides a method for generating a dense point cloud based on multi-view infrared, and the method for generating a dense point cloud includes:

[0007] Obtain the internal and external parameters of multi-eye low-resolution infrared cameras and high-resolution infrared cameras;

[0008] Using a multi-eye low-resolution infrared camera and a high-resolution infrared camera to obtain a multi-eye first low-resolution infrared image and a first high-resolution infrared image;

[0009] Constructing a data set, the data set including internal and external parameters of the multi-eye low-resolution infrared camera and the high-resolution infrared camera, a multi-eye second low-resolution infrared image and a second high-resolution infrared image, wherein the second low-resolution infrared image is obtained by intercepting the first low-resolution infrared image, and the second high-resolution infrared image is obtained by intercepting the first high-resolution infrared image;

[0010] Inputting the data set into a dense point cloud generation network to be trained until the loss function converges, thereby obtaining a trained dense point cloud generation network, wherein the dense point cloud generation network includes a depth estimation subnetwork, a multi-eye infrared image registration module, and a multi-eye super-resolution subnetwork;

[0011] The multi-eye low-resolution infrared images to be generated are input into the trained dense point cloud generation network to obtain the disparity map and high-resolution infrared image of the multi-eye low-resolution infrared image, so as to generate a dense point cloud according to the disparity map and high-resolution infrared image of the multi-eye low-resolution infrared image.

[0012] In one embodiment of the present invention, obtaining the internal parameters and external parameters of multiple low-resolution infrared cameras and high-resolution infrared cameras includes:

[0013] Use multiple low-resolution infrared cameras and high-resolution infrared cameras to simultaneously shoot infrared calibration targets to obtain several groups of calibration images;

[0014] The calibration images are used to calibrate to obtain the intrinsic and extrinsic parameters of all cameras.

[0015] In one embodiment of the present invention, the data set is input into a dense point cloud generation network to be trained until the loss function converges, thereby obtaining a trained dense point cloud generation network, including:

[0016] Inputting the data set into the depth estimation subnetwork to obtain a disparity map of the multi-eye second low-resolution infrared image, and converting the disparity map of the multi-eye second low-resolution infrared image into a depth map;

[0017] Based on the projection matrix between the low-resolution infrared camera and the high-resolution infrared camera and the depth map of the multi-eye second low-resolution infrared image, the multi-eye second low-resolution infrared image is registered by a projection method in the multi-eye infrared image registration module to obtain a multi-eye registered second low-resolution infrared image;

[0018] Inputting the 2-channel data obtained by splicing the second low-resolution infrared image after each eye registration and the depth map corresponding to the second low-resolution infrared image into the multi-eye super-resolution sub-network to obtain a high-resolution infrared image;

[0019] The loss function is optimized by the gradient descent method, and the parameters of the dense point cloud generation network are iteratively updated until the loss function converges to obtain a trained dense point cloud generation network.

[0020] In one embodiment of the present invention, the loss function is:

[0021] L=λ1L SR +λ2L REP

[0022]

[0023] L REP =∑||I rep -I lr ||1

[0024] Among them, L represents the loss function of the dense point cloud generation network, L SR represents the super-resolution loss, L REP represents the reprojection loss, represents the mean square error operation, ||·||1 represents the one-norm operation, I sr represents the super-resolution infrared image, I hr Represents the reprojected infrared image of the high-resolution infrared image in the dataset projected to the position of the multi-eye infrared camera, I rep Represents an image where a low-resolution infrared image is reprojected to the position of another low-resolution infrared image, I lrrepresents the low-resolution infrared image in the dataset, λ1 and λ2 represent the weights of super-resolution loss and reprojection loss.

[0025] In one embodiment of the present invention, the training process of the depth estimation subnetwork is:

[0026] Inputting the third low-resolution infrared image and the fourth low-resolution infrared image into the depth estimation subnetwork to obtain a disparity map of the third low-resolution infrared image;

[0027] Obtaining a projection image of the fourth low-resolution infrared image onto the third low-resolution infrared image based on the fourth low-resolution infrared image, a depth map corresponding to the disparity map of the third low-resolution infrared image, and internal and external parameters;

[0028] The loss function of the depth estimation subnetwork is optimized by the gradient descent method, and the parameters of the depth estimation subnetwork are iteratively updated until the loss function of the depth estimation subnetwork converges, thereby obtaining a trained depth estimation subnetwork.

[0029] In one embodiment of the present invention, the structure of the depth estimation subnetwork includes:

[0030] Input layer, first convolution layer, first residual layer, second convolution layer, second residual layer, third convolution layer, third residual layer, fourth convolution layer, fourth residual layer, fifth convolution layer, fifth residual layer, sixth convolution layer, sixth residual layer, seventh convolution layer, seventh residual layer, eighth convolution layer, eighth residual layer, ninth convolution layer, disparity cost aggregation layer, first 3D convolution layer, first 3D residual layer, second 3D convolution layer, third 3D convolution layer, second 3D residual layer, fourth 3D convolution layer, fifth 3D convolution layer, third 3D residual layer, sixth 3D convolution layer, seventh 3D convolution layer, fourth 3D residual layer, eighth 3D convolution layer, ninth 3D convolution layer, fifth 3D residual layer, first 3D deconvolution layer, second 3D deconvolution layer, third 3D deconvolution layer, fourth 3D deconvolution layer, fifth 3D deconvolution layer, disparity regression layer;

[0031] The input of the Mth convolutional layer is the sum of the Nth convolutional layer and the Nth residual layer, N is one of one to eight, and M is N plus one; the inputs of the second 3D convolutional layer, the fourth 3D convolutional layer, the sixth 3D convolutional layer, and the eighth 3D convolutional layer are respectively the output of the disparity cost aggregation layer, the output of the second 3D convolutional layer, the output of the fourth 3D convolutional layer, and the output of the sixth 3D convolutional layer; the inputs of the second 3D deconvolutional layer, the third 3D deconvolutional layer, the fourth 3D deconvolutional layer, and the fifth 3D deconvolutional layer are respectively the output of the first 3D deconvolutional layer and the output of the fourth 3D residual layer, the output of the second 3D deconvolutional layer and the output of the third 3D residual layer, the output of the third 3D deconvolutional layer and the output of the second 3D residual layer, and the output of the fourth 3D deconvolutional layer and the output of the first 3D residual layer.

[0032] In one embodiment of the present invention, the convolution kernel size of the input layer is set to 5×5, the step size is set to 2, and the output feature map channel size is set to 32;

[0033] The convolution kernel sizes of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer, the seventh convolution layer, the eighth convolution layer, and the ninth convolution layer are all set to 3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 32;

[0034] The convolution kernel sizes of the first residual layer, the second residual layer, the third residual layer, the fourth residual layer, the fifth residual layer, the sixth residual layer, the seventh residual layer, and the eighth residual layer are all set to 3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 32;

[0035] The convolution kernel size of the first 3D convolutional layer is set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 32;

[0036] The convolution kernel sizes of the second 3D convolution layer, the fourth 3D convolution layer, and the sixth 3D convolution layer are all set to 3×3×3, the step sizes are all set to 2, and the output feature map channel sizes are all set to 64;

[0037] The convolution kernel sizes of the third 3D convolution layer, the fifth 3D convolution layer, and the seventh 3D convolution layer are all set to 3×3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 64;

[0038] The convolution kernel sizes of the eighth 3D convolution layer and the ninth 3D convolution layer are both set to 3×3×3, the step sizes are both set to 2, and the output feature map channel sizes are both set to 128;

[0039] The convolution kernel size of the first 3D residual layer is set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 32;

[0040] The convolution kernel sizes of the second 3D residual layer, the third 3D residual layer, and the fourth 3D residual layer are all set to 3×3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 64;

[0041] The convolution kernel size of the fifth 3D residual layer is set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 128;

[0042] The convolution kernel size of the first 3D deconvolution layer, the second 3D deconvolution layer, and the third 3D deconvolution layer is set to 3×3×3, the step size is set to 2, and the output feature map channel size is set to 64;

[0043] The convolution kernel size of the fourth 3D deconvolution layer is set to 3×3×3, the step size is set to 2, and the output feature map channel size is set to 32;

[0044] The convolution kernel size of the fifth 3D deconvolution layer is set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 1.

[0045] In one embodiment of the present invention, the structure of the multi-eye super-resolution sub-network includes:

[0046] feature extraction layer, tenth convolution layer, ninth residual layer, eleventh convolution layer, tenth residual layer, twelfth convolution layer, eleventh residual layer, thirteenth convolution layer, twelfth residual layer, fourteenth convolution layer, thirteenth residual layer, fifteenth convolution layer, fourteenth residual layer, sixteenth convolution layer, fifteenth residual layer, seventeenth convolution layer, sixteenth residual layer, eighteenth convolution layer, seventeenth residual layer, nineteenth convolution layer, eighteenth residual layer, twentieth convolution layer, nineteenth residual layer, twenty-first convolution layer, twentieth residual layer, twenty-second convolution layer, twenty-first residual layer, twenty-third convolution layer, twenty-second residual layer, twenty-fourth convolution layer, twenty-third residual layer, twenty-fifth convolution layer, twenty-fourth residual layer, pixel recombination layer, output layer;

[0047] The feature extraction layer shares weights, and its input is two-channel data spliced ​​from a low-resolution infrared image after projection alignment and its corresponding depth map; the tenth convolutional layer input is a spliced ​​feature map of multiple low-resolution images in the channel direction through multi-channel output feature maps of the feature extraction layer with shared weights; the input of the remaining convolutional layers is the output of the previous convolutional layer plus the output of the previous residual layer; the pixel reconstruction layer input is the output of the twenty-fourth residual layer plus the output of the feature extraction layer.

[0048] In one embodiment of the present invention, the convolution kernel size of the feature extraction layer is set to 3×3, the step size is set to 1, and the output feature map channel size is k×64, where k is the number of eyes of the multi-eye low-resolution infrared camera;

[0049] The convolution kernel sizes of the tenth convolution layer, the eleventh convolution layer, the twelfth convolution layer, the thirteenth convolution layer, the fourteenth convolution layer, the fifteenth convolution layer, the sixteenth convolution layer, the seventeenth convolution layer, the eighteenth convolution layer, the nineteenth convolution layer, the twentieth convolution layer, the twenty-first convolution layer, the twenty-second convolution layer, the twenty-third convolution layer, the twenty-fourth convolution layer, and the twenty-fifth convolution layer are all set to 3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 64;

[0050] The convolution kernel sizes of the ninth residual layer, the tenth residual layer, the eleventh residual layer, the twelfth residual layer, the thirteenth residual layer, the fourteenth residual layer, the fifteenth residual layer, the sixteenth residual layer, the seventeenth residual layer, the eighteenth residual layer, the nineteenth residual layer, the twentieth residual layer, the twenty-first residual layer, the twenty-second residual layer, and the twenty-third residual layer are all set to 3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 64;

[0051] The convolution kernel size of the 24th residual layer is set to 3×3, the step size is set to 1, and the output feature map channel size is set to k×64;

[0052] The magnification factor of the pixel reorganization layer is set to s, and the output feature map channel size is set to 64, where s is the multiple of super-resolution image reconstruction, s=2 n , n is an integer greater than or equal to 1;

[0053] The convolution kernel size of the output layer is set to 3×3, the step size is set to 1, and the output feature map channel size is set to 2.

[0054] In one embodiment of the present invention, generating a dense point cloud based on a disparity map of multiple low-resolution infrared images and a high-resolution infrared image includes:

[0055] A multi-eye low-resolution depth map is obtained according to the disparity map of the multi-eye low-resolution infrared image;

[0056] The low-resolution depth maps of the multiple eyes are interpolated and up-sampled to obtain high-resolution depth maps of the multiple eyes;

[0057] Calculate the average value of multiple high-resolution depth maps and use the average value as the final high-resolution depth map;

[0058] The three-dimensional coordinates of the dense point cloud are obtained according to the inverse matrix of the internal reference of the high-resolution infrared image, the final high-resolution depth map and the pixel coordinates of the high-resolution infrared image;

[0059] The pixel values ​​of the high-resolution infrared image are assigned to the dense point cloud according to the pixel coordinates to generate a dyed dense point cloud.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] First, the present invention adopts a multi-eye pure vision solution, which has a stronger anti-interference ability and can avoid the limitations of mutual interference between lidars and active infrared ranging solutions that cannot be applied to strong light environments such as outdoor environments, thereby improving the robustness of the constructed system.

[0062] Second, the present invention constructs a dense point cloud generation network, and the multi-eye super-resolution subnetwork of the network contains a super-resolution structure, so that the network model can be effectively supervised by high-resolution input. This improves the resolution of the generated point cloud, achieves performance that exceeds the upper limit of the input data device, and improves the visual sensory experience of people and the quality of input data for subsequent tasks.

[0063] Third, the present invention adds a reprojection loss to the total loss function, which can supervise the depth estimation value using multi-view images, making it possible to use a multi-view pure vision solution.

[0064] Fourth, the present invention adopts a multi-eye pure vision solution, the data acquisition equipment of this solution is the most common and cheap, so it can effectively reduce the equipment cost and data acquisition cost, so that the present invention reduces the equipment requirements for system construction.

[0065] Other aspects and features of the present invention will become apparent from the following detailed description with reference to the accompanying drawings. It should be understood, however, that the drawings are designed for illustrative purposes only and are not intended to limit the scope of the present invention. It should also be understood that, unless otherwise indicated, the drawings are not necessarily drawn to scale and are intended only to conceptually illustrate the structures and processes described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a flow chart of a method for generating a dense point cloud based on multi-view infrared provided in an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of a dense point cloud generation network constructed according to an embodiment of the present invention;

[0068] Figure 3a-3fA set of simulation images of a dense point cloud generation method based on multi-view infrared provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0070] Embodiment 1

[0071] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of a flow chart of a method for generating a dense point cloud based on multi-view infrared provided by an embodiment of the present invention, Figure 2 A schematic diagram of a dense point cloud generation network constructed according to an embodiment of the present invention is provided. The present invention provides a dense point cloud generation method based on multi-view infrared, and the dense point cloud generation method includes:

[0072] Step 1: Obtain the intrinsic and extrinsic parameters of the multi-eye low-resolution infrared camera and the high-resolution infrared camera.

[0073] Specifically, it is necessary to obtain the intrinsic parameters and extrinsic parameters of multiple low-resolution infrared cameras and the intrinsic parameters and extrinsic parameters of a high-resolution infrared camera. The intrinsic parameters are the properties of the camera itself, including the intrinsic parameter matrix K, rotation matrix R and translation matrix T. The extrinsic parameters are the posture relationship between one infrared camera and another infrared camera, that is, the projection matrix.

[0074] In a specific embodiment, step 1 may specifically include:

[0075] Step 1.1: Use multiple low-resolution infrared cameras and high-resolution infrared cameras to simultaneously shoot infrared calibration targets to obtain several sets of calibration images.

[0076] Specifically, the multi-eye low-resolution infrared camera and the high-resolution infrared camera are fixed, and the field of view of the high-resolution infrared camera is ensured to be within the field of view of the low-resolution infrared camera. All infrared cameras are used to simultaneously shoot the infrared calibration target to obtain a set of calibration images. Afterwards, the relative position between the multi-eye low-resolution infrared camera and the high-resolution infrared camera can be changed, and the position of the infrared calibration target also needs to be changed, so that repeated shooting can be performed to obtain multiple sets of calibration images.

[0077] Step 1.2: Use the calibration image to calibrate the camera to obtain the intrinsic and extrinsic parameters of all cameras.

[0078] Specifically, the calibration image is calibrated using Zhang Zhengyou calibration method to obtain the intrinsic and extrinsic parameters of all low-resolution infrared cameras and high-resolution infrared cameras.

[0079] Step 2: Use a multi-eye low-resolution infrared camera and a high-resolution infrared camera to obtain a multi-eye first infrared image and a first high-resolution infrared image.

[0080] Specifically, all infrared cameras are used to simultaneously capture the same scene, for example, a natural scene, so that the multi-eye low-resolution infrared cameras capture the multi-eye original first low-resolution infrared images, and the high-resolution infrared cameras capture the first high-resolution infrared image.

[0081] Step 3, construct a data set, the data set includes the internal and external parameters of the multi-eye low-resolution infrared camera and the high-resolution infrared camera, the multi-eye second low-resolution infrared image and the second high-resolution infrared image, the second low-resolution infrared image is obtained by intercepting the first low-resolution infrared image, and the second high-resolution infrared image is obtained by intercepting the first high-resolution infrared image.

[0082] Preferably, the common viewing area of ​​all first low-resolution infrared images and first high-resolution infrared images is intercepted to obtain multi-eye second low-resolution infrared images and second high-resolution infrared images of the same size.

[0083] In this embodiment, the multi-eye first low-resolution infrared image I lri (i=1,2...) and the first high-resolution infrared image I hr The relative position is fixed, and the first low-resolution infrared image I is known. lr The internal parameter matrix K lr , the first high-resolution infrared image I hr The internal parameter matrix K hr , the first low-resolution infrared image I lr To the first high-resolution infrared image I hr The projection matrix T between lh In the case of a known point in space in the first high-resolution infrared image I lr The image coordinates (u l ,v l ) and depth d l , the first high-resolution infrared image I can be calculated by projection hr The image coordinates (u h ,v h ) and depth d h ,Right now:

[0084]

[0085] The internal parameter matrix f x 、f y Represents the focal length, u0 and v0 represent the offset of the center point; projection matrix Tlh represents the projection matrix from the low-resolution infrared camera to the high-resolution infrared camera, T ll The projection matrix between low-resolution infrared cameras, R and T represent the rotation matrix and translation matrix (translation matrix is ​​also called translation vector) between low-resolution infrared cameras and high-resolution infrared cameras or between low-resolution infrared cameras, respectively.

[0086] Among them, the constructed data set can be divided into a training sample set and a test sample set according to proportion.

[0087] Step 4: Input the data set into the dense point cloud generation network to be trained until the loss function converges to obtain a trained dense point cloud generation network. The dense point cloud generation network includes a depth estimation subnetwork, a multi-eye infrared image registration module, and a multi-eye super-resolution subnetwork.

[0088] In this embodiment, the structure of the depth estimation subnetwork includes:

[0089] Input layer, first convolution layer, first residual layer, second convolution layer, second residual layer, third convolution layer, third residual layer, fourth convolution layer, fourth residual layer, fifth convolution layer, fifth residual layer, sixth convolution layer, sixth residual layer, seventh convolution layer, seventh residual layer, eighth convolution layer, eighth residual layer, ninth convolution layer, disparity cost aggregation layer, first 3D convolution layer, first 3D residual layer, second 3D convolution layer, third 3D convolution layer, second 3D residual layer, fourth 3D convolution layer, fifth 3D convolution layer, third 3D residual layer, sixth 3D convolution layer, seventh 3D convolution layer, fourth 3D residual layer, eighth 3D convolution layer, ninth 3D convolution layer, fifth 3D residual layer, first 3D deconvolution layer, second 3D deconvolution layer, third 3D deconvolution layer, fourth 3D deconvolution layer, fifth 3D deconvolution layer, disparity regression layer;

[0090] The input of the Mth convolutional layer is the sum of the Nth convolutional layer and the Nth residual layer, where N is one of one to eight, and M is N plus one; the inputs of the second 3D convolutional layer, the fourth 3D convolutional layer, the sixth 3D convolutional layer, and the eighth 3D convolutional layer are respectively the output of the disparity cost aggregation layer, the output of the second 3D convolutional layer, the output of the fourth 3D convolutional layer, and the output of the sixth 3D convolutional layer; the inputs of the second 3D deconvolutional layer, the third 3D deconvolutional layer, the fourth 3D deconvolutional layer, and the fifth 3D deconvolutional layer are respectively the output of the first 3D deconvolutional layer and the output of the fourth 3D residual layer, the output of the second 3D deconvolutional layer and the output of the third 3D residual layer, the output of the third 3D deconvolutional layer and the output of the second 3D residual layer, and the output of the fourth 3D deconvolutional layer and the output of the first 3D residual layer.

[0091] Furthermore, the parameters of each layer of the depth estimation subnetwork are set as follows:

[0092] The convolution kernel size of the input layer is set to 5×5, the stride is set to 2, and the output feature map channel size is set to 32;

[0093] The convolution kernel size of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer, the seventh convolution layer, the eighth convolution layer, and the ninth convolution layer are all set to 3×3, the step size is set to 1, and the output feature map channel size is set to 32;

[0094] The convolution kernel size of the first residual layer, the second residual layer, the third residual layer, the fourth residual layer, the fifth residual layer, the sixth residual layer, the seventh residual layer, and the eighth residual layer are all set to 3×3, the step size is all set to 1, and the output feature map channel size is all set to 32;

[0095] The convolution kernel size of the first 3D convolutional layer is set to 3×3×3, the stride is set to 1, and the output feature map channel size is set to 32;

[0096] The convolution kernel size of the second 3D convolution layer, the fourth 3D convolution layer, and the sixth 3D convolution layer are all set to 3×3×3, the step size is set to 2, and the output feature map channel size is set to 64;

[0097] The convolution kernel size of the third 3D convolution layer, the fifth 3D convolution layer, and the seventh 3D convolution layer are all set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 64;

[0098] The convolution kernel size of the eighth and ninth 3D convolutional layers is set to 3×3×3, the step size is set to 2, and the output feature map channel size is set to 128;

[0099] The convolution kernel size of the first 3D residual layer is set to 3×3×3, the stride is set to 1, and the output feature map channel size is set to 32;

[0100] The convolution kernel size of the second 3D residual layer, the third 3D residual layer, and the fourth 3D residual layer are all set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 64;

[0101] The convolution kernel size of the fifth 3D residual layer is set to 3×3×3, the stride is set to 1, and the output feature map channel size is set to 128;

[0102] The convolution kernel size of the first 3D deconvolution layer, the second 3D deconvolution layer, and the third 3D deconvolution layer is set to 3×3×3, the step size is set to 2, and the output feature map channel size is set to 64;

[0103] The convolution kernel size of the fourth 3D deconvolution layer is set to 3×3×3, the stride is set to 2, and the output feature map channel size is set to 32;

[0104] The convolution kernel size of the fifth 3D deconvolution layer is set to 3×3×3, the stride is set to 1, and the output feature map channel size is set to 1;

[0105] The parallax cost aggregation layer includes calculating the projection parallax slope direction and the parallax feature map splicing; calculating the coordinates (u) in the A-eye image according to the internal and external parameters of the A-eye camera and the B-eye camera (both the A-eye camera and the B-eye camera are low-resolution infrared cameras); A ,v A ) The coordinate line equation projected to the B-eye image at different depths is expressed in the form of a parametric equation:

[0106]

[0107] Among them, the variable d -1 Indicates the corresponding coordinates of the A-eye image (u A ,v A ), θ and δ represent the angle and offset of the corresponding projection in the B-eye image at different depths, δu represents the offset of the corresponding projection in the B-eye image in the u direction, and δv represents the offset of the corresponding projection in the B-eye image in the v direction.

[0108] The input of the disparity cost aggregation layer is the A-eye feature map F of size C×H×W. A and the B-mesh feature map F of size C×H×W B , the output is a spliced ​​feature map of size 2C×M×H×W after splicing; where M represents the maximum disparity preset in advance, and the spliced ​​feature map is spliced ​​along the disparity channel according to the number of disparity pixels; where the spliced ​​feature map F corresponds to:

[0109] F[1:C,disp,:,disp:W]=F A [:,:,:,disp:W]

[0110] F[C+1:2C,disp,:,disp:W]=F B [C+1:2C,:,:,1:W-disp]

[0111] disp represents disparity, and the rest of the spliced ​​feature map has a value of 0.

[0112] The disparity regression layer estimates the disparity value by using a soft argmin operation on the disparity dimension. The calculation formula for disparity regression is:

[0113]

[0114] Where d represents the disparity, ranging from 0 to the maximum disparity D max ,σ represents the sigmoid function, c d The channel dimension of the feature map representing the input of the disparity regression layer, where:

[0115]

[0116] Among them, t represents the input element.

[0117] In addition, this embodiment also provides a training process of a depth estimation subnetwork, and the training process is:

[0118] S1. Inputting the third low-resolution infrared image and the fourth low-resolution infrared image into a depth estimation subnetwork to obtain a disparity map of the third low-resolution infrared image;

[0119] S2. Obtaining a projection image of the fourth low-resolution infrared image onto the third low-resolution infrared image based on the fourth low-resolution infrared image, a depth map corresponding to the disparity map of the third low-resolution infrared image, and internal and external parameters;

[0120] S3. Optimize the loss function of the depth estimation subnetwork by using the gradient descent method, iteratively update the parameters of the depth estimation subnetwork until the loss function of the depth estimation subnetwork converges, and obtain a trained depth estimation subnetwork.

[0121] Specifically, the third low-resolution infrared image and the fourth low-resolution infrared image are images taken by relatively low-resolution infrared cameras. The third low-resolution infrared image is, for example, a left-eye image, and the fourth low-resolution infrared image is a right-eye image. Thus, the third low-resolution infrared image and the fourth low-resolution infrared image are input into the depth estimation subnetwork, and the output of the depth estimation subnetwork is the disparity map of the third low-resolution infrared image. According to the disparity map of the third low-resolution infrared image, the depth map of the third low-resolution infrared image can be obtained. According to the binocular stereo matching principle: depth depth = focal length f*baseline distance b / disparity disp, focal length f is equal to the internal parameter fx, and the baseline distance is equal to tx in the translation matrix T, wherein T has three parameters tx, ty, and tz, which respectively represent the offsets in the three directions of x, y, and z. It should be known to those skilled in the art that, under the conditions of the fourth low-resolution infrared image, the depth map of the third low-resolution infrared image, and the internal and external parameters of the infrared camera, the projection map of the fourth low-resolution infrared image to the third low-resolution infrared image can be obtained. Therefore, by optimizing the loss function of the depth estimation subnetwork through the gradient descent method and iteratively updating the parameters of the depth estimation subnetwork until the loss function converges, a trained depth estimation subnetwork can be obtained. At this time, the projection image of the fourth low-resolution infrared image to the third low-resolution infrared image is closest to the third low-resolution infrared image.

[0122] In this embodiment, the multi-eye infrared image registration module is used to register the low-resolution infrared image by projection method according to the projection matrix between the low-resolution infrared camera and the high-resolution infrared camera and the depth of the low-resolution infrared image, so as to obtain the low-resolution infrared image after projection registration.

[0123] In this embodiment, the structure of the multi-eye super-resolution sub-network includes:

[0124] feature extraction layer, tenth convolution layer, ninth residual layer, eleventh convolution layer, tenth residual layer, twelfth convolution layer, eleventh residual layer, thirteenth convolution layer, twelfth residual layer, fourteenth convolution layer, thirteenth residual layer, fifteenth convolution layer, fourteenth residual layer, sixteenth convolution layer, fifteenth residual layer, seventeenth convolution layer, sixteenth residual layer, eighteenth convolution layer, seventeenth residual layer, nineteenth convolution layer, eighteenth residual layer, twentieth convolution layer, nineteenth residual layer, twenty-first convolution layer, twentieth residual layer, twenty-second convolution layer, twenty-first residual layer, twenty-third convolution layer, twenty-second residual layer, twenty-fourth convolution layer, twenty-third residual layer, twenty-fifth convolution layer, twenty-fourth residual layer, pixel recombination layer, output layer;

[0125] The feature extraction layer shares weights, and its input is a two-channel data composed of a low-resolution infrared image after projection registration and its corresponding depth map. The input of the tenth convolutional layer is a spliced ​​feature map of the multi-channel output feature maps of the feature extraction layer with shared weights of multiple low-resolution images in the channel direction. The input of the remaining convolutional layers is the output of the previous convolutional layer plus the output of the previous residual layer. The input of the pixel reconstruction layer is the output of the twenty-fourth residual layer plus the output of the feature extraction layer.

[0126] Furthermore, set the parameters of each layer of the multi-eye super-resolution sub-network:

[0127] The convolution kernel size of the feature extraction layer is set to 3×3, the step size is set to 1, and the output feature map channel size is set to 64. Since the feature extraction layer shares weights, the actual number of output channels is k×64, where k is the number of eyes of the multi-eye low-resolution infrared camera. In this embodiment, k is four eyes as an example, so the actual number of output channels is 256;

[0128] The convolution kernel size of the tenth convolution layer, the eleventh convolution layer, the twelfth convolution layer, the thirteenth convolution layer, the fourteenth convolution layer, the fifteenth convolution layer, the sixteenth convolution layer, the seventeenth convolution layer, the eighteenth convolution layer, the nineteenth convolution layer, the twentieth convolution layer, the twenty-first convolution layer, the twenty-second convolution layer, the twenty-third convolution layer, the twenty-fourth convolution layer, and the twenty-fifth convolution layer are all set to 3×3, the step size is set to 1, and the output feature map channel size is set to 64;

[0129] The convolution kernel sizes of the ninth residual layer, the tenth residual layer, the eleventh residual layer, the twelfth residual layer, the thirteenth residual layer, the fourteenth residual layer, the fifteenth residual layer, the sixteenth residual layer, the seventeenth residual layer, the eighteenth residual layer, the nineteenth residual layer, the twentieth residual layer, the twenty-first residual layer, the twenty-second residual layer, and the twenty-third residual layer are all set to 3×3, the strides are all set to 1, and the output feature map channel sizes are all set to 64;

[0130] The convolution kernel size of the 24th residual layer is set to 3×3, the stride is set to 1, and the output feature map channel size is set to k×64, i.e. 256;

[0131] The magnification factor of the pixel reorganization layer is set to s, and the output feature map channel size is set to 64, where s is the multiple of super-resolution image reconstruction, s = 2 n , n is an integer greater than or equal to 1, and this embodiment takes n as 4 as an example;

[0132] The convolution kernel size of the output layer is set to 3×3, the stride is set to 1, and the output feature map channel size is set to 2.

[0133] The multi-eye super-resolution sub-network input is a multi-eye low-resolution projection image; the multi-eye super-resolution sub-network outputs a high-resolution image.

[0134] In a specific embodiment, step 4 may specifically include:

[0135] Step 4.1: Input the multi-view second low-resolution infrared image in the data set into the depth estimation subnetwork to obtain the disparity map of the multi-view second low-resolution infrared image, and convert the disparity map of the multi-view second low-resolution infrared image into a depth map.

[0136] Step 4.2: Based on the projection matrix between the low-resolution infrared camera and the high-resolution infrared camera and the depth map of the multi-view second low-resolution infrared image, the multi-view second low-resolution infrared image is registered by the projection method in the multi-view infrared image registration module to obtain the second low-resolution infrared image after multi-view registration.

[0137] Step 4.3: Input the two-channel data formed by stitching the second low-resolution infrared image after each eye registration and the depth map corresponding to the second low-resolution infrared image into the multi-eye super-resolution sub-network to obtain a high-resolution infrared image.

[0138] Step 4.4: Optimize the loss function by using the gradient descent method, iteratively update the parameters of the dense point cloud generation network until the loss function converges, and obtain a trained dense point cloud generation network.

[0139] In this embodiment, the loss function of the dense point cloud generation network includes super-resolution loss and reprojection loss.

[0140] Specifically, the super-resolution loss is calculated as follows: multiple second low-resolution infrared images in the dataset are input into the dense point cloud generation network to obtain a high-resolution infrared image, and the loss between the generated high-resolution infrared image and the high-resolution infrared image in the dataset is calculated. This loss is the super-resolution loss.

[0141] According to the pose relationship between the multi-view low-resolution infrared cameras and the high-resolution infrared cameras, the high-resolution infrared image is reprojected to the position of other cropped low-resolution infrared images, and the loss between the projected high-resolution infrared image and the cropped low-resolution infrared image is calculated.

[0142] Specifically, the reprojection loss is calculated as follows: input the multi-view low-resolution infrared images in the dataset into the dense point cloud generation network to obtain a high-resolution infrared image; reproject the high-resolution infrared image to the position of other cropped low-resolution infrared images according to the posture relationship between the multi-view infrared cameras; and calculate the loss between the projected high-resolution infrared image and the low-resolution infrared image in the dataset.

[0143] In this embodiment, multi-view low-resolution infrared images are input into a dense point cloud generation network, the super-resolution loss and the reprojection loss are added to obtain the loss function of the dense point cloud generation network, the loss function is optimized by the gradient descent method, and the network parameters of the dense point cloud generation network are iteratively updated until the loss function converges to obtain a trained dense point cloud generation network model.

[0144] The loss function of the dense point cloud generation network is as follows:

[0145] L=λ1L SR +λ2L REP

[0146]

[0147] L REP =∑||I rep -I lr ||1

[0148] Among them, L represents the loss function of the dense point cloud generation network, L SR represents the super-resolution loss, L REP represents the reprojection loss, represents the mean square error operation, ||·||1 represents the one-norm operation, I sr Indicates that the first channel in the output of the trained dense point cloud generation network generates a super-resolution infrared image, I rep Represents the image where the low-resolution infrared image in the dataset is reprojected to the location of other low-resolution infrared images, I hr Represents the reprojected infrared image where the high-resolution infrared image in the dataset is projected to the position of multiple low-resolution infrared cameras, I lr represents the cropped low-resolution infrared image in the dataset, λ1 and λ2 represent the weights of the super-resolution loss and the reprojection loss.

[0149] Step 5: Input the multi-eye low-resolution infrared image to be generated into the trained dense point cloud generation network to obtain the disparity map and high-resolution infrared image of the multi-eye low-resolution infrared image, so as to generate a dense point cloud according to the disparity map and high-resolution infrared image of the multi-eye low-resolution infrared image.

[0150] In a specific embodiment, step 5 may specifically include:

[0151] Step 5.1: Obtain a multi-view low-resolution depth map based on the disparity maps of the multi-view low-resolution infrared images.

[0152] Specifically, the multi-view low-resolution infrared images to be generated are input into the trained dense point cloud generation network, which can output the disparity map of the multi-view low-resolution infrared images and the high-resolution infrared images accordingly, thereby converting the disparity map of the low-resolution infrared image into a depth map, that is, into a low-resolution depth map.

[0153] Step 5.2: Perform interpolation and upsampling processing on the multi-eye low-resolution depth maps respectively to obtain the corresponding multi-eye high-resolution depth maps.

[0154] Step 5.3, calculate the average value of the multi-eye high-resolution depth map, and use the average value as the final high-resolution depth map;

[0155] Step 5.4: The three-dimensional coordinates of the dense point cloud are obtained according to the inverse matrix of the internal reference of the high-resolution infrared image, the final high-resolution depth map and the pixel coordinates of the high-resolution infrared image. The calculation model is:

[0156]

[0157] The internal parameter of the high-resolution infrared image (i.e., super-resolution infrared image) is equal to the internal parameter of the original low-resolution infrared image multiplied by the high-resolution multiple, (x w ,y w 、z w ) represents the three-dimensional coordinates of the dense point cloud, represents the inverse intrinsic matrix of the high-resolution infrared image, d sr represents the average value of multi-eye high-resolution depth maps, i.e., the final high-resolution depth map, (u sr 、v sr , 1) and v sr Represents the pixel coordinates of a high-resolution infrared image.

[0158] Step 5.5: The pixel values ​​of the high-resolution infrared image are assigned to the dense point cloud according to the pixel coordinates to generate a dyed dense point cloud.

[0159] Specifically, a high-resolution dense point cloud can be generated by assigning pixel values ​​of original pixels of a high-resolution infrared image to a point cloud color.

[0160] The effect of the present invention can be further described through the following simulation experiments.

[0161] 1. Experimental conditions:

[0162] The hardware test platform of the simulation experiment of the present invention is: CPU: i7-9700K 3.60GHz, 32G memory, GPU: TITAN Xp;

[0163] The software simulation platform of the present invention is: Windows 10 64-bit operating system, Pycharm development platform;

[0164] The software simulation language and deep learning framework used in the present invention are: Python and Pytorch.

[0165] The data used in the simulation experiment of the present invention is a self-made multi-view infrared image data set, which is made by using three low-resolution infrared cameras and one high-resolution infrared camera arranged in a straight line to shoot. Each set of training data in the data set contains three low-resolution infrared images with a pixel size of 288×384 and one high-resolution infrared image with a pixel size of 1024×1281, and the image format is png.

[0166] 2. Experimental content and results analysis

[0167] The experiment of the present invention adopts the dense point cloud generation method proposed in the present invention to process the input three-eye low-resolution infrared image to generate a dense point cloud.

[0168] Combine the following Figure 3a to Figure 3f The simulation diagram of the present invention is further described.

[0169] Figure 3a , Figure 3c , Figure 3e There are three sets of input three-eye low-resolution infrared images respectively.

[0170] Figure 3b , Figure 3d , Figure 3f The dense point clouds generated for the three groups of inputs are images from different perspectives.

[0171] pass Figure 3a to Figure 3f It can be seen that the dense point cloud generation method proposed in the present invention can generate high-resolution dense point clouds through multi-eye low-resolution infrared images, filling the gap in infrared point cloud generation of this type of method.

[0172] First, the present invention adopts a multi-eye pure vision solution, the data acquisition equipment of which is the most common and cheap, so it can effectively reduce the equipment cost and data acquisition cost, so that the present invention reduces the equipment requirements for system construction.

[0173] Second, the present invention adopts a multi-eye pure vision solution, which has a stronger anti-interference ability and can avoid the limitations of mutual interference between lidars and active infrared ranging solutions that cannot be applied to strong light environments such as outdoor environments. This improves the robustness of the constructed system.

[0174] Third, the present invention constructs a dense point cloud generation network, and the multi-eye super-resolution subnetwork of the network contains a super-resolution structure, so that the network model can be effectively supervised by high-resolution input. This improves the resolution of the generated point cloud, achieves performance that exceeds the upper limit of the input data device, and improves the visual sensory experience of people and the quality of input data for subsequent tasks.

[0175] Fourth, the present invention adds a reprojection loss to the total loss function, which can supervise the depth estimation value using multi-view images, making it possible to use a multi-view pure vision solution.

[0176] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0177] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristic data points described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristic data points described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0178] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A dense point cloud generation method based on multi-view infrared, characterized in that: The dense point cloud generation method comprises: Obtain the internal and external parameters of multi-eye low-resolution infrared cameras and high-resolution infrared cameras; Using a multi-eye low-resolution infrared camera and a high-resolution infrared camera to obtain a multi-eye first low-resolution infrared image and a first high-resolution infrared image; Constructing a data set, the data set including internal and external parameters of the multi-eye low-resolution infrared camera and the high-resolution infrared camera, a multi-eye second low-resolution infrared image and a second high-resolution infrared image, wherein the second low-resolution infrared image is obtained by intercepting the first low-resolution infrared image, and the second high-resolution infrared image is obtained by intercepting the first high-resolution infrared image; Inputting the data set into a dense point cloud generation network to be trained until the loss function converges, thereby obtaining a trained dense point cloud generation network, wherein the dense point cloud generation network includes a depth estimation subnetwork, a multi-eye infrared image registration module, and a multi-eye super-resolution subnetwork; Inputting the multi-eye low-resolution infrared image to be generated into the trained dense point cloud generation network to obtain the disparity map and high-resolution infrared image of the multi-eye low-resolution infrared image, so as to generate a dense point cloud according to the disparity map and high-resolution infrared image of the multi-eye low-resolution infrared image; The loss function is: L=λ1L SR +λ2L REP L REP =∑||I rep -I lr ||1 Among them, L represents the loss function of the dense point cloud generation network, L SR represents the super-resolution loss, L REP represents the reprojection loss, represents the mean square error operation, ||·||1 represents the one-norm operation, I sr represents the super-resolution infrared image, I hr Represents the reprojected infrared image of the high-resolution infrared image in the dataset projected to the position of the multi-eye infrared camera, I rep Represents an image where a low-resolution infrared image is reprojected to the position of another low-resolution infrared image, I lr represents the low-resolution infrared image in the dataset, λ1 and λ2 represent the weights of super-resolution loss and reprojection loss.

2. The method for generating dense point cloud based on multi-view infrared according to claim 1, characterized in that: Obtain the internal and external parameters of multi-view low-resolution infrared cameras and high-resolution infrared cameras, including: Use multiple low-resolution infrared cameras and high-resolution infrared cameras to simultaneously shoot infrared calibration targets to obtain several groups of calibration images; The calibration images are used to calibrate to obtain the intrinsic and extrinsic parameters of all cameras.

3. The method for generating dense point cloud based on multi-view infrared according to claim 1, characterized in that: The data set is input into the dense point cloud generation network to be trained until the loss function converges, thereby obtaining a trained dense point cloud generation network, including: Inputting the data set into the depth estimation subnetwork to obtain a disparity map of the multi-eye second low-resolution infrared image, and converting the disparity map of the multi-eye second low-resolution infrared image into a depth map; Based on the projection matrix between the low-resolution infrared camera and the high-resolution infrared camera and the depth map of the multi-eye second low-resolution infrared image, the multi-eye second low-resolution infrared image is registered by a projection method in the multi-eye infrared image registration module to obtain a multi-eye registered second low-resolution infrared image; Inputting the 2-channel data obtained by splicing the second low-resolution infrared image after each eye registration and the depth map corresponding to the second low-resolution infrared image into the multi-eye super-resolution sub-network to obtain a high-resolution infrared image; The loss function is optimized by the gradient descent method, and the parameters of the dense point cloud generation network are iteratively updated until the loss function converges to obtain a trained dense point cloud generation network.

4. The method for generating dense point cloud based on multi-view infrared according to claim 1, characterized in that: The training process of the depth estimation subnetwork is: Inputting the third low-resolution infrared image and the fourth low-resolution infrared image into the depth estimation subnetwork to obtain a disparity map of the third low-resolution infrared image; Obtaining a projection image of the fourth low-resolution infrared image onto the third low-resolution infrared image based on the fourth low-resolution infrared image, a depth map corresponding to the disparity map of the third low-resolution infrared image, and internal and external parameters; The loss function of the depth estimation subnetwork is optimized by the gradient descent method, and the parameters of the depth estimation subnetwork are iteratively updated until the loss function of the depth estimation subnetwork converges, thereby obtaining a trained depth estimation subnetwork.

5. The method for generating dense point cloud based on multi-view infrared according to claim 1, characterized in that: The structure of the depth estimation subnetwork includes: Input layer, first convolution layer, first residual layer, second convolution layer, second residual layer, third convolution layer, third residual layer, fourth convolution layer, fourth residual layer, fifth convolution layer, fifth residual layer, sixth convolution layer, sixth residual layer, seventh convolution layer, seventh residual layer, eighth convolution layer, eighth residual layer, ninth convolution layer, disparity cost aggregation layer, first 3D convolution layer, first 3D residual layer, second 3D convolution layer, third 3D convolution layer, second 3D residual layer, fourth 3D convolution layer, fifth 3D convolution layer, third 3D residual layer, sixth 3D convolution layer, seventh 3D convolution layer, fourth 3D residual layer, eighth 3D convolution layer, ninth 3D convolution layer, fifth 3D residual layer, first 3D deconvolution layer, second 3D deconvolution layer, third 3D deconvolution layer, fourth 3D deconvolution layer, fifth 3D deconvolution layer, disparity regression layer; The input of the Mth convolutional layer is the sum of the Nth convolutional layer and the Nth residual layer, N is one of one to eight, and M is N plus one; the inputs of the second 3D convolutional layer, the fourth 3D convolutional layer, the sixth 3D convolutional layer, and the eighth 3D convolutional layer are respectively the output of the disparity cost aggregation layer, the output of the second 3D convolutional layer, the output of the fourth 3D convolutional layer, and the output of the sixth 3D convolutional layer; the inputs of the second 3D deconvolutional layer, the third 3D deconvolutional layer, the fourth 3D deconvolutional layer, and the fifth 3D deconvolutional layer are respectively the output of the first 3D deconvolutional layer and the output of the fourth 3D residual layer, the output of the second 3D deconvolutional layer and the output of the third 3D residual layer, the output of the third 3D deconvolutional layer and the output of the second 3D residual layer, and the output of the fourth 3D deconvolutional layer and the output of the first 3D residual layer.

6. The method for generating dense point cloud based on multi-view infrared according to claim 5, characterized in that: The convolution kernel size of the input layer is set to 5×5, the step size is set to 2, and the output feature map channel size is set to 32; The convolution kernel sizes of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer, the sixth convolution layer, the seventh convolution layer, the eighth convolution layer, and the ninth convolution layer are all set to 3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 32; The convolution kernel sizes of the first residual layer, the second residual layer, the third residual layer, the fourth residual layer, the fifth residual layer, the sixth residual layer, the seventh residual layer, and the eighth residual layer are all set to 3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 32; The convolution kernel size of the first 3D convolutional layer is set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 32; The convolution kernel sizes of the second 3D convolution layer, the fourth 3D convolution layer, and the sixth 3D convolution layer are all set to 3×3×3, the step sizes are all set to 2, and the output feature map channel sizes are all set to 64; The convolution kernel sizes of the third 3D convolution layer, the fifth 3D convolution layer, and the seventh 3D convolution layer are all set to 3×3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 64; The convolution kernel sizes of the eighth 3D convolution layer and the ninth 3D convolution layer are both set to 3×3×3, the step sizes are both set to 2, and the output feature map channel sizes are both set to 128; The convolution kernel size of the first 3D residual layer is set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 32; The convolution kernel sizes of the second 3D residual layer, the third 3D residual layer, and the fourth 3D residual layer are all set to 3×3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 64; The convolution kernel size of the fifth 3D residual layer is set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 128; The convolution kernel size of the first 3D deconvolution layer, the second 3D deconvolution layer, and the third 3D deconvolution layer is set to 3×3×3, the step size is set to 2, and the output feature map channel size is set to 64; The convolution kernel size of the fourth 3D deconvolution layer is set to 3×3×3, the step size is set to 2, and the output feature map channel size is set to 32; The convolution kernel size of the fifth 3D deconvolution layer is set to 3×3×3, the step size is set to 1, and the output feature map channel size is set to 1.

7. The method for generating dense point cloud based on multi-view infrared according to claim 1, characterized in that: The structure of the multi-eye super-resolution sub-network includes: feature extraction layer, tenth convolution layer, ninth residual layer, eleventh convolution layer, tenth residual layer, twelfth convolution layer, eleventh residual layer, thirteenth convolution layer, twelfth residual layer, fourteenth convolution layer, thirteenth residual layer, fifteenth convolution layer, fourteenth residual layer, sixteenth convolution layer, fifteenth residual layer, seventeenth convolution layer, sixteenth residual layer, eighteenth convolution layer, seventeenth residual layer, nineteenth convolution layer, eighteenth residual layer, twentieth convolution layer, nineteenth residual layer, twenty-first convolution layer, twentieth residual layer, twenty-second convolution layer, twenty-first residual layer, twenty-third convolution layer, twenty-second residual layer, twenty-fourth convolution layer, twenty-third residual layer, twenty-fifth convolution layer, twenty-fourth residual layer, pixel recombination layer, output layer; The feature extraction layer shares weights, and its input is two-channel data spliced ​​from a low-resolution infrared image after projection alignment and its corresponding depth map; the tenth convolutional layer input is a spliced ​​feature map of multiple low-resolution images in the channel direction through multi-channel output feature maps of the feature extraction layer with shared weights; the input of the remaining convolutional layers is the output of the previous convolutional layer plus the output of the previous residual layer; the pixel reconstruction layer input is the output of the twenty-fourth residual layer plus the output of the feature extraction layer.

8. The method for generating dense point cloud based on multi-view infrared according to claim 7, characterized in that: The convolution kernel size of the feature extraction layer is set to 3×3, the step size is set to 1, and the output feature map channel size is k×64, where k is the number of eyes of the multi-eye low-resolution infrared camera; The convolution kernel sizes of the tenth convolution layer, the eleventh convolution layer, the twelfth convolution layer, the thirteenth convolution layer, the fourteenth convolution layer, the fifteenth convolution layer, the sixteenth convolution layer, the seventeenth convolution layer, the eighteenth convolution layer, the nineteenth convolution layer, the twentieth convolution layer, the twenty-first convolution layer, the twenty-second convolution layer, the twenty-third convolution layer, the twenty-fourth convolution layer, and the twenty-fifth convolution layer are all set to 3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 64; The convolution kernel sizes of the ninth residual layer, the tenth residual layer, the eleventh residual layer, the twelfth residual layer, the thirteenth residual layer, the fourteenth residual layer, the fifteenth residual layer, the sixteenth residual layer, the seventeenth residual layer, the eighteenth residual layer, the nineteenth residual layer, the twentieth residual layer, the twenty-first residual layer, the twenty-second residual layer, and the twenty-third residual layer are all set to 3×3, the step sizes are all set to 1, and the output feature map channel sizes are all set to 64; The convolution kernel size of the 24th residual layer is set to 3×3, the step size is set to 1, and the output feature map channel size is set to k×64; The magnification factor of the pixel reorganization layer is set to s, and the output feature map channel size is set to 64, where s is the multiple of super-resolution image reconstruction, s=2 n , n is an integer greater than or equal to 1; The convolution kernel size of the output layer is set to 3×3, the step size is set to 1, and the output feature map channel size is set to 2.

9. The method for generating dense point cloud based on multi-view infrared according to claim 1, characterized in that: Generate a dense point cloud based on the disparity map of multi-view low-resolution infrared images and high-resolution infrared images, including: A multi-eye low-resolution depth map is obtained according to the disparity map of the multi-eye low-resolution infrared image; The low-resolution depth maps of the multiple eyes are interpolated and up-sampled to obtain high-resolution depth maps of the multiple eyes; Calculate the average value of multiple high-resolution depth maps and use the average value as the final high-resolution depth map; The three-dimensional coordinates of the dense point cloud are obtained according to the inverse matrix of the internal reference of the high-resolution infrared image, the final high-resolution depth map and the pixel coordinates of the high-resolution infrared image; The pixel values ​​of the high-resolution infrared image are assigned to the dense point cloud according to the pixel coordinates to generate a dyed dense point cloud.

Images