A neural network-based super-resolution reconstruction method for 3D range images of lidar

Abstract

This invention relates to a neural network-based super-resolution reconstruction method for 3D range images of LiDAR. The invention pertains to the field of 3D range image super-resolution reconstruction technology. Based on a set of high-resolution intensity images Z and low-resolution range images X, any pixel s on the low-resolution range image X is represented by an ideal high-resolution range image Y. A nonlinear mapping relationship is determined between the high-resolution intensity image Z and the ideal high-resolution range image Y, and a loss function is established. The pixel mapping range of the high-resolution intensity image Z is increased, and image pixel coordinate encoding and region pixel encoding are introduced for super-resolution reconstruction. This invention outperforms bicubic interpolation, guided filtering, TGV, standard image guided algorithms, and region similarity guided algorithms. For low-resolution Gm-APD LiDAR 3D range images, the algorithm of this invention shows superior performance in reconstructing image edge sharpness and restoring information such as the edge contour of the low-resolution range image.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional range image super-resolution reconstruction technology, and is a method for super-resolution reconstruction of three-dimensional range images of LiDAR based on neural networks. Background Technology

[0002] In real-world imaging and detection, extending traditional two-dimensional imaging to three-dimensional imaging with distance information is of great significance. Three-dimensional imaging methods include stereo vision, stereo structured light, depth measurement cameras, scanning lidar, solid-state lidar, and flash lidar. Among these, lidar has unparalleled inherent advantages: small size, light weight, high data density, high ranging accuracy, fast imaging speed, strong penetration through vegetation, unaffected by solar altitude angle and shadows, good concealment, and strong anti-interference capabilities.

[0003] Gm-APD (Geiger-based APD array) lidar is a non-scanning laser 3D imaging radar. It is easy to integrate, has high detection sensitivity, and has single-photon detection capability. The laser source of Gm-APD emits laser pulses, and the detector measures the flight time of the pulses to calculate the distance value of each pixel and obtain a 3D distance image. Therefore, Gm-APD has a greater advantage in detecting weak signal targets.

[0004] Although Gm-APD lidar boasts high range resolution, its spatial resolution is limited by the size of the Gm-APD imaging unit array. The most direct way to improve image resolution is to upgrade the optical hardware of the acquisition system; however, due to the difficulty in significantly improving manufacturing processes and the extremely high manufacturing costs, physically solving the low-resolution image problem is often too costly. Therefore, it is necessary to address the issue from a software and algorithm perspective to improve image quality, using specific algorithms to reconstruct a corresponding high-resolution image from a given low-resolution Gm-APD lidar image.

[0005] For images defined in two-dimensional space, traditional methods treat them as two-dimensional Markov random fields, using Markov properties to describe the relationships between neighboring pixels in the image, thus simulating the digital image and transforming the super-resolution reconstruction problem into solving an optimization model of the Markov random field. However, real images are not strictly Markov random fields. Traditional methods employ mathematical approximations to establish the optimization equations and use mathematical approximations in the solution process, resulting in less than ideal reconstruction results. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for super-resolution reconstruction of LiDAR 3D range images based on neural networks, aiming to obtain high spatial resolution LiDAR 3D range images. The invention offers the following technical solutions:

[0007] A method for super-resolution reconstruction of 3D range images from a LiDAR system based on neural networks, the method comprising the following steps:

[0008] Step 1: Based on a set of high-resolution intensity images Z and low-resolution distance images X, represent any pixel s on the low-resolution distance image X with an ideal high-resolution distance image Y;

[0009] Step 2: Determine the nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution range image Y, and establish the loss function;

[0010] Step 3: Increase the pixel mapping range of the high-resolution intensity image Z, introduce image pixel coordinate encoding and region pixel encoding, and perform super-resolution reconstruction.

[0011] Preferably, step 1 specifically comprises:

[0012] For a set of corresponding high-resolution intensity images Z and low-resolution range images X, the ideal high-resolution range image is Y, and the ideal high-resolution range image Y has the same dimensions as the high-resolution intensity image Z. The m×m region on the low-resolution range image X corresponds to the (D×m)×(D×m) region mapped to the ideal high-resolution range image Y, where D is the ratio of the high-resolution range image Y to the low-resolution range image X. Any pixel s on the low-resolution range image X is represented by the following formula:

[0013]

[0014] Preferably, step 2 specifically comprises:

[0015] There is a nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution range image Y, which is expressed by the following equation:

[0016]

[0017] For a real image, any pixel s on the low-resolution range image X is represented by the following formula:

[0018]

[0019] Where q represents a pixel in the D×D region of the high-resolution intensity image Z, and Q represents the 5×5 neighborhood of q, where the neighborhood size is insufficient to be filled with zeros.

[0020] Under constraints, we expect to obtain a set of parameters θ that minimizes the loss function, expressed by the following formula:

[0021]

[0022] When using the L2 norm for regularization in a network, the loss function can be expressed as follows:

[0023]

[0024] Preferably, step 3 specifically comprises:

[0025] By increasing the pixel mapping range of image Z and introducing image pixel coordinate encoding, f1 is for a single pixel, so single-pixel encoding is performed using the pixel index; f2 is for region pixels, so pixels within the same region have the same index for region pixel encoding. The loss function can then be expressed as:

[0026]

[0027] When the original range image has low resolution, interpolation is required to compare the range values ​​with the processed high-resolution range image. This involves interpolating and upsampling the original range image to complete super-resolution sampling.

[0028] Preferably, since each pixel in the intensity image is not completely independent of its neighborhood, a 2-layer 3×3 convolution is used for each pixel block to extract neighborhood information.

[0029] Preferably, the high-resolution intensity image Z is taken as the ICCD intensity image, and the low-resolution distance image X is taken as the Gm-APD distance image.

[0030] Preferably, the high-resolution intensity image Z is taken as the Gm-APD intensity image after super-resolution processing, and the low-resolution distance image X is taken as the Gm-APD distance image.

[0031] A neural network-based super-resolution reconstruction system for 3D range images of lidar, the system comprising:

[0032] A pixel extraction module, which uses an ideal high-resolution distance image Y to represent any pixel s on the low-resolution distance image X based on a set of high-resolution intensity images Z and low-resolution distance images X;

[0033] The loss function module establishes a loss function based on the determination that there is a nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution distance image Y.

[0034] The super-resolution reconstruction module increases the pixel mapping range of the high-resolution intensity image Z, introduces image pixel coordinate encoding and region pixel encoding, and performs super-resolution reconstruction.

[0035] A computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a neural network-based super-resolution reconstruction method for three-dimensional range images of a lidar system.

[0036] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a neural network-based super-resolution reconstruction method for three-dimensional range images of a lidar system.

[0037] The present invention has the following beneficial effects:

[0038] This invention uses neural networks to acquire high spatial resolution three-dimensional range images of lidar, mainly targeting two application methods: the first is to use a high spatial resolution ICCD camera and a high range resolution Gm-APD lidar for composite imaging, and use the ICCD intensity map to guide the Gm-APD range map; the second is to use only high range resolution Gm-APD lidar imaging, and use the Gm-APD intensity map to guide its own range map.

[0039] The reconstructed image obtained by the method of this invention has higher clarity, richer detail, and sharper edges compared to other algorithms, and exhibits less texture duplication compared to TGV and standard image-guided algorithms. As shown in Table 1, at higher magnification levels, where edge contours and other information are significantly lost in low-resolution images, the algorithm of this invention achieves the best performance indicators. At lower magnification levels, the algorithm's performance indicators are worse. Considering all magnification levels, the algorithm of this invention achieves the best performance indicators, with a maximum improvement of 72% in RMSE and a maximum improvement of 7% in SSIM compared to other algorithms.

[0040] This invention outperforms bicubic interpolation, guided filtering, TGV, standard image guided algorithm, and region similarity guided algorithm. For low-resolution Gm-APD lidar 3D range images, which are severely lacking in edge contour information, this invention's algorithm outperforms low-resolution range images with more complete edge contour information in terms of reconstructing image edge sharpness and restoring edge contour information. Attached Figure Description

[0041] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the interpolation upsampling method for the original distance image;

[0043] Figure 2 This is a schematic diagram of the imaging system.

[0044] Figure 3 A graph showing the processing results of the Middleburg dataset;

[0045] Figure 4 This is a diagram showing the result of processing the lidar distance image.

[0046] Figure 5 This is a diagram showing the result of processing the lidar distance image. Detailed Implementation

[0047] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0049] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0050] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0051] The present invention will be described in detail below with reference to specific embodiments. Specific Implementation Example 1:

[0053] according to Figures 1 to 5As shown, the specific optimized technical solution adopted by the present invention to solve the above-mentioned technical problems is: The present invention relates to a super-resolution reconstruction method for three-dimensional range images of lidar based on neural networks.

[0054] This invention provides a method for super-resolution reconstruction of three-dimensional range images from a LiDAR system based on neural networks. The method includes the following steps:

[0055] Step 1: Based on a set of high-resolution intensity images Z and low-resolution distance images X, represent any pixel s on the low-resolution distance image X with an ideal high-resolution distance image Y;

[0056] Step 2: Determine the nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution range image Y, and establish the loss function;

[0057] Step 3: Increase the pixel mapping range of the high-resolution intensity image Z, introduce image pixel coordinate encoding and region pixel encoding, and perform super-resolution reconstruction.

[0058] This invention uses neural networks to acquire high spatial resolution three-dimensional range images of lidar, mainly targeting two application methods: the first is to use a high spatial resolution ICCD camera and a high range resolution Gm-APD lidar for composite imaging, and use the ICCD intensity map to guide the Gm-APD range map; the second is to use only high range resolution Gm-APD lidar imaging, and use the Gm-APD intensity map to guide its own range map. Specific Implementation Example 2:

[0060] The only difference between Embodiment 2 and Embodiment 1 of this application is that:

[0061] Step 1 specifically involves:

[0062] For a set of corresponding high-resolution intensity images Z and low-resolution range images X, the ideal high-resolution range image is Y, and the ideal high-resolution range image Y has the same dimensions as the high-resolution intensity image Z. The m×m region on the low-resolution range image X corresponds to the (D×m)×(D×m) region mapped to the ideal high-resolution range image Y, where D is the ratio of the high-resolution range image Y to the low-resolution range image X. Any pixel s on the low-resolution range image X is represented by the following formula:

[0063] Specific Implementation Example 3:

[0065] The only difference between Embodiment 3 and Embodiment 2 of this application is that:

[0066] Step 2 specifically involves:

[0067] There is a nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution range image Y, which is expressed by the following equation:

[0068]

[0069] For a real image, any pixel s on the low-resolution range image X is represented by the following formula:

[0070]

[0071] Where q represents a pixel in the D×D region of the high-resolution intensity image Z, and Q represents the 5×5 neighborhood of q, where the neighborhood size is insufficient to be filled with zeros.

[0072] Under constraints, we expect to obtain a set of parameters θ that minimizes the loss function, expressed by the following formula:

[0073]

[0074] When using the L2 norm for regularization in a network, the loss function can be expressed as follows:

[0075] Specific Implementation Example 4:

[0077] The only difference between Embodiment 4 and Embodiment 3 of this application is that:

[0078] Step 3 specifically involves:

[0079] By increasing the pixel mapping range of image Z and introducing image pixel coordinate encoding, f1 is for a single pixel, so single-pixel encoding is performed using the pixel index; f2 is for region pixels, so pixels within the same region have the same index for region pixel encoding. The loss function can then be expressed as:

[0080]

[0081] When the original range image has low resolution, interpolation is required to compare the range values ​​with the processed high-resolution range image. This involves upsampling the original range image through interpolation to complete super-resolution reconstruction.

[0082] Unlike intensity images, range images contain texture information of the target, so super-resolution reconstruction of intensity images focuses only on restoring the target's texture information. Range images, however, represent the target's distance information, therefore super-resolution reconstruction of range images must maintain consistency with the original distance values ​​as much as possible. Since the original range image has a low resolution, interpolation is required to compare its distance values ​​with the processed high-resolution range image. Therefore, a suitable interpolation method is needed to upsample the original range image.

[0083] The proposed original distance image interpolation upsampling method is as follows: Figure 1 As shown. Compared to the high-resolution range image, the low-resolution range image lacks distance values ​​for corresponding pixel regions. For distance values ​​not detected in the actual scene, pixel blocks should be used to fill in the missing values, meaning that the distance values ​​in the upsampled high-resolution range image regions should be the same. The PSNR index of the interpolated upsampled range image and the method of this invention can be used to evaluate the difference in distance values ​​between the range image obtained by the method of this invention and the original range image.

[0084] This invention addresses the issue that each pixel in an intensity image is not entirely independent of its neighborhood, and proposes to extract neighborhood information by using a 2-layer 3×3 convolution for each pixel block.

[0085] This invention addresses the problem of upsampling range images in Gm-APD by proposing that the range image cannot be interpolated using neighborhood calculation methods, that is, only nearest neighbor interpolation can be used, and linear interpolation or cubic spline interpolation cannot be used.

[0086] This invention addresses the issue that Gm-APD intensity images have too uniform grayscale compared to color cameras by proposing a method of segmenting Gm-APD images into blocks using superpixel segmentation.

[0087] The algorithm was quantitatively evaluated using the Middleburg dataset. To verify the super-resolution reconstruction effect, different factors (x2, x4, x8, x16) were used to downsample the distance image to simulate low-resolution distance images. The algorithm of this invention was compared with bicubic interpolation, guided filtering, TGV, standard image guided algorithms, and region similarity guided algorithms. Figure 3 The visual effect at x8 magnification.

[0088] Super-resolution tests were performed on real data acquired by the ICCD camera and Gm-APD lidar. The range image acquired by the Gm-APD lidar has a resolution of 64×64, while the intensity image acquired by the ICCD camera has a resolution of 480×640. Due to the different fields of view, an intensity image with a resolution of 328×366 was obtained after registration. Figure 4 The visual effects of different algorithms are shown. Figure 4 (c) is Figure 4 (a) shows the grayscale transformation display. Figure 4 (d) is Figure 4 The grayscale transformation of (j) shows that... Figure 4 (d) The building (roof) in the upper left corner has been restored.

[0089] Super-resolution tests were performed on real data acquired by the Gm-APD lidar. The intensity and range images acquired by the Gm-APD lidar both have a resolution of 64×64. Since the intensity and range images come from the same detection device, there is no need to register the intensity and range images. Figure 5 The processing effect of the method of the present invention on Gm-APD lidar images is shown.

[0090] The root mean square error (RMSE) and structural similarity (SSIM) are two reference image quality evaluation metrics to objectively evaluate the reconstruction results of different methods on the Middleburg dataset. The evaluation results are shown in Table 1.

[0091] Table 1 Comparison of reference image quality evaluation indicators for different methods

[0092]

[0093] from Figure 3 As can be seen, the reconstructed image by the method of this invention has higher clarity, richer detail, and sharper edges compared to other algorithms, and exhibits less texture duplication compared to TGV and standard image-guided algorithms. Table 1 shows that at higher magnification levels, where edge contours and other information are severely lacking in low-resolution images, the algorithm of this invention achieves the best performance indicators. At lower magnification levels, the algorithm of this invention performs worse. Considering all evaluation indicators across magnification levels, the algorithm of this invention achieves the best performance indicators, with a maximum improvement of 72% in RMSE and a maximum improvement of 7% in SSIM compared to other algorithms.

[0094] from Figure 4 It is evident that both the standard image guidance algorithm and the algorithm of this invention can recover the building in the upper left corner. Furthermore, the reconstructed image by the algorithm of this invention has sharper edges and maintains greater consistency with the distance values ​​of the original distance image, meaning that the target color and target edges are better preserved. Figure 5 As can be seen, the algorithm of this invention recovers more target details from the distance image, that is, the window shape is more complete. Furthermore, using... Figure 5 (b) and Figure 5 (d) The calculated MSE is 1.550 and PSNR is 46.228, which shows that the distance values ​​of the distance image obtained by the method of the present invention are very small compared with the original distance image.

[0095] Based on the experimental results of both simulation and real data, it can be seen that the algorithm of this invention outperforms bicubic interpolation, guided filtering, TGV, standard image guided algorithm, and region similarity guided algorithm. For low-resolution Gm-APD lidar 3D range images, due to the severe loss of edge contour information, the algorithm of this invention outperforms low-resolution range images with more complete edge contour information in terms of reconstructing image edge sharpness and restoring edge contour information. Specific Implementation Example 5:

[0097] The difference between Embodiment 5 and Embodiment 4 of this application lies only in:

[0098] Since each pixel in the intensity image is not completely independent of its neighborhood, a 2-layer 3×3 convolution is used for each pixel block to extract neighborhood information. Specific Implementation Example Six:

[0100] The only difference between Embodiment Six and Embodiment Five of this application is that:

[0101] The first application method is to use ICCD camera and Gm-APD lidar to create composite imaging. The ICCD intensity map guides the Gm-APD range map. The high-resolution intensity image Z is taken as the ICCD intensity image, and the low-resolution range image X is taken as the Gm-APD range image. Specific Implementation Example 7:

[0103] The only difference between Embodiment 7 and Embodiment 6 of this application is that:

[0104] The second application method involves using only Gm-APD lidar imaging. First, the resolution of the Gm-APD intensity map is enhanced, and then the Gm-APD range map is guided by the Gm-APD intensity map. The high-resolution intensity image Z is taken as the super-resolution Gm-APD intensity image, and the low-resolution range image X is taken as the Gm-APD range image. Specific Implementation Example 8:

[0106] The difference between Embodiment 8 and Embodiment 7 of this application lies only in:

[0107] This invention provides a neural network-based super-resolution reconstruction system for 3D range images of lidar, the system comprising:

[0108] A pixel extraction module, which uses an ideal high-resolution distance image Y to represent any pixel s on the low-resolution distance image X based on a set of high-resolution intensity images Z and low-resolution distance images X;

[0109] The loss function module establishes a loss function based on the determination that there is a nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution distance image Y.

[0110] The super-resolution reconstruction module increases the pixel mapping range of the high-resolution intensity image Z, introduces image pixel coordinate encoding and region pixel encoding, and performs super-resolution reconstruction. Specific Implementation Example Nine:

[0112] The difference between Embodiment Nine and Embodiment Eight in this application lies only in:

[0113] The present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a neural network-based method for super-resolution reconstruction of three-dimensional range images of lidar. Specific Implementation Example 10:

[0115] The only difference between Embodiment 10 and Embodiment 9 of this application is that:

[0116] The present invention provides a computer device, including a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a super-resolution reconstruction method for three-dimensional range images of LiDAR based on neural networks.

[0117] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or N embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified. Any process or method described in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logical functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain. The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or N wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM).Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory. It should be understood that various parts of the invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0118] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments. Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0119] The above description is merely a preferred embodiment of a neural network-based super-resolution reconstruction method for 3D range images of LiDAR. The scope of protection for this neural network-based super-resolution reconstruction method for 3D range images of LiDAR is not limited to the above embodiments; all technical solutions falling within this conceptual framework are within the scope of protection of this invention. It should be noted that for those skilled in the art, any improvements and variations made without departing from the principles of this invention should also be considered within the scope of protection of this invention.

Claims

1. A super-resolution reconstruction method for 3D range images of lidar based on neural networks, characterized by: The method includes the following steps: Step 1: Based on a set of high-resolution intensity images Z and low-resolution distance images X, represent any pixel s on the low-resolution distance image X with an ideal high-resolution distance image Y; Step 2: Determine the nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution range image Y, and establish the loss function; Step 2 specifically involves: There is a nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution range image Y, which is expressed by the following equation: For a real image, any pixel s on the low-resolution range image X is represented by the following formula: Where q represents a pixel in the D×D region of the high-resolution intensity image Z, and Q represents the 5×5 neighborhood of q, where the neighborhood size is insufficient to be filled with zeros. Under constraints, we expect to obtain a set of parameters θ that minimizes the loss function, expressed by the following formula: When using the L2 norm for regularization in a network, the loss function can be expressed as follows: Step 3: Increase the pixel mapping range of the high-resolution intensity image Z, introduce image pixel coordinate encoding and region pixel encoding, and perform super-resolution reconstruction.

2. The method for super-resolution reconstruction of three-dimensional range images of lidar based on neural networks according to claim 1, characterized in that: Step 1 specifically involves: For a set of corresponding high-resolution intensity images Z and low-resolution range images X, the ideal high-resolution range image is Y, and the ideal high-resolution range image Y has the same dimensions as the high-resolution intensity image Z. The m×m region on the low-resolution range image X corresponds to the (D×m)×(D×m) region mapped to the ideal high-resolution range image Y, where D is the ratio of the high-resolution range image Y to the low-resolution range image X. Any pixel s on the low-resolution range image X is represented by the following formula:

3. The method for super-resolution reconstruction of three-dimensional range images of lidar based on neural networks according to claim 2, characterized in that: Step 3 specifically involves: By increasing the pixel mapping range of image Z and introducing image pixel coordinate encoding, f1 is for a single pixel, so single-pixel encoding is performed using the pixel index; f2 is for region pixels, so pixels within the same region have the same index for region pixel encoding. The loss function can then be expressed as: When the original range image has low resolution, interpolation is required to compare the range values ​​with the processed high-resolution range image. This involves upsampling the original range image through interpolation to complete super-resolution reconstruction.

4. The method for super-resolution reconstruction of three-dimensional range images of lidar based on neural networks according to claim 3, characterized in that: Since each pixel in a high-resolution intensity image is not completely independent of its neighborhood, a 2-layer 3×3 convolution is used for each pixel block to extract neighborhood information.

5. A method for super-resolution reconstruction of three-dimensional range images of lidar based on neural networks according to any one of claims 1-4, characterized in that: The high-resolution intensity image Z is taken as the ICCD intensity image, and the low-resolution distance image X is taken as the Gm-APD distance image.

6. A method for super-resolution reconstruction of three-dimensional range images of lidar based on neural networks according to any one of claims 1-4, characterized in that: The high-resolution intensity image Z is taken as the Gm-APD intensity image after super-resolution processing, and the low-resolution distance image X is taken as the Gm-APD distance image.

7. A neural network-based super-resolution reconstruction system for three-dimensional range images of lidar, the system operating based on the neural network-based super-resolution reconstruction method for three-dimensional range images of lidar according to claim 1, characterized in that: The system includes: A pixel extraction module, which uses an ideal high-resolution distance image Y to represent any pixel s on the low-resolution distance image X based on a set of high-resolution intensity images Z and low-resolution distance images X; The loss function module establishes a loss function based on the determination that there is a nonlinear mapping relationship between the high-resolution intensity image Z and the ideal high-resolution distance image Y. The super-resolution reconstruction module increases the pixel mapping range of the high-resolution intensity image Z, introduces image pixel coordinate encoding and region pixel encoding, and performs super-resolution reconstruction.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement a neural network-based super-resolution reconstruction method for three-dimensional range images of LiDAR as described in any one of claims 1-6.

9. A computer device, characterized in that, The system includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a neural network-based super-resolution reconstruction method for three-dimensional range images of a lidar system according to any one of claims 1-6.

Images