A train ranging method and system based on binocular stereo matching with point cloud supervision

Abstract

The present invention relates to a train ranging method based on binocular stereo matching supervised by point cloud, the method comprising the following steps: S1: collecting binocular images and point cloud data of trains; S2: constructing a binocular stereo matching network, inputting the binocular images into the binocular stereo matching network, using the point cloud data as a supervisory signal, and conducting supervisory training on the binocular stereo matching network to obtain a coarse disparity image; S3: constructing a residual optimization network, adding detailed information of the point cloud data reconstructed in advance to the coarse disparity image through the residual optimization network for network training, and obtaining a final disparity image; S4: converting the final disparity image into a depth image, and obtaining the driving distance between trains according to the depth image. The present invention supervises the binocular depth stereo matching network through the point cloud data collected by the offline radar to optimize the area with errors in the disparity image, thereby improving the accuracy of train ranging.

Description

Technical Field

[0001] The present invention relates to the field of train distance measurement methods, and in particular to a train distance measurement method and system based on binocular stereo matching under point cloud supervision. Background Art

[0002] Stereo matching is an important research direction in the field of computer vision, which aims to recover the depth information of a scene from two or more images with different perspectives. Traditional stereo matching methods usually include four key steps: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization.

[0003] In the matching cost calculation stage, the algorithm evaluates the correlation between the pixels to be matched and the candidate pixels, providing a basis for the subsequent matching process. Cost aggregation improves the robustness and accuracy of matching by establishing connections between adjacent pixels. Commonly used methods such as semi-global matching (SGM) can comprehensively consider the spatial relationship between pixels, thereby improving matching accuracy. The disparity calculation stage determines the optimal disparity value of the pixel in the cost volume through algorithms such as the winner-takes-all (WTA). This process is the core of stereo matching and directly determines the accuracy of depth information. The disparity optimization stage aims to correct erroneous matching points and improve the quality of the final disparity map. This is usually achieved through algorithms such as left-right consistency check, which effectively reduces the error differences caused by occlusion and noise.

[0004] In recent years, the rapid development of deep learning technology has provided a new solution for stereo matching. Stereo matching methods based on deep learning extract image features through a deep network, construct correlation volumes, and then perform disparity calculation and optimization. Among them, feature extraction and correlation volume construction are two key steps based on deep learning methods.

[0005] In the feature extraction stage, the deep network can extract rich feature information from the left and right view images, and use multi-scale strategies to capture details at different levels. In the correlation volume construction stage, the algorithm needs to accurately reflect the correlation between the left and right viewpoint pixels to provide accurate matching information for subsequent disparity calculations. Although deep learning-based stereo matching methods have achieved great success, there are still some problems to be solved, such as establishing correspondence and posture information between the left and right image views in mirrors, low-texture areas, and occluded areas. Summary of the invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problem that the prior art cannot accurately capture the detail information of low-texture areas such as object edges and mirrors, and provide a train ranging method based on binocular stereo matching supervised by point cloud, comprising the following steps:

[0007] S1: Collect binocular images and point cloud data of the train;

[0008] S2: constructing a binocular stereo matching network, inputting the binocular image into the binocular stereo matching network, using the point cloud data as a supervision signal, performing supervision training on the binocular stereo matching network, and obtaining a coarse disparity image;

[0009] S3: constructing a residual optimization network, adding the detail information of the point cloud data reconstructed in advance to the coarse disparity image through the residual optimization network to perform network training, and obtaining a final disparity image;

[0010] S4: Convert the final disparity image into a depth image, and obtain the running distance between trains according to the depth image.

[0011] In one embodiment of the present invention, step S1 further includes performing epipolar correction on the binocular image and performing superposition correction on the front and rear frames of the point cloud data.

[0012] In one embodiment of the present invention, a specific method for obtaining a coarse parallax image includes:

[0013] S21: performing multi-scale downsampling operations on the binocular images to obtain feature images of multiple scales at left and right viewing angles, and stacking the feature images of multiple scales and the feature images of the original scale together according to the left and right viewing angles to form a multi-scale feature pyramid in the left image and a multi-scale feature pyramid in the right image;

[0014] S22: Define a search operator Lc, given a current disparity d, use the search operator Lc to extract features related to the current disparity from the multi-scale feature pyramid of the left image and the different-scale feature maps of the multi-scale feature pyramid of the right image, connect the retrieved disparity values ​​of the same scale into a single feature map, determine the optimal disparity value of each pixel by comparing the features at different scales, and combine the optimal disparity values ​​of all pixels together to form a coarse disparity image.

[0015] In one embodiment of the present invention, the method for obtaining the final disparity image includes: converting the point cloud data into a two-dimensional sparse disparity image through posture information, inputting the coarse disparity image output by the binocular stereo matching network and the two-dimensional sparse disparity image into the residual optimization network, and using the disparity image predicted by the residual optimization network and the point cloud supervision image to perform error estimation using the L1 norm to obtain the final disparity image.

[0016] In one embodiment of the present invention, the method for acquiring the two-dimensional sparse parallax image includes: calculating a projection matrix according to the posture information, and projecting the point cloud data onto a two-dimensional plane based on the projection matrix to obtain a two-dimensional sparse parallax image; wherein the posture information is the position and posture of the laser radar and the binocular camera when acquiring the point cloud data.

[0017] In one embodiment of the present invention, the regression loss function L used by the binocular stereo matching network and the residual optimization network during network training is:

[0018]

[0019] Among them, I pre represents the predicted disparity image, I gt Represents the sparse disparity map generated from point cloud data, d represents the disparity value, and n represents the number of images.

[0020] In one embodiment of the present invention, the calculation formula for converting the final disparity image into a depth map is:

[0021]

[0022] Where D is the depth value of the current pixel, B is the baseline of the optical center of the binocular camera, f is the focal length of the camera, and Disp is the disparity value of the current pixel.

[0023] Based on the same inventive concept, the present invention also provides a train ranging system based on binocular stereo matching supervised by point cloud, comprising the following steps:

[0024] The train's binocular image and point cloud data acquisition module is used to collect the train's binocular image and point cloud data;

[0025] A coarse disparity image generation module is used to construct a binocular stereo matching network, input the binocular image into the binocular stereo matching network, use the point cloud data as a supervision signal, perform supervision training on the binocular stereo matching network, and obtain a coarse disparity image;

[0026] A final disparity image generation module is used to construct a residual optimization network, and add detailed information of the dense laser radar point cloud reconstructed in advance to the coarse disparity image through the residual optimization network to obtain a final disparity image;

[0027] The module for calculating the running distance between trains is used to convert the final parallax image into a depth image, and obtain the running distance between trains according to the depth image.

[0028] The present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected via the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the train ranging method based on binocular stereo matching with point cloud supervision.

[0029] The present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the train ranging method based on binocular stereo matching under point cloud supervision.

[0030] The above technical solution of the present invention has the following advantages compared with the prior art:

[0031] 1. The present invention can effectively improve the accuracy of stereo matching by introducing point cloud data as a supervisory signal to supervise the training of the binocular stereo matching network. Point cloud data contains rich spatial information, which can make up for the deficiency of image information in depth perception, thereby improving the quality of parallax images, and then obtaining more accurate depth images and the measurement of the driving distance between trains.

[0032] 2. The present invention constructs and applies a residual optimization network, so that the detailed information of the reconstructed point cloud data can be effectively added to the coarse disparity image. This can not only repair the disparity errors caused by various reasons, but also improve the detail expression ability of the disparity map, making the final disparity image closer to the real scene, thereby improving the reliability of train ranging.

[0033] 3. The present invention can handle train ranging problems in different scenarios including complex environments, occluded areas, low-texture areas, etc. By combining point cloud data and image information, more robust stereo matching and depth estimation can be achieved in these challenging scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0035] Figure 1 It is a flow chart of an implementation of a train ranging method based on binocular stereo matching of point cloud supervision provided in an embodiment of the present invention;

[0036] Figure 2 is a specific flow chart of generating a final parallax image provided in an embodiment of the present invention;

[0037] Figure 3 is a coarse disparity image generated by a binocular stereo matching network provided in an embodiment of the present invention;

[0038] Figure 4 is the final disparity image generated by residual network fusion provided in an embodiment of the present invention;

[0039] Figure 5 1 is a structural schematic diagram of a train ranging system based on binocular stereo matching of point cloud supervision provided in an embodiment of the present invention;

[0040] Description of the accompanying drawings in the specification: 100, binocular image and point cloud data acquisition module of the train; 200, coarse parallax image generation module; 300, final parallax image generation module; 400, driving distance calculation module between trains. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0042] Embodiment 1

[0043] Reference Figures 1-2 As shown, the present invention provides a train ranging method based on binocular stereo matching of point cloud supervision, and the method comprises the following steps:

[0044] S1: Collect binocular images and point cloud data of the train;

[0045] S2: constructing a binocular stereo matching network, inputting the binocular image into the binocular stereo matching network, using the point cloud data as a supervision signal, performing supervision training on the binocular stereo matching network, and obtaining a coarse disparity image;

[0046] S3: constructing a residual optimization network, adding the detail information of the point cloud data reconstructed in advance to the coarse disparity image through the residual optimization network to perform network training, and obtaining a final disparity image;

[0047] S4: Convert the final disparity image into a depth image, and obtain the running distance between trains according to the depth image.

[0048] In one embodiment of the present invention, step S1 further includes performing epipolar correction on the binocular image and performing superposition correction on the front and rear frames of the point cloud data.

[0049] In step S2, the specific method of obtaining the coarse parallax image includes:

[0050] S21: performing multi-scale downsampling operations on the binocular images to obtain feature images of multiple scales at left and right viewing angles, and stacking the feature images of multiple scales and the feature images of the original scale together according to the left and right viewing angles to form a multi-scale feature pyramid in the left image and a multi-scale feature pyramid in the right image;

[0051] S22: Define a search operator Lc, given a current disparity d, use the search operator Lc to extract features related to the current disparity from the multi-scale feature pyramid of the left image and the different-scale feature maps of the multi-scale feature pyramid of the right image, connect the retrieved disparity values ​​of the same scale into a single feature map, determine the optimal disparity value of each pixel by comparing the features at different scales, and combine the optimal disparity values ​​of all pixels together to form a coarse disparity image.

[0052] Specifically, the binocular image is downsampled by 8 times, 16 times, and 32 times respectively to obtain feature images of 1 / 8, 1 / 16, and 1 / 32 of the original size. These feature images are stacked with the original resolution feature image into a 4-level feature pyramid, which makes it easier to extract detail features to optimize the final disparity image. At the same time, in order to index the relevant pyramid, a search operator L is defined c The operator operates on feature maps with resolutions of 1 / 8, 1 / 16, and 1 / 32 simultaneously. Given the current disparity d, a one-dimensional grid with integer offsets is constructed around the current disparity, and then the lookup operator L is used to find c Index at each scale, then connect the same disparity values ​​of each scale retrieved into a single feature map, determine the optimal disparity value of each pixel by comparing the features at different scales, and combine the optimal disparity values ​​of all pixels together to form a feature map such as Figure 3 Coarse parallax image shown.

[0053] In step S3, if Figure 2 As shown, the method for obtaining the final disparity image includes: converting the point cloud data into a two-dimensional sparse disparity image through posture information, inputting the coarse disparity image output by the binocular stereo matching network and the two-dimensional sparse disparity image into the residual optimization network, and using the disparity image predicted by the residual optimization network and the point cloud supervision image to perform error estimation using the L1 norm to obtain the following Figure 4 The final disparity image is shown.

[0054] Specifically, the method for acquiring the two-dimensional sparse disparity image includes: calculating a projection matrix according to the posture information, and projecting the point cloud data onto a two-dimensional plane based on the projection matrix to obtain a two-dimensional sparse disparity image; wherein the posture information is the position and posture of the laser radar and the binocular camera when acquiring the point cloud data.

[0055] Accordingly, the values ​​of the disparity value d greater than 0 are regarded as valid pixels, so the regression loss function L used in the network training process of constructing the binocular stereo matching network and the residual optimization network is:

[0056]

[0057] Among them, I pre represents the predicted disparity image, I gt Represents the sparse disparity map generated from point cloud data, d represents the disparity value, and n represents the number of images.

[0058] In step S4, the calculation formula for converting the final disparity image into a depth map is:

[0059]

[0060] Where D is the depth value of the current pixel, B is the baseline of the optical center of the binocular camera, f is the focal length of the camera, and Disp is the disparity value of the current pixel.

[0061] Embodiment 2

[0062] Based on the same inventive concept as the train ranging method described in Example 1, Figure 5 As shown, the present invention also provides a train ranging system based on binocular stereo matching of point cloud supervision, which includes the following steps:

[0063] The train binocular image and point cloud data acquisition module 100 is used to acquire the train binocular image and point cloud data;

[0064] A coarse disparity image generation module 200 is used to construct a binocular stereo matching network, input the binocular image into the binocular stereo matching network, use the point cloud data as a supervision signal, perform supervision training on the binocular stereo matching network, and obtain a coarse disparity image;

[0065] The final disparity image generation module 300 is used to construct a residual optimization network, and add the detail information of the dense laser radar point cloud reconstructed in advance to the coarse disparity image through the residual optimization network to obtain a final disparity image;

[0066] The module 400 for calculating the distance between trains is used to convert the final parallax image into a depth image, and obtain the distance between trains according to the depth image.

[0067] Embodiment 3

[0068] The present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected via the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the train ranging method based on binocular stereo matching based on point cloud supervision described in Example 1.

[0069] Embodiment 4

[0070] The present invention also provides a computer storage medium, which stores a computer software product. The computer software product includes several instructions for enabling a computer device to execute the train ranging method based on binocular stereo matching based on point cloud supervision described in Example 1.

[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A train ranging method based on binocular stereo matching with point cloud supervision, characterized in that: The following steps are involved: S1: Collect binocular images and point cloud data of the train; S2: constructing a binocular stereo matching network, inputting the binocular image into the binocular stereo matching network, using the point cloud data as a supervision signal, performing supervision training on the binocular stereo matching network, and obtaining a coarse disparity image; The specific method of obtaining the coarse parallax image includes: S21: performing multi-scale downsampling operations on the binocular images to obtain feature images of multiple scales at left and right viewing angles, and stacking the feature images of multiple scales and the feature images of the original scale together according to the left and right viewing angles to form a multi-scale feature pyramid in the left image and a multi-scale feature pyramid in the right image; S22: Define a search operator L c , given the current disparity d, using the search operator L c Extracting features related to the current disparity from the multi-scale feature pyramid in the left figure and the feature maps of different scales in the multi-scale feature pyramid in the right figure, connecting the retrieved disparity values ​​of the same scale into a single feature map, determining the optimal disparity value of each pixel by comparing the features at different scales, and combining the optimal disparity values ​​of all pixels together to form a coarse disparity image; S3: constructing a residual optimization network, adding the detail information of the point cloud data reconstructed in advance to the coarse disparity image through the residual optimization network to perform network training, and obtaining a final disparity image; The method for obtaining the final disparity image includes: converting the point cloud data into a two-dimensional sparse disparity image through posture information, inputting the coarse disparity image output by the binocular stereo matching network and the two-dimensional sparse disparity image into the residual optimization network, and performing error estimation using the disparity image predicted by the residual optimization network and the point cloud supervision image using the L1 norm to obtain the final disparity image; S4: Convert the final disparity image into a depth image, and obtain the running distance between trains according to the depth image.

2. The train ranging method based on binocular stereo matching of point cloud supervision according to claim 1 is characterized in that: Step S1 also includes performing epipolar correction on the binocular image and performing superposition correction on the front and back frames of the point cloud data.

3. The train ranging method based on binocular stereo matching of point cloud supervision according to claim 1, characterized in that: The method for acquiring the two-dimensional sparse disparity image includes: calculating a projection matrix according to the posture information, and projecting the point cloud data onto a two-dimensional plane based on the projection matrix to obtain a two-dimensional sparse disparity image; wherein the posture information is the position and posture of the laser radar and the binocular camera when acquiring the point cloud data.

4. The train ranging method based on binocular stereo matching supervised by point cloud according to claim 1, characterized in that: The regression loss function L used by the binocular stereo matching network and the residual optimization network during network training is: Among them, I pre represents the predicted disparity image, I gt Represents the sparse disparity map generated from point cloud data, d represents the disparity value, and n represents the number of images.

5. The train ranging method based on binocular stereo matching of point cloud supervision according to claim 1, characterized in that: The calculation formula for converting the final disparity image into a depth image is: Where D is the depth value of the current pixel, B is the baseline of the optical center of the binocular camera, f is the focal length of the camera, and Disp is the disparity value of the current pixel.

6. A train ranging system based on binocular stereo matching with point cloud supervision, characterized in that: A train ranging method based on binocular stereo matching supervised by point cloud according to any one of claims 1 to 5 is implemented, wherein the system comprises the following modules: The train's binocular image and point cloud data acquisition module is used to collect the train's binocular image and point cloud data; A coarse disparity image generation module is used to construct a binocular stereo matching network, input the binocular image into the binocular stereo matching network, use the point cloud data as a supervision signal, perform supervision training on the binocular stereo matching network, and obtain a coarse disparity image; A final disparity image generation module is used to construct a residual optimization network, and add detailed information of the dense laser radar point cloud reconstructed in advance to the coarse disparity image through the residual optimization network to obtain a final disparity image; The module for calculating the running distance between trains is used to convert the final parallax image into a depth image, and obtain the running distance between trains according to the depth image.

7. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the train ranging method based on binocular stereo matching based on point cloud supervision as described in any one of claims 1 to 5.

8. A computer storage medium, characterized in that: The computer storage medium stores a computer software product, and the computer software product includes a number of instructions for enabling a computer device to execute the train ranging method based on binocular stereo matching based on point cloud supervision as described in any one of claims 1 to 5.

Images