Train front scene depth estimation method based on binocular vision

Abstract

The invention relates to the field of image data, and provides a train front scene depth estimation method based on binocular vision, and the method comprises the steps: obtaining the image data of a train front scene through a binocular camera; after data enhancement, extracting a network to obtain multi-scale features; constructing a cost body by using the features of 1 / 4 resolution; after the geometric features of the cost body and the context features of the image are fused, inputting a pre-constructed multi-scale space attention network, and optimizing an aggregation result of the cost body; performing parallax estimation on an aggregation result of the cost body through parallax regression to obtain an initial parallax map; and optimizing the initial disparity map by using an error perception enhancement module to obtain a final result of the depth of field in front of the train. The method and the device are used for solving the defect of low accuracy during train front depth-of-field estimation in related technologies, depth-of-field estimation is carried out based on image data of a train front scene, and compared with a traditional technology, the method and the device have higher recognition accuracy and keep a higher prediction speed at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for estimating the depth of a scene ahead of a train based on binocular vision. Background Art

[0002] When a train is moving, objects in the scene ahead may hinder the normal movement of the train. Therefore, it is very important to estimate the depth of field of the scene ahead of the train for the safe operation of the train.

[0003] The traditional train depth detection method uses radar as the main perception sensor, which has the advantages of good robustness and strong anti-interference ability.

[0004] However, in actual applications, the traditional method of using radar for depth of field detection has its own defects, that is, the lack of perception of the specific details of the scene in front, that is, the detection accuracy is not high. Summary of the invention

[0005] The present invention provides a method for estimating the depth of the scene in front of a train based on binocular vision, which is used to solve the defect of low accuracy in estimating the depth of field in front of the train in the related art. In the solution of the present application, computer vision technology is used to perform depth estimation based on image data of the scene in front of the train, which has higher recognition accuracy than traditional technologies.

[0006] The present invention provides a method for estimating the depth of a scene ahead of a train based on binocular vision, comprising: Obtain the left view and right view of the scene in front of the train through the binocular camera; After data enhancement is performed on the left view and the right view, a lightweight feature extractor is used to extract features from the left view and the right view to obtain multi-scale features; Use the group correlation method to construct the cost volume using 1 / 4 resolution features; After multiplying the geometric features contained in the cost volume with the expanded image context features, the features are input into a pre-built multi-scale spatial attention network to obtain an aggregation result of the cost volume; The disparity of the aggregation result of the cost volume is estimated through disparity regression, and the initial disparity map is obtained after upsampling; The error perception enhancement module is applied to connect the initial disparity map, the left image, the reconstruction error and the global features of the left image, and input them into the attention enhancement module. Then, the double hourglass model is used to optimize the depth residual map. The initial disparity map and the depth residual map are added to obtain the final estimation result of the depth of field in front of the train.

[0007] According to a method for estimating the depth of a scene in front of a train based on binocular vision provided by the present invention, the data enhancement includes: random image enhancement, geometric asymmetry enhancement and random occlusion enhancement.

[0008] According to a method for estimating the depth of a scene ahead of a train based on binocular vision provided by the present invention, the multi-scale spatial attention network includes a first convolutional layer, a second convolutional layer, a third convolutional layer and a fourth convolutional layer; The first convolutional layer is used to preprocess the fused features; The second convolutional layer is used to obtain multi-scale spatial information of the input data, and to splice the obtained spatial information in the channel dimension; The third convolutional layer is used to convert the spliced ​​spatial information into weights, and perform weighted summation on the spatial information based on the weights; The fourth convolutional layer is used to output the weighted summed aggregation result of the cost volume.

[0009] According to a method for estimating the depth of a scene ahead of a train based on binocular vision provided by the present invention, disparity estimation is performed on the aggregation result of the cost volume through disparity regression, and an initial disparity map is obtained after upsampling, including: For the aggregated cost volume, the expected disparity is calculated for the first two values ​​at each pixel through a normalized exponential function, and the expected disparity is upsampled to the original resolution to obtain an initial disparity map.

[0010] According to a method for estimating the depth of a scene ahead of a train based on binocular vision provided by the present invention, the process of constructing the depth residual map includes: The right image and the initial disparity map are used to perform a warping operation, and the reconstruction error is calculated by comparing the warped right image with the original left image. The initial disparity map, left image, reconstruction error and global features of the left image are connected along the channel dimension and input into the attention enhancement module, and the deep residual map is optimized using the double hourglass model.

[0011] According to a method for estimating the depth of a scene ahead of a train based on binocular vision provided by the present invention, the application error perception enhancement module connects the initial disparity map, the left image, the reconstruction error and the global features of the left image, and inputs them into the attention enhancement module, and then optimizes the double hourglass model to obtain a depth residual map, and adds the initial disparity map to the depth residual map to obtain the final estimation result of the depth of field ahead of the train, including: The initial disparity map, left image, reconstruction error and left image global features are connected by the following formula: ; in, is the initial disparity map, is the left image, is the reconstruction error, Global features of the left image, It is the fusion feature after connection in the channel dimension; The final estimated result of the depth of field in front of the train is calculated by the following formula: ; ; ; in, is the attention enhancement module, is the fusion feature after attention enhancement, It is a double hourglass model. is the depth residual map, This is the estimated result of the depth of field in front of the train.

[0012] In a method for estimating the depth of a scene ahead of a train based on binocular vision provided by the present invention, feature extraction, cost body construction, cost body aggregation, and depth of field estimation results based on the aggregated cost body are performed. In this process, cost body aggregation is achieved through a pre-constructed aggregation network and a multi-scale spatial attention network. Multi-scale spatial attention is used to more effectively combine the geometric features of the cost body and the expanded context features, which can significantly improve the aggregation effect. Furthermore, an error-aware enhancement model is proposed when calculating the depth of field estimation results, and the global feature information on the left is introduced. In combination with the attention mechanism, the accuracy of the initial disparity map is optimized, and the network has good generalization in different scenarios and weather conditions. In summary, the method provided by the present application can accurately identify the depth of field ahead of the train and improve the safety of the train driving process. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 This is one of the flow charts of a method for estimating the depth of a scene ahead of a train based on binocular vision provided by an embodiment of the present invention; Figure 2 This is a second flow chart of a method for estimating the depth of a scene ahead of a train based on binocular vision provided by an embodiment of the present invention; Figure 3is a schematic diagram of the structure of a multi-scale spatial attention network provided by an embodiment of the present invention; Figure 4 is a structural diagram of an error perception enhancement module provided by an embodiment of the present invention; Figure 5 is a schematic diagram of the structure of an attention enhancement module provided by an embodiment of the present invention; Figure 6 is a schematic diagram of the structure of a train front scene depth estimation system based on binocular vision provided by an embodiment of the present invention; Figure 7 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Figure 1 This is one of the flow charts of a method for estimating the depth of a scene ahead of a train based on binocular vision provided by an embodiment of the present invention.

[0017] Figure 2 This is the second flow chart of a method for estimating the depth of a scene ahead of a train based on binocular vision provided in an embodiment of the present invention.

[0018] like Figure 1 and Figure 2 As shown, this embodiment provides a method for estimating the depth of a scene ahead of a train based on binocular vision, which can be executed by a processing device of the train, such as a processor of a driving system, or a cloud processor, and the method includes: Step 101, obtaining a left view and a right view of the scene in front of the train through a binocular camera.

[0019] Depth of field in this embodiment refers to scene depth. Furthermore, in this solution, by detecting the scene depth of the scene in front of the train, the relative distance information between each object in the scene in front of the train and the train can be determined. This information is crucial for train driving and can help the train driver or the train's automatic driving system to promptly identify potential dangers and respond in a timely manner.

[0020] Step 102: After data enhancement is performed on the left view and the right view, a lightweight feature extractor is used to extract features from the left view and the right view to obtain multi-scale features.

[0021] In this embodiment, random image enhancement can be achieved by randomly adjusting the brightness, gamma value, contrast, and saturation of the left and right images; geometric asymmetry enhancement can be achieved through random rotation and displacement and random cropping; and random occlusion enhancement can be achieved by randomly blocking an area in the image and filling it with the average value of its pixels to simulate the situation in which some areas are invisible in the actual scene.

[0022] The purpose of data enhancement is to increase the diversity of data and improve the generalization ability of the model.

[0023] In implementation, a U-Net-style encoder-decoder structure can be used to extract local features from image pairs. Specifically, the encoder structure can use the pre-trained MobileNetV2 as a lightweight backbone to obtain features at four scales. In the decoder stage, the spatial resolution of the feature map is gradually restored through three upsampling modules, and the feature map of the corresponding layer in the encoding stage is fused with the feature map of the current layer through jump connections. Finally, feature maps of four scales with resolutions of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original resolution are obtained. Then, the feature map with 1 / 4 resolution is used to construct the cost body, and the features at other scales of 1 / 8, 1 / 16, and 1 / 32 can be used as a guide for subsequent cost aggregation.

[0024] In practical applications, the size of image data may be H×W×3, where H represents the height of the image, W represents the width of the image, and 3 represents the three channels of the image.

[0025] Step 103, using the group correlation method, constructing a cost volume using features with 1 / 4 resolution.

[0026] The idea of ​​group correlation is to group the features, and each group of features is only correlated with another group of features, thereby reducing the amount of calculation. In this embodiment, the feature vectors can be grouped according to the channel dimension, that is, all channels are divided into several groups, and each feature group contains the same number of feature channels.

[0027] The mathematical expression related to grouping is: ; in, represents the inner product of vectors, The channel dimension representing the feature, Indicates the number of groups, The left feature map The characteristic map of the group, is the right feature map The feature map of the group, the group correlation cost body can be regarded as A set of cost volumes, each cost volume is calculated by the corresponding feature group.

[0028] Step 104, during the cost aggregation process, the geometric features contained in the cost volume are multiplied by the expanded image context features and then input into a pre-constructed multi-scale spatial attention network to optimize the aggregation result of the cost volume.

[0029] The structure of the aggregation network of the model includes three downsampling layers and three upsampling layers. Each downsampling layer includes a 3D convolution with a kernel of 3×3×3 and a stride of 2, and a 3D convolution with a kernel of 3×3×3 and a stride of 1. Each upsampling layer includes a 3D transposed convolution with a kernel of 4×4×4 and a stride of 2, and a 3D convolution with a kernel of 3×3×3 and a stride of 1.

[0030] Context features in this embodiment , which corresponds to the image features of 1 / 8, 1 / 16, and 1 / 32 scales generated in step 102 above. Geometric features It can be obtained through the three downsampling layers of the aggregation network. If the extended context features are directly added to the geometric features, the difference in the disparity dimension will be ignored, resulting in invalid fusion. During the cost aggregation process, we alternately use the multi-scale attention fusion module and the upsampling module. In this embodiment, the multi-scale spatial attention network can more effectively combine context and geometric information.

[0031] Figure 3 Schematic diagram of the structure of a multi-scale spatial attention network provided by an embodiment of the present invention.

[0032] like Figure 3 As shown, in an exemplary embodiment, the multi-scale spatial attention network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer; The first convolutional layer multiplies the expanded contextual features with the geometric features and then preprocesses them through a 3D convolution operation, and then concatenates the preprocessed features with the original features in the channel dimension; One branch of the second convolutional layer includes two 3D convolutions with a kernel of 3×3×3, which are used to obtain low-level spatial information; the other branch includes one 3D convolution with a kernel of 3×3×3, one 3D convolution with a kernel of 5×5×5, one 3D convolution with a kernel of 7×7×7, and one 3D convolution with a kernel of 9×9×9, which are used to capture features at different scales, increase the receptive field layer by layer, and obtain richer multi-level spatial information. The obtained results are spliced ​​in the channel dimension; The third convolutional layer includes a 3D convolution with a kernel of 1×1×1, which is used to adjust the channel dimension and convert the spliced ​​multi-scale spatial attention vector into a weight; The fourth convolutional layer is used to output the weighted summed aggregation result of the cost volume.

[0033] In practical applications, the first convolutional layer satisfies the following formula: ; ; ; in, is the extended context feature, is the geometric feature, For Hadamard, is the fused feature, is the preprocessed feature, The features after splicing.

[0034] The second convolutional layer above satisfies the following formula: ; in, is the spatial information after splicing, For splicing operation, For the first branch, For the second branch, For input data.

[0035] The third convolutional layer satisfies the following formula: ; in, is the weight, for function.

[0036] The fourth convolutional layer satisfies the following formula: ; in, It is the aggregation result of the output cost body.

[0037] Step 105 , performing disparity estimation on the aggregation result of the cost volume through disparity regression, and obtaining an initial disparity map after upsampling.

[0038] In practical applications, for the aggregated cost volume, the first two values ​​at each pixel can be selected, and these values ​​can be performed The activation function is used to calculate the expected disparity, which is of size B×1×H / 4×W / 4. After that, the disparity map can be upsampled to the original resolution B×1×H×W using the weights around each pixel to obtain the initial disparity map.

[0039] Step 106, applying the error perception enhancement module, connecting the initial disparity map, the left image, the reconstruction error and the global features of the left image, and inputting them into the attention enhancement module, and then optimizing the double hourglass model to obtain the depth residual map, adding the initial disparity map and the depth residual map to obtain the final estimation result of the depth of field in front of the train.

[0040] Figure 4 It is a structural diagram of the error perception enhancement module provided in an embodiment of the present invention.

[0041] In practice, the initial disparity map generated in the above embodiment can be used to estimate the depth of field in front of the train, but it still has certain limitations when applied to complex and fine areas. Therefore, the solution in this embodiment is to perform error perception enhancement on the initial disparity map by calculating reconstruction errors, attention enhancement and other operations. Specifically, it is to restore subtle image details by integrating information from the left image and the right image.

[0042] like Figure 4 As shown, in an exemplary embodiment, a distorted right image is obtained based on the right image and the initial disparity map, and a reconstruction error is determined based on the distorted right image and the original left image, including: The distorted right image is obtained by the following formula: ; in is the distorted right image, For data distortion operation, For the right image, is the initial disparity map; The data distortion operation is a method of processing data, and the image data can be processed through Euclidean transformation, similarity transformation, affine transformation, and projection transformation.

[0043] The reconstruction error is determined by the following formula: ; in, is the reconstruction error, For the left image.

[0044] In practical applications, continuous encoding-decoding steps are used in the process of disparity aggregation, which will cause the feature map to lose some of the original semantic information and make the initial disparity map blurry. In order to enhance the disparity prediction ability of the initial disparity map in pathological areas, more original feature information is introduced in this embodiment. The 1 / 4 resolution left image features containing rich semantic information are upsampled to the original resolution through the context network to obtain global feature information, which contains more semantic information and edge high-frequency information. The initial disparity map, the left image, the reconstruction error and the global features of the left image are connected along the channel dimension. The formula is as follows: ; in, is the initial disparity map, is the left image, is the reconstruction error, is the global feature of the left image, It is the fusion feature after the connection in the channel dimension; Traditional residual learning lacks evidence indicating the occurrence of errors. To alleviate this problem, in this embodiment, the concatenated features are input into the attention enhancement module, so that the residual learning process pays more attention to the inaccurate areas. The attention enhancement module is as follows: Figure 5 As shown, it includes channel attention layer and spatial attention layer. First, the importance of each channel is weighted by entering the channel attention layer, and then the spatial attention layer is input to obtain the final weighted result. The formula is as follows: ; in, is the attention enhancement module, It is the fusion feature after attention enhancement.

[0045] The double hourglass model is used for further optimization to obtain the depth residual map, and then the initial disparity map and the residual map are added to obtain the final depth of field estimation result. The formula is as follows: ; ; in, It is a double hourglass model. is the depth residual map, This is the estimated result of the depth of field in front of the train.

[0046] In practical applications, the double hourglass model is built on a cascaded U-Net structure. Each U-Net structure contains 4 upsampling layers, 4 downsampling layers and skip connections, and each layer contains two 3×3 two-dimensional convolutions.

[0047] In an exemplary embodiment, the aggregation network and the multi-scale spatial attention network can be trained in an end-to-end manner. The training process can be trained using the Smooth L1 function, and the total loss function of the training process conforms to the following formula: ; in, and are loss weights, is the initial disparity map with a resolution of 1 / 4 described in the above embodiment, is the final disparity map at full resolution, Represents the true ground disparity map.

[0048] The solution of the present application uses binocular vision to estimate the depth of the scene in front of the train, maintaining a high speed while generating a higher-precision disparity map. The multi-scale spatial attention fusion and error perception enhancement method is adopted to expand the receptive field, fuse spatial information of different scales, and enhance the ability of error perception refinement, which can significantly improve the matching accuracy of the model in shadows, edges, weak textures or textureless areas, thereby reducing the occurrence of mismatches and facilitating subsequent work. The solution of the present application has strong generalization ability, performs data enhancement processing on the left and right images, increases the diversity of data, and enables the model to achieve good results when predicting different scenes.

[0049] A system for estimating the depth of a scene ahead of a train based on binocular vision provided by the present invention is described below. The system for estimating the depth of a scene ahead of a train based on binocular vision described below and the method for estimating the depth of a scene ahead of a train based on binocular vision described above can refer to each other.

[0050] Figure 6 It is a structural schematic diagram of a train front scene depth estimation system based on binocular vision provided by an embodiment of the present invention.

[0051] like Figure 6 As shown, a train front scene depth estimation system based on binocular vision provided in this embodiment includes: Data acquisition module 601, used to acquire the left view and right view of the scene in front of the train through the binocular camera A feature extraction module 602 is used to perform feature extraction on the left view and the right view using a lightweight feature extractor after data enhancement is performed on the left view and the right view to obtain multi-scale features; A cost volume construction module 603 is used to construct a cost volume using a group correlation method and features with a 1 / 4 resolution; The cost volume aggregation module 604 multiplies the geometric features contained in the cost volume with the expanded image context features, and then inputs the multi-scale spatial attention network constructed in advance to obtain the aggregation result of the cost volume; The disparity map generation module 605 performs disparity estimation on the aggregation result of the cost volume through disparity regression, and obtains an initial disparity map after upsampling; The depth of field estimation module 606 is used to apply the error perception enhancement module, connect the initial disparity map, the left image, the reconstruction error and the global features of the left image, and input them into the attention enhancement module. Then, the double hourglass model is used to optimize the depth residual map, and the initial disparity map is added to the depth residual map to obtain the final estimation result of the depth of field in front of the train.

[0052] The specific implementation method of the binocular vision-based train front scene depth estimation system provided in this embodiment can be implemented with reference to the above embodiment and will not be repeated here.

[0053] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a method for estimating the depth of a scene ahead of a train based on binocular vision, the method comprising: Obtain the left view and right view of the scene in front of the train through the binocular camera; After data enhancement is performed on the left view and the right view, a lightweight feature extractor is used to extract features from the left view and the right view to obtain multi-scale features; Use the group correlation method to construct the cost volume using 1 / 4 resolution features; After multiplying the geometric features contained in the cost volume with the expanded image context features, the features are input into a pre-built multi-scale spatial attention network to obtain an aggregation result of the cost volume; The disparity of the aggregation result of the cost volume is estimated through disparity regression, and the initial disparity map is obtained after upsampling; The error perception enhancement module is applied to connect the initial disparity map, the left image, the reconstruction error and the global features of the left image, and input them into the attention enhancement module. Then, the double hourglass model is used to optimize the depth residual map. The initial disparity map and the depth residual map are added to obtain the final estimation result of the depth of field in front of the train.

[0054] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0055] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for estimating the depth of a scene ahead of a train based on binocular vision provided by the above methods. The method includes: Obtain the left view and right view of the scene in front of the train through the binocular camera; After data enhancement is performed on the left view and the right view, a lightweight feature extractor is used to extract features from the left view and the right view to obtain multi-scale features; Use the group correlation method to construct the cost volume using 1 / 4 resolution features; After multiplying the geometric features contained in the cost volume with the expanded image context features, the features are input into a pre-built multi-scale spatial attention network to obtain an aggregation result of the cost volume; The disparity of the aggregation result of the cost volume is estimated through disparity regression, and the initial disparity map is obtained after upsampling; The error perception enhancement module is applied to connect the initial disparity map, the left image, the reconstruction error and the global features of the left image, and input them into the attention enhancement module. Then, the double hourglass model is used to optimize the depth residual map. The initial disparity map and the depth residual map are added to obtain the final estimation result of the depth of field in front of the train.

[0056] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the method for estimating the depth of a scene ahead of a train based on binocular vision provided by the above methods, the method comprising: Obtain the left view and right view of the scene in front of the train through the binocular camera; After data enhancement is performed on the left view and the right view, a lightweight feature extractor is used to extract features from the left view and the right view to obtain multi-scale features; Use the group correlation method to construct the cost volume using 1 / 4 resolution features; After multiplying the geometric features contained in the cost volume with the expanded image context features, the features are input into a pre-built multi-scale spatial attention network to obtain an aggregation result of the cost volume; The disparity of the aggregation result of the cost volume is estimated through disparity regression, and the initial disparity map is obtained after upsampling; The error perception enhancement module is applied to connect the initial disparity map, the left image, the reconstruction error and the global features of the left image, and input them into the attention enhancement module. Then, the double hourglass model is used to optimize the depth residual map. The initial disparity map and the depth residual map are added to obtain the final estimation result of the depth of field in front of the train.

[0057] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0058] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating the depth of a scene ahead of a train based on binocular vision, characterized in that: include: Obtain the left view and right view of the scene in front of the train through the binocular camera; After data enhancement is performed on the left view and the right view, a lightweight feature extractor is used to extract features from the left view and the right view to obtain multi-scale features; Use the group correlation method to construct the cost volume using 1 / 4 resolution features; After multiplying the geometric features contained in the cost volume with the expanded image context features, the features are input into a pre-built multi-scale spatial attention network to obtain an aggregation result of the cost volume; The disparity of the aggregation result of the cost volume is estimated through disparity regression, and the initial disparity map is obtained after upsampling; The error perception enhancement module is applied to connect the initial disparity map, the left image, the reconstruction error and the global features of the left image, and input them into the attention enhancement module. Then, the double hourglass model is used to optimize the depth residual map. The initial disparity map and the depth residual map are added to obtain the final estimation result of the depth of field in front of the train.

2. A method for estimating the depth of a scene ahead of a train based on binocular vision according to claim 1, characterized in that: The data enhancement includes: random image enhancement, geometric asymmetry enhancement and random occlusion enhancement.

3. The method for estimating the depth of a scene ahead of a train based on binocular vision according to claim 1, characterized in that: The multi-scale spatial attention network includes a first convolutional layer, a second convolutional layer, a third convolutional layer and a fourth convolutional layer; The first convolutional layer is used to preprocess the fused features; The second convolutional layer is used to obtain multi-scale spatial information of the input data, and to splice the obtained spatial information in the channel dimension; The third convolutional layer is used to convert the spliced ​​spatial information into weights, and perform weighted summation on the spatial information based on the weights; The fourth convolutional layer is used to output the weighted summed aggregation result of the cost volume.

4. The method for estimating the depth of a scene ahead of a train based on binocular vision according to claim 1, characterized in that: The disparity estimation is performed on the aggregation result of the cost volume through disparity regression, and the initial disparity map is obtained after upsampling, including: For the aggregated cost volume, the expected disparity is calculated for the first two values ​​at each pixel through a normalized exponential function, and the expected disparity is upsampled to the original resolution to obtain an initial disparity map.

5. The method for estimating the depth of a scene ahead of a train based on binocular vision according to claim 1, characterized in that: The process of constructing the depth residual map includes: The right image and the initial disparity map are used to perform a distortion operation, and the reconstruction error is calculated by comparing the distorted right image with the original left image. The initial disparity map, left image, reconstruction error and global features of the left image are connected along the channel dimension and input into the attention enhancement module, and the deep residual map is optimized using the double hourglass model.

6. The method for estimating the depth of a scene ahead of a train based on binocular vision according to claim 1, characterized in that: The application error perception enhancement module connects the initial disparity map, the left image, the reconstruction error and the global features of the left image, and inputs them into the attention enhancement module. Then, the double hourglass model is used to optimize the depth residual map, and the initial disparity map and the depth residual map are added to obtain the final estimation result of the depth of field in front of the train, including: The initial disparity map, left image, reconstruction error and left image global features are connected in the channel dimension using the following formula: ； in, is the initial disparity map, is the left image, is the reconstruction error, is the global feature of the left image, It is the fusion feature after connection in the channel dimension; The final estimated result of the depth of field in front of the train is calculated by the following formula: ； ； ； in, is the attention enhancement module, is the fusion feature after attention enhancement, It is a double hourglass model. is the depth residual map, This is the estimated result of the depth of field in front of the train.

Images