Distance measuring method and device based on binocular vision and mobile equipment

Abstract

The invention relates to a distance measuring method and device based on binocular vision and mobile equipment. The method is applied to a mobile device carrying a binocular camera, and comprises the following steps: obtaining an initial disparity map at the current moment according to previous moment positioning data, current moment positioning data and a historical disparity map corresponding to the previous moment of the mobile device; the historical disparity map is obtained by predicting a binocular image at the previous moment acquired by a binocular camera through a pre-trained distance measurement model; inputting the initial disparity map at the current moment and the binocular image at the current moment into a distance measurement model to obtain a target disparity map corresponding to the current moment; obtaining a target depth map corresponding to the current moment according to the target disparity map at the current moment; the distance measurement model is obtained through different-scale cascade pre-training according to a sample binocular image in a training set, a corresponding disparity map label and an initial sample disparity map. According to the scheme provided by the invention, long-distance real-time distance measurement can be accurately carried out.

Description

Technical Field

[0001] The present application relates to the field of binocular distance measurement technology, and in particular to a distance measurement method, device and mobile device based on binocular vision. Background Art

[0002] Binocular ranging technology uses two different cameras to shoot the same scene from different angles, and uses the parallax between the two images to calculate the distance of the target object. Among them, the traditional binocular ranging algorithm traverses the pixels of the left image, uses the matching algorithm to obtain the corresponding feature points on the right image, calculates the parallax of the feature points on the left and right images, and then obtains the depth of the pixel points on the left image based on the parallax and depth relationship. However, in binocular ranging technology, affected by factors such as the scene and ambient lighting, such as the lack of sufficient visual features in the scene, the degradation of imaging quality caused by changes in ambient lighting angle or intensity, will lead to difficulty in feature extraction or even matching failure, which in turn leads to inaccurate or invalid depth estimation results, especially in real-time ranging at long distances (such as more than 300 meters).

[0003] Therefore, in the long-distance ranging scenarios of aircraft flight and take-off and landing, how to use binocular ranging technology to accurately measure the distance of target objects in real time still needs to be improved. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a binocular vision-based ranging method, device and mobile device.

[0005] In a first aspect, the present application provides a binocular vision-based ranging method, which is applied to a mobile device equipped with a binocular camera, comprising: According to the positioning data of the mobile device at the last moment, the positioning data at the current moment, and the historical disparity map corresponding to the last moment, an initial disparity map at the current moment is obtained; wherein the historical disparity map is obtained by predicting the binocular image at the last moment acquired by the binocular camera using a pre-trained ranging model; Inputting the initial disparity map at the current moment and the binocular image at the current moment into the distance measurement model to obtain the target disparity map corresponding to the current moment; According to the target disparity map at the current moment, obtain the target depth map corresponding to the current moment; The distance measurement model is pre-trained by cascading different scales according to the sample binocular images in the training set and the corresponding disparity map labels and the initial sample disparity map.

[0006] In the present application, the target disparity map corresponding to the current moment is obtained by inputting the initial disparity map and the binocular image at the current moment into a pre-trained ranging model, wherein the initial disparity map is determined according to the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment. With the help of the initial disparity map, the temporal characteristics of the scene can be fully utilized, thereby reducing the iteration rounds and the time consumption of the ranging model, improving the calculation efficiency, and achieving the effect of real-time ranging; at the same time, in the ranging model, multiple cascade iterations of different scales can iterate the disparity from the lowest resolution to a higher resolution to obtain a globally unified disparity, thereby reducing the flying point problem caused by the model in the upsampling process, and improving the accuracy of disparity estimation. Finally, the target depth map corresponding to the current moment is obtained according to the target disparity map at the current moment, which can improve the accuracy of the target depth map, that is, to achieve accurate long-distance real-time ranging.

[0007] In some implementations, obtaining the initial disparity map at the current moment according to the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment includes: Update the external parameters of the binocular camera according to the current positioning data of the mobile device and the previous positioning data to obtain updated external parameters; According to the updated external parameters, the historical disparity map corresponding to the previous moment is converted into an initial disparity map at the current moment.

[0008] In some implementations, the step of inputting the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment includes: Perform feature extraction on the binocular image at the current moment to obtain binocular feature maps of different scales; According to the binocular feature maps of different scales, the initial disparity map at the current moment is updated in cascade at different scales to obtain the target disparity map corresponding to the current moment.

[0009] In some implementations, the ranging model is pre-trained in the following manner: Acquire a training set, wherein the training set includes a sample binocular image and a corresponding disparity map label and an initial sample disparity map, wherein the sample binocular image includes a first image and a second image; Performing feature extraction on the first image to obtain a plurality of frames of first feature maps of different scales; performing feature extraction on the second image to obtain a plurality of frames of second feature maps of different scales; According to the first feature map and the second feature map, the initial sample disparity map of the training set is updated multiple times at the same scale to obtain a first disparity map corresponding to each update, and the first disparity map obtained by the last update of the current scale is scale-transformed to serve as the initial sample disparity map of the next scale; Respectively performing scale transformation on each frame of the first disparity map at each scale to obtain a corresponding second disparity map with an original scale; The loss is calculated according to the disparity map label and all the second disparity maps, and the parameters of the ranging model are optimized by back propagation until the calculated loss reaches the training stop condition, thereby obtaining the ranging model.

[0010] In some implementations, the scaling of the first disparity map obtained by the last update of the current scale as the initial sample disparity map of the next scale includes: The first disparity map obtained by the last update of the current scale is upsampled to obtain the same first disparity map as the next level scale, and is used as the initial sample disparity map for the first disparity update of the next level scale.

[0011] In some implementations, performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map to obtain a first disparity map corresponding to each update includes: At the same scale, the first disparity map obtained by the current disparity update is used as the initial sample disparity map for the next disparity update.

[0012] In some implementations, performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map to obtain a first disparity map corresponding to each update includes: In a k-th disparity update at the same scale, obtaining a spatial correlation of the first feature map and the second feature map at a current scale, where k is a positive integer; According to the spatial correlation, a disparity search is performed on the initial sample disparity map of the k-th disparity update with a set radius to obtain a correlation index; splicing the correlation index, the first feature map of the current scale, and the initial sample disparity map of the k-th disparity update, and using the matrix obtained by the splicing as a hidden state; Performing a disparity update on the hidden state using a gated recurrent unit to obtain a disparity increment corresponding to the k-th disparity update; The disparity increment of the k-th disparity update is superimposed on the initial sample disparity map of the k-th disparity update to obtain a first disparity map corresponding to the k-th disparity update of the current scale.

[0013] In some implementations, the scaling of each frame of the first disparity map at each scale to obtain a corresponding second disparity map with an original scale includes: Performing multiple upsampling convolutions on a first feature map having the same scale as the first disparity map to generate a corresponding mask; According to the mask, the first disparity map is scaled in a sliding window and / or interpolation form to obtain a second disparity map of an original scale.

[0014] A second aspect of the present application provides a binocular vision-based ranging device, which is applied to a mobile device equipped with a binocular camera, and includes a disparity conversion module, a disparity prediction module, and a depth acquisition module; The disparity conversion module is used to obtain an initial disparity map at the current moment according to the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment; wherein the historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model; The disparity prediction module is used to input the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment; The depth acquisition module is used to obtain a target depth map corresponding to the current moment according to the target disparity map at the current moment; The distance measurement model is pre-trained by cascading different scales according to the sample binocular images in the training set and the corresponding disparity map labels and the initial sample disparity map.

[0015] A third aspect of the present application provides a mobile device, including: Processor; and The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0016] In some implementations, the mobile device is an aircraft or a flying car.

[0017] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of a vehicle, the processor is caused to execute the method as described above.

[0018] A fifth aspect of the present application provides a computer program product or a computer program, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described above.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0021] Figure 1 It is a schematic flow chart of the binocular vision-based ranging method shown in the present application; Figure 2 It is a flowchart of the training method of the ranging model shown in this application; Figure 3 A structural diagram of the training method of the ranging model shown in this application; Figure 4 Another schematic diagram of the flow of the training method of the ranging model shown in this application; Figure 5 is another schematic flow chart of a binocular vision-based distance measurement method shown in the present application; Figure 6 Another schematic diagram of the process of the binocular vision-based distance measurement method shown in this application; Figure 7 It is a structural schematic diagram of a binocular vision-based distance measuring device shown in the present application; Figure 8 It is a schematic diagram of the structure of the mobile device shown in this application. DETAILED DESCRIPTION

[0022] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0023] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0024] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0025] In binocular ranging technology, factors such as the scene and ambient lighting affect the image quality, such as the lack of sufficient visual features in the scene, and changes in ambient lighting angle or intensity, which can lead to difficulty in feature extraction or even matching failure, and thus inaccurate or invalid depth estimation results, especially in long-distance (such as more than 300 meters) real-time ranging. Therefore, in the long-distance ranging scenario of aircraft flight and takeoff and landing, how to accurately measure the distance of the target object in real time with the help of binocular ranging technology still needs to be improved.

[0026] In response to the above problems, the present application provides a ranging method based on binocular vision, which can achieve accurate long-distance real-time ranging.

[0027] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0028] Figure 1 It is a flow chart of the binocular vision-based ranging method shown in this application.

[0029] See also Figure 1 The present application shows a binocular vision-based ranging method, which is applied to a mobile device equipped with a binocular camera, including: S110, obtaining an initial disparity map at the current moment according to the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment; wherein the historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model.

[0030] In some embodiments, the mobile device may be a flying device, or an amphibious device with both flying and land functions, such as a flying car, etc. This is only an example and is not intended to be limiting. A binocular camera is an imaging device with two lenses. The two lenses are usually arranged in parallel at a certain distance (called a baseline). It simulates the visual principle of human eyes and simultaneously obtains two images of the same scene through two lenses. The two images have a slight difference in perspective, similar to the different scenes seen by a person's left and right eyes. The binocular camera in this embodiment refers to a left camera and a right camera installed on the same mobile device to shoot the same real scene based on different perspectives.

[0031] Among them, the disparity map is used to record the difference information of the corresponding pixel positions between the images captured by the binocular camera under two different perspectives in the same scene, and the difference information is caused by the different installation positions of the left camera and the right camera in the binocular camera on the mobile device. To explain, the disparity map is a form of image representation, which displays the disparity information between the left and right images of the binocular camera in the form of a grayscale image or a color image; in the disparity map, the value of each pixel represents the disparity size of the corresponding position of the pixel in the left and right images. Assuming that the optical axes of the left camera and the right camera are parallel, each pixel in the disparity map represents the horizontal displacement of a point in the real scene in the two images. For example, for a point in the real scene, its coordinates in the image taken by the left camera are (X L , Y), the coordinates in the image taken by the right camera are (X R , Y), the disparity d can be defined as d=X L -X R .

[0032] In this step, when the aircraft is flying and moving, the positioning data at different times are different, so the external parameters of the binocular camera will naturally be different. Therefore, the new external parameters can be determined by the positioning data of two adjacent moments, and the historical disparity map corresponding to the previous moment is converted into the initial disparity map corresponding to the current moment, maintaining the temporal correlation. Among them, the historical disparity map corresponding to the previous moment is specifically obtained by predicting the binocular image of the previous moment acquired by the binocular camera according to the pre-trained ranging model, that is, the historical disparity map corresponding to the previous moment is the target disparity map corresponding to the previous moment output by the ranging model. The embodiment of the present application converts the target disparity map corresponding to the previous moment into the initial disparity map of the current moment through the positioning data of the previous moment and the current moment, introduces the timing characteristics, and with the help of the initial disparity map, the timing characteristics of the passing scene can be fully utilized, thereby reducing the iteration rounds and the time consumption of the ranging model, and can solve the problem that the iteration rounds are too time-consuming and cannot meet the real-time ranging.

[0033] It should be noted that when the ranging model is run for the first time, the initial disparity map at the current moment is a given disparity map, and the disparity value of the given disparity map can be zero. In the subsequent operation of the ranging model, the initial disparity map at the current moment is converted from the target disparity map corresponding to the previous moment, which can reduce the number of iterations and meet the needs of long-distance real-time ranging.

[0034] S120, inputting the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment; wherein the ranging model is pre-trained by cascading different scales according to the sample binocular images in the training set and the corresponding disparity map labels and the initial sample disparity map.

[0035] In this step, the ranging model is obtained by pre-training. It can be understood that in the process of pre-training of the ranging model, the training set used includes the initial sample disparity map, the sample binocular image and the corresponding disparity map label, wherein the ranging model performs cascade updates of different scales according to the initial sample disparity map and the sample binocular image, and then performs loss calculation according to the update result and the disparity map label to train until convergence, and obtains a trained ranging model. In the ranging model, multiple cascade iterations of different scales can iterate the disparity from low resolution to higher resolution to obtain a globally unified disparity, thereby reducing the flying point problem caused by the upsampling process and improving the accuracy of disparity estimation. In this embodiment, after obtaining the initial disparity map at the current moment and the binocular image at the current moment, the initial disparity map at the current moment and the binocular image at the current moment can be input into the ranging model to obtain the target disparity map corresponding to the current moment, which is obtained by the ranging model processing, and also reduces the flying point problem caused by upsampling the acquired binocular image during the dynamic flight of the aircraft, thereby improving the accuracy of disparity estimation.

[0036] S130, obtaining a target depth map corresponding to the current moment according to the target disparity map at the current moment.

[0037] The depth map describes the distance information of each pixel in the scene relative to the binocular camera in the form of a grayscale image or a color image, that is, the value of each pixel represents the depth value of the point in three-dimensional space. The larger the depth value, the farther the scene point corresponding to the pixel is from the binocular camera, and vice versa. The depth map is closely related to the disparity map. Each pixel in the disparity map represents the horizontal displacement of the target point in the space between the two images. Then each pixel in the depth map represents the depth value of the target point in the space. It is known that the focal length of the camera is f, the baseline of the left camera and the right camera is b, and the disparity of the left camera and the right camera is d=X L -X R, then, according to the disparity solution formula of the binocular camera, we can know the depth of the target point Z = (f·b) / d. Therefore, we can obtain the target depth map corresponding to the current moment based on the target disparity map at the current moment to accurately judge the distance of the target object (obstacle) in front.

[0038] In binocular ranging technology, the method of using deep learning is based on an end-to-end neural network model to directly map image input to depth map output. Represented by RAFT (Recurrent All-Pairs Field Transforms for Optical Flow, an optical flow estimation method based on cyclic full-pairs field transform), most binocular ranging based on deep learning uses disparity iterative optimization to perform depth estimation. In practical scenarios, long-distance ranging needs to be performed on high-resolution images, and as the number of iterations increases, the amount of calculation increases, and the real-time performance of the model is difficult to guarantee. Further, deep learning algorithms include convolutional neural networks (CNNs), Transformers, etc. CNNs can capture local and global information in images, but their ability to capture global contextual information is limited; Transformers have global capture capabilities but high computational complexity, and their ability to capture local details and texture information in images may not be as good as CNNs. Given that binocular ranging requires obtaining the disparity of the corresponding pixels of the left and right eyes, CNNs are still widely used in current binocular ranging neural networks. However, using CNN on high-resolution images requires downsampling and upsampling the input image by dozens or even tens of times, which may lead to inaccurate ranging results and flying spot problems.

[0039] In order to further improve the accuracy of long-distance real-time distance measurement, the distance measurement method based on binocular vision in this application also improves the training of the distance measurement model. Figure 2 , Figure 3 and Figure 4 , the ranging model needs to be pre-trained. In some implementations, the training method of the ranging model of the present application includes: S210, obtaining a training set, where the training set includes a sample binocular image and a corresponding disparity map label and an initial sample disparity map, and the sample binocular image includes a first image and a second image.

[0040] The first image may be a left-eye image or a right-eye image, and the second image may be a right-eye image or a left-eye image. In this embodiment, the first image is a left-eye image, and the second image is a right-eye image.

[0041] In step S210, the data sources of the training set include collected data, public data sets, simulation data, etc., wherein the collected data can be obtained by using the onboard binocular camera to obtain the left eye image and the right eye image, and by using laser radar, SLAM (Simultaneous Localization and Mapping) mapping, etc. to obtain the depth map in the image coordinate system, and then obtain the internal parameters, external parameters and pixel depth of the left and right cameras according to the binocular camera calibration to obtain the disparity map of the left eye image, and the disparity value of the initial sample disparity map is zero. The obtained training set is divided into a training set and a verification set, which are used for training and verification of the ranging model, respectively. Before the data is input into the ranging model to be trained, the data in the training set needs to be preprocessed, and the preprocessing can include cleaning the data according to the actual situation, such as removing irrelevant or erroneous data points.

[0042] S220, extract features from the first image to obtain multiple frames of first feature maps of different scales; extract features from the second image to obtain multiple frames of second feature maps of different scales.

[0043] In this embodiment of the present application, the scales of the first feature map and the second feature map can be determined according to the number of cascade layers. In this embodiment, for the sake of convenience, the first feature map is the left-eye feature map and the second feature map is the right-eye feature map. If the number of cascade layers in the cascade structure of different scales is X, then the corresponding scale can be 1 / 2 X+1 , 1 / 2 X , …, 1 / 8, 1 / 4, 1 / 2 and 1. Assuming that the number of cascade layers in the cascade structure of different scales is three layers, according to actual needs, the corresponding scales can be 1 / 16, 1 / 8, 1 / 4, or 1 / 8, 1 / 4, 1 / 2, or 1 / 32, 1 / 16, 1 / 8. Assuming that the number of layers is four, the corresponding scales can be 1 / 32, 1 / 16, 1 / 8, 1 / 4. In this embodiment, the number of cascade layers can be three layers, and the different scales correspond to 1 / 16, 1 / 8, and 1 / 4. After processing at different scales, considering that the fine details in the original image may be lost at some small scales, and the original image of scale 1 / 1 can completely retain these details, then in the last update, the scale 1 / 1 is combined for updating, and these important details can be integrated into the processing results, thereby improving the accuracy of the prediction results.

[0044] In one embodiment of step S220, the input left-eye image I L and right eye image I R , the left eye feature map f with scales of 1 / 16, 1 / 8, 1 / 4, and 1 / 1 is extracted by the first 2D convolution module with shared network parameters L1and the right eye feature map f with scales of 1 / 16, 1 / 8, and 1 / 4 respectively R1 Among them, the first 2D convolution module includes N 3*3 convolution layers, BN (BatchNormalization) layers and RELU (Rectified Linear Unit) layers. By setting the stride (step length) of the first 2D convolution module to 2, a feature map that is downsampled by 2 times is obtained, and the corresponding feature map is obtained after 4 convolution layers with a stride of 2.

[0045] like Figure 3 As shown, in another embodiment of step S220, the input left-eye image I L and right eye image I R , the left eye feature map f with scales of 1 / 16, 1 / 8, and 1 / 4 is extracted by the first 2D convolution module with shared network parameters L1 and the right eye feature map f with scales of 1 / 16, 1 / 8, and 1 / 4 respectively R1 At the same time, the second 2D convolution module is used to extract the left eye feature map f of 1 / 16, 1 / 8, 1 / 4, and 1 / 1 of the left eye separately. L2 The second 2D convolution module is used to connect the feature extraction layers of different downsampling multiples in the first 2D convolution module, and the feature map with a scale of 1 / 16 is the feature map with the lowest resolution. The second 2D convolution module includes N 3*3 convolution layers with a sride (step) of 1, a BN layer, and a RELU layer.

[0046] Therefore, the second 2D convolution module can select the left-eye feature of the first 2D convolution module, and is also used to output a left-eye feature map of scale 1 / 1; the second 2D convolution module can also be connected to the feature extraction layer with different downsampling multiples in the first 2D convolution module, and is also used to output a left-eye feature map of scale 1 / 1. This embodiment preferably is another embodiment of the above-mentioned step S220.

[0047] S230, performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map, obtaining a first disparity map corresponding to each update, and performing a scale transformation on the first disparity map obtained by the last update of the current scale as the initial sample disparity map of the next level scale.

[0048] In step S230, multiple disparity updates are performed on the initial sample disparity map of the training set at the same scale, and disparity updates are also performed at different scales. The disparity update is not limited to the disparity update at the same scale or at different scales, and the process of each disparity update is similar, wherein the first disparity map obtained in the last disparity update is directly or indirectly used as the initial sample disparity map of the current disparity update.

[0049] In some embodiments, for updates between different scales, the first disparity map obtained by the last update of the current scale is scale-transformed and used as the initial sample disparity map of the next scale. In at least one embodiment, the updates between different scales may include upsampling the first disparity map obtained by the last update of the current scale to obtain the same first disparity map as the next scale, and use it as the initial sample disparity map for the first disparity update of the next scale.

[0050] Assuming that the current scale is 1 / 16 and the next level scale is 1 / 8, then the first disparity map obtained by the last disparity update of scale 1 / 16 is upsampled by 2 times through interpolation to obtain a disparity map of scale 1 / 8, and the disparity map of scale 1 / 8 is used as the initial sample disparity map for the first disparity update of scale 1 / 8. It should be noted that the initial sample disparity map of the first disparity update of scale 1 / 16 is a disparity map with a given disparity value of zero, and its scale is the same as the left eye feature map f L2 same.

[0051] In some embodiments, the update between the same scales may include using the first disparity map obtained by the current disparity update as the initial sample disparity map for the next disparity update at the same scale.

[0052] Assuming that the current scale is 1 / 16, for multiple disparity updates at the scale of 1 / 16, the first disparity map obtained in the last disparity update is used as the initial sample disparity map of the current disparity update.

[0053] Furthermore, multiple disparity updates are performed on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map to obtain the first disparity map corresponding to each update, including obtaining the spatial correlation of the current scale of the first feature map and the second feature map in the k-th disparity update at the same scale, where k is a positive integer; based on the spatial correlation, a disparity search is performed on the initial sample disparity map of the k-th disparity update with a set radius to obtain a correlation index; the correlation index, the first feature map of the current scale, and the initial sample disparity map of the k-th disparity update are spliced, and the spliced ​​matrix is ​​used as a hidden state; the hidden state is disparity updated using a gated recurrent unit to obtain a disparity increment corresponding to the k-th disparity update; the disparity increment of the k-th disparity update is superimposed on the initial sample disparity map of the k-th disparity update to obtain the first disparity map corresponding to the k-th disparity update of the current scale.

[0054] It should be noted that, except for the initial sample disparity map of the first disparity update of the ranging model which is a given disparity map, other initial sample disparity maps used are directly or indirectly obtained from the first disparity map obtained in the last disparity update.

[0055] Among them, taking the first disparity update of the scale 1 / 16 corresponding to the lowest resolution as an example, first, the left eye feature map f is calculated L1 and the right eye feature map f R1 The corresponding scale is 1 / 16 of the spatial correlation volume. The Correlation Volume is used to characterize the correlation between the left eye feature map and the right eye feature map. The Correlation Volume is first calculated on the 1 / 16 low-resolution binocular feature map to reduce the amount of calculation to obtain a rough overall disparity. Secondly, for each pixel coordinate on the given initial sample disparity map, a disparity search is performed in the Correlation Volume within a set radius r to obtain a Correlation Volume index with a channel number of 2r+1. Thirdly, the left eye feature map f L2 , the given initial sample disparity map, and the Correlation Volume index are spliced ​​into a matrix as the hidden state (HiddenState), which is input into the gated recurrent unit GRU for state update, and the updated hidden state is passed through 2~3 3*3 convolutional layers to obtain the disparity increment Δd. Finally, the disparity increment Δd is superimposed on the given initial sample disparity map to obtain the disparity map d1 of scale 1 / 16 corresponding to the first disparity update, and a disparity update is completed. Thereafter, the disparity map of scale 1 / 16 obtained from the first disparity update is used as the initial sample disparity map for the second disparity update, and the above steps are repeated i times to obtain i disparity maps d1~d at scale 1 / 16. i , i can be a positive integer greater than 1.

[0056] The disparity map d is obtained at the i-th disparity update of scale 1 / 16. i After that, the disparity map d i The disparity map with a scale of 1 / 8 is obtained by upsampling by 2 times through interpolation, and the disparity map with a scale of 1 / 8 obtained here is used as the initial sample disparity map for the first disparity update with a scale of 1 / 8. The disparity update is repeated j times, and the difference is that the left eye feature map f is calculated. L1 and the right eye feature map f R1 The corresponding spatial correlation volume at scale 1 / 8, as well as other corresponding parts, can obtain j disparity maps d at scale 1 / 8 i+1 ~d i+j , by updating the parallax with the second lowest resolution of 1 / 8 scale, the medium-detail parallax can be refined while preserving the overall parallax.

[0057] The j-th disparity update at scale 1 / 8 obtains the disparity map d i+j After that, the disparity map d i+jThe disparity map of scale 1 / 4 is obtained by upsampling by 2 times through interpolation, and the obtained disparity map of scale 1 / 4 is used as the initial sample disparity map of scale 1 / 4 for the first disparity update. Then the disparity update is repeated p times to obtain p disparity maps d at scale 1 / 4. i+j+1 ~d i+j+p Among them, the sub-high resolution disparity update at scale 1 / 4 can obtain more detailed disparity results and more accurate ranging. At the same time, the initial sample disparity map comes from the update results of scale 1 / 16 and scale 1 / 8, which can reduce the number of required update iterations.

[0058] It should be noted that Correlation Volume is a data structure. In the field of computer vision and machine learning, especially when it comes to tasks such as feature matching and pose estimation, Correlation Volume is a data structure used to measure the correlation between two feature maps. It is not a simple specific value, but a set of multiple correlation values, which represent the degree of correlation between two feature maps under different positions, different dimensions, etc.

[0059] The correlation index, that is, the Correlation Volume index, belongs to the disparity dimension index. In the disparity dimension, the index is used to access the related information corresponding to different disparities. As mentioned earlier, the disparity search in the Correlation Volume with a radius r range obtains the index of the channel number 2r+1. The disparity index here can range from 0 to 2r, and each index corresponds to a disparity value. For example, index 0 may correspond to the minimum disparity value (such as at the beginning of the disparity search range), index 1 corresponds to a slightly larger disparity value, and so on. Through this disparity dimension index, the correlation measure corresponding to different disparity values ​​at a specific spatial position can be obtained, which can be used to compare the correlation under different disparities and find the best disparity match.

[0060] In the gated recurrent unit, the hidden state is an abstract representation of sequence information, which is similar to the hidden state in the traditional recurrent neural network (RNN), and is used to store and transmit past information to help the model process and predict the current elements in the sequence.

[0061] S240 , performing scale transformation on each frame of the first disparity map at each scale to obtain a corresponding second disparity map with the original scale.

[0062] Among them, step S240 may include performing multiple upsampling convolutions on the first feature map of the same scale as the first disparity map to generate a corresponding mask; according to the mask, scaling the first disparity map in the form of sliding window and / or interpolation to obtain a second disparity map of the original scale. It is supplemented that the above-mentioned first disparity map refers to the disparity map d at the corresponding scale obtained each time. z The second disparity map is a disparity map fd obtained by scaling the first disparity map to obtain the original scale 1 / 1 z , where z=1, 2,…, i+j+p.

[0063] It should be noted that step S230 and step S240 may be performed in any order or simultaneously.

[0064] Assuming that the process is carried out in sequence, after obtaining all disparity maps d1~d i+j+p After that, each disparity map is scaled to obtain the disparity map fd1~fd corresponding to the scale 1 / 1 i+j+p Assuming that the operations are performed simultaneously, after all first disparity maps at the same scale are obtained, scale conversion is performed on each first disparity map.

[0065] Taking the disparity map d1 as an example, the left eye feature map f at scale 1 / 16 L2 Then, a multi-layer 3*3 convolution is performed to obtain a mask with n*n*3*3 channels, where n is a positive integer. The number of convolution layers can be increased or decreased according to the time consumption and performance requirements, and can be 2~3 layers. Then, a sliding window is used to weight each disparity of the disparity map d1 with a set sliding window size to obtain a 1 / 4 downsampled disparity map, and then a bilinear interpolation is performed to obtain a disparity map f with the original scale of 1 / 1. d1 Similarly, with the disparity map d i+1 For example, the left eye feature map f at scale 1 / 8 L2 Connect multiple layers of 3*3 convolution to obtain the corresponding disparity map f of scale 1 / 1 di+1 .

[0066] At the i+j+pth update, the disparity map f is obtained through the disparity map d1 d1 The difference is that in the 1 / 1 left eye feature map f L2 Connect multiple layers of 3*3 convolution to obtain a mask with n*n channels. According to the original scale 1 / 1 mask, the 1 / 4 disparity map d i+j+p Perform weighted upsampling and learn the left eye feature map f through mask L2 The characteristics of the weighting process can effectively reduce the flying point problem caused by the weighting process.

[0067] S250, calculating the loss according to the disparity map label and all the second disparity maps, and optimizing the parameters of the ranging model by back propagation until the calculated loss reaches the training stop condition, thereby obtaining the ranging model.

[0068] like Figure 4 As shown, in step S250, the disparity maps fd1~fd i+j+p The predicted values ​​are y1, y2, ... y i+j+p , the true value of the disparity map label is y, where the disparity map label is obtained by solving the real depth map. Assume that the loss function is L=1 / (i+j+p)·[(y-y1)+(y-y2)+…+(yy i+j+p )]. Of course, this application may also use other loss functions, which are only given as examples and are not intended to be limiting.

[0069] In the forward propagation process of step S220 to step S240, the predicted value is obtained, the loss is calculated, and then a back propagation is performed to iterate the network parameters of the ranging model. When the calculated loss is below the set loss value, the training is considered to have converged, and the training is stopped to obtain a trained ranging model.

[0070] Among them, in the training process of the ranging model, the initial weight and initial deviation of the ranging model can be randomly initialized or loaded by a pre-trained model. The input data is forward propagated through the ranging model, the loss is calculated, and the gradient is calculated by back propagation. Finally, the optimizer is used to update the network parameters of the ranging model, and the ranging model can also be an algorithm model. In this process, multiple cycles will be iterated until the ranging model can reach stability or meet the set training stop conditions. Common loss functions for binocular disparity estimation include L1 loss, smooth L1 loss, etc., and optimizers include SGD, Adam, RMSprop, etc. The network weights in the ranging model are adjusted by the optimizer to minimize the loss function. The embodiment of the present application can be used or not according to actual needs. The performance of the ranging model can be periodically evaluated on the validation set to detect the overfitting or undertraining of the ranging model. Common evaluation indicators include EPE, D1, etc. When the validation set indicators meet the requirements, stop training to obtain the final ranging model.

[0071] In some variations of the embodiments, the multi-scale cascade structure may be applied to different scales. For example, a first disparity map may be generated at a scale of 1 / 32 times downsampling, and then downsampled to 1 / 16 times to continue disparity updating.

[0072] It can be seen that in the training process of the embodiment of the present application, the update and iteration characteristics of the ranging model are fully utilized, a multi-scale cascade structure is designed, the initial sample disparity is determined from the feature map with the lowest resolution, and then the initial sample disparity is input to the feature map with a higher resolution than the lowest resolution for disparity update, so that a globally unified disparity can be obtained on the low-resolution feature map while reducing the model time consumption, and the downsampled disparity map is weighted by learning the feature map of the original scale, thereby reducing the flying point problem caused by the upsampling process.

[0073] The binocular vision-based ranging method of the present application will be described in more detail below with reference to the accompanying drawings.

[0074] like Figure 5 and 6 As shown, in some implementations, the binocular vision-based ranging method of the present application is applied to an aircraft equipped with a binocular camera, including steps S310 to S330.

[0075] S310, obtaining an initial disparity map at the current moment according to the positioning data of the aircraft at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment output by the ranging model.

[0076] Among them, the aircraft is in motion, and the positioning data includes the camera posture. The posture at the previous moment is not the same as the posture at the current moment. According to the dynamic change of the aircraft posture, the historical disparity map corresponding to the previous moment can be converted into the initial disparity map at the current moment, wherein the historical disparity map is the target disparity map corresponding to the previous moment.

[0077] Further, step S310 may include updating the external parameters of the binocular camera according to the current positioning data of the aircraft and the positioning data of the previous moment to obtain updated external parameters; according to the updated external parameters, converting the historical disparity map corresponding to the previous moment into the initial disparity map of the current moment.

[0078] In this embodiment, the aircraft is in motion, and the camera pose at the current moment and the previous moment is obtained by the positioning means of the aircraft. The camera pose includes the position and attitude corresponding to the camera. The position is the three-dimensional coordinate of the camera in space, and the attitude is the direction of the camera (such as pitch angle, yaw angle, roll angle). The external parameters are solved according to the current moment pose and the previous moment pose. In this embodiment, the external parameters can be used to convert the points in the camera coordinate system of the previous moment to the current moment, that is, to convert the historical disparity map corresponding to the previous moment (the target disparity map corresponding to the previous moment) to the initial disparity map of the current moment. Generally speaking, the external parameters are used to convert the points in the world coordinate system back to the camera coordinate system at the current moment. Obviously, the camera external parameters are indirectly involved in the conversion of adjacent moments.

[0079] In this embodiment, new extrinsic parameters are determined by the camera poses at two adjacent moments, and the target disparity map corresponding to the previous moment is converted into an initial disparity map corresponding to the current moment, so as to maintain the temporal correlation and introduce the temporal characteristics. With the help of the initial disparity map, the temporal characteristics of the scene can be fully utilized, thereby reducing the time consumption of iteration rounds and the ranging model, and solving the problem that the iteration rounds are too time-consuming and cannot meet the real-time ranging requirements.

[0080] It should be noted that when the ranging model is applied for the first time, the initial disparity map may be a given disparity map, and the disparity value of the given disparity map may be zero. In subsequent applications of the ranging model, the initial disparity map at the current moment is converted from the target disparity map corresponding to the previous moment, which can reduce the number of iterations and meet the needs of long-distance real-time continuous ranging.

[0081] S320, inputting the initial disparity map at the current moment and the binocular image at the current moment into a distance measurement model to obtain a target disparity map corresponding to the current moment.

[0082] The ranging model includes a multi-scale cascade structure, and multiple cascade iterations of different scales can iterate the disparity from the lowest resolution to a higher resolution to obtain a globally unified disparity, thereby reducing the flying point problem caused by the upsampling process and improving the accuracy of disparity estimation. In this embodiment, after obtaining the initial disparity map at the current moment and the binocular image at the current moment, the initial disparity map at the current moment and the binocular image at the current moment can be input into the ranging model to obtain the target disparity map corresponding to the current moment, which is obtained by processing the ranging model, and the flying point problem caused by upsampling the acquired binocular image during the dynamic flight of the aircraft is reduced, thereby improving the accuracy of disparity estimation.

[0083] Further, step S320 may include performing feature extraction on the binocular image at the current moment to obtain binocular feature maps of different scales; and performing cascade updates of different scales on the initial disparity map at the current moment according to the binocular feature maps of different scales to obtain a target disparity map corresponding to the current moment.

[0084] In this embodiment, performing cascade updates of different scales on the initial disparity map at the current moment may further include: performing multiple disparity updates on the initial disparity map according to the extracted features for the same scale.

[0085] Among them, no matter the disparity update is at the same scale or at different scales, the update process is similar. The difference is that at the same scale, the first disparity map obtained by the current disparity update is used as the initial sample disparity map for the next disparity update; at different scales, the first disparity map obtained by the last update of the current scale is upsampled to obtain the same first disparity map as the next level scale, and is used as the initial sample disparity map for the first disparity update at the next level scale.

[0086] Further, step S320 may include performing feature extraction on the left and right eye images to obtain multiple frames of left eye feature maps of different scales and multiple frames of right eye feature maps of different scales; performing multiple disparity updates on the initial disparity map at the same scale according to the left eye feature map and the right eye feature map to obtain a disparity map corresponding to each update, and performing a scale transformation on the disparity map obtained by the last update of the current scale as the initial disparity map of the next scale, and the scale of the initial disparity map of the next scale is the same as that of the next scale; performing a scale transformation on each frame of the disparity map at each scale to obtain a corresponding disparity map with an original scale, and outputting the disparity map with the original scale obtained by the last update, which is the target disparity map.

[0087] Furthermore, in the m-th disparity update of the same scale, the spatial correlation of the binocular feature map at the current scale is obtained, where m is a positive integer; based on the spatial correlation, a disparity search is performed on the initial disparity map of the m-th disparity update with a set radius to obtain a correlation index; the correlation index, the left-eye feature map of the current scale, and the initial disparity map of the m-th disparity update are spliced, and the spliced ​​matrix is ​​used as a hidden state; the hidden state is disparity updated using a gated recurrent unit to obtain a disparity increment corresponding to the m-th disparity update; the disparity increment of the m-th disparity update is superimposed on the initial disparity map of the m-th disparity update to obtain a disparity map corresponding to the m-th disparity update of the current scale.

[0088] The embodiment of the present application can be used for continuous ranging, wherein, for a plurality of continuous binocular image data, the initial disparity map used by the ranging model can be converted from the target disparity map corresponding to the previous moment. For a plurality of discontinuous binocular images, the initial disparity map used by the ranging model can use an initial disparity map with a given disparity value of zero.

[0089] S330, obtaining a target depth map corresponding to the current moment according to the target disparity map at the current moment.

[0090] Obviously, based on the target depth map, the distance between the obstacle and the aircraft can be obtained. In addition, the target disparity map at the current moment is used to convert into the initial disparity map at the next moment, and the calculation is repeated in this way to achieve the purpose of continuous distance measurement.

[0091] It can be seen that the embodiment of the present application makes full use of the disparity update characteristics of the ranging model and the timing characteristics of the actual use scenario. Through the positioning data, the target disparity map output at the previous moment is converted into the initial disparity map at the current moment and input into the ranging model, so that the ranging model iteration rounds are reduced, the running time of the ranging model can be reduced, and the ranging can be performed continuously in real time. At the same time, the multi-scale cascade method can update the disparity from the lowest resolution feature map to the higher resolution feature map, and can also reduce the flying point problem caused by the upsampling process.

[0092] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a binocular vision-based ranging device and corresponding embodiments.

[0093] Figure 7 It is a schematic diagram of the structure of a binocular vision-based distance measuring device shown in this application.

[0094] See also Figure 7 , the binocular vision-based distance measuring device shown in the present application is applied to a mobile device equipped with a binocular camera, including a disparity conversion module 710, a disparity prediction module 720 and a depth acquisition module 730; The disparity conversion module 710 is used to obtain an initial disparity map at the current moment according to the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment; wherein the historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model; The disparity prediction module 720 is used to input the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment; The depth acquisition module 730 is used to obtain a target depth map corresponding to the current moment according to the target disparity map at the current moment; Among them, the ranging model is pre-trained by cascading different scales based on the sample binocular images and corresponding disparity map labels and initial sample disparity maps in the training set.

[0095] Furthermore, the disparity conversion module 710 can be used to update the external parameters of the binocular camera according to the current positioning data of the mobile device and the positioning data of the previous moment to obtain updated external parameters; according to the updated external parameters, the historical disparity map corresponding to the previous moment is converted into the initial disparity map of the current moment.

[0096] Furthermore, the disparity prediction module 720 can be used to extract features from the binocular image at the current moment to obtain binocular feature maps of different scales; based on the binocular feature maps of different scales, the initial disparity map at the current moment is cascaded and updated at different scales to obtain a target disparity map corresponding to the current moment.

[0097] Furthermore, the distance measurement model involved in the above-mentioned disparity prediction module 720 is generated according to the following distance measurement model training device: A training set acquisition unit, used to acquire a training set, wherein the training set includes a sample binocular image and a corresponding disparity map label and an initial sample disparity map, and the sample binocular image includes a first image and a second image; A feature extraction unit is used to extract features from the first image to obtain multiple frames of first feature maps of different scales; and to extract features from the second image to obtain multiple frames of second feature maps of different scales; a disparity updating unit, configured to perform multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map, obtain a first disparity map corresponding to each update, and scale-transform the first disparity map obtained by the last update of the current scale as the initial sample disparity map of the next scale; A scale transformation unit, used to transform the scale of each frame of the first disparity map at each scale to obtain a corresponding second disparity map with an original scale; The parameter optimization unit is used to calculate the loss according to the disparity map label and all the second disparity maps, and optimize the parameters of the ranging model through back propagation until the calculated loss reaches the training stop condition, thereby obtaining the ranging model.

[0098] Further, in the disparity update unit, the first disparity map obtained by the last update of the current scale is scale-transformed and used as the initial sample disparity map of the next scale, including: upsampling the first disparity map obtained by the last update of the current scale to obtain the same first disparity map as the next scale, and using it as the initial sample disparity map for the first disparity update of the next scale.

[0099] Furthermore, in the disparity update unit, the initial sample disparity map of the training set is updated multiple times at the same scale according to the first feature map to obtain the first disparity map corresponding to each update, including: at the same scale, using the first disparity map obtained by the current disparity update as the initial sample disparity map for the next disparity update.

[0100] Furthermore, in the disparity update unit, the initial sample disparity map of the training set is updated with disparity for multiple times at the same scale according to the first feature map to obtain the first disparity map corresponding to each update, including obtaining the spatial correlation of the current scale of the first feature map and the second feature map in the k-th disparity update at the same scale, where k is a positive integer; according to the spatial correlation, a disparity search is performed on the initial sample disparity map of the k-th disparity update with a set radius to obtain a correlation index; the correlation index, the first feature map of the current scale and the initial sample disparity map of the k-th disparity update are spliced, and the spliced ​​matrix is ​​used as a hidden state; the hidden state is disparity updated using a gated recurrent unit to obtain a disparity increment corresponding to the k-th disparity update; the disparity increment of the k-th disparity update is superimposed on the initial sample disparity map of the k-th disparity update to obtain the first disparity map corresponding to the k-th disparity update of the current scale.

[0101] Furthermore, the scale transformation unit is used to perform multiple upsampling convolutions on the first feature map of the same scale as the first disparity map to generate a corresponding mask; according to the mask, the first disparity map is scaled in the form of a sliding window and / or interpolation to obtain a second disparity map of the original scale.

[0102] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0103] Figure 8 It is a schematic diagram of the structure of the mobile device shown in this application.

[0104] See also Figure 8 , the mobile device 800 includes a memory 810 and a processor 820 .

[0105] The processor 820 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0106] The memory 810 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 820 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at runtime. In addition, the memory 810 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 810 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0107] The memory 810 stores executable codes, and when the executable codes are processed by the processor 820 , the processor 820 can execute part or all of the methods described above.

[0108] In some optional embodiments, the mobile device may be an aircraft or a flying car, etc.

[0109] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0110] Alternatively, the present application can also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) on which an executable code (or a computer program or a computer instruction code) is stored. When the executable code (or a computer program or a computer instruction code) is executed by a processor of a server (or a server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0111] The embodiments of the present application also provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any of the methods described in the above embodiments.

[0112] The computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0113] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0114] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A binocular vision-based distance measurement method, applied to a mobile device equipped with a binocular camera, characterized in that: include: According to the positioning data of the mobile device at the last moment, the positioning data at the current moment, and the historical disparity map corresponding to the last moment, an initial disparity map at the current moment is obtained; wherein the historical disparity map is obtained by predicting the binocular image at the last moment acquired by the binocular camera using a pre-trained ranging model; Inputting the initial disparity map at the current moment and the binocular image at the current moment into the distance measurement model to obtain the target disparity map corresponding to the current moment; According to the target disparity map at the current moment, obtain the target depth map corresponding to the current moment; The distance measurement model is pre-trained by cascading different scales according to the sample binocular images in the training set and the corresponding disparity map labels and the initial sample disparity map.

2. The method according to claim 1, characterized in that: The obtaining of the initial disparity map at the current moment according to the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment comprises: Update the external parameters of the binocular camera according to the current positioning data of the mobile device and the previous positioning data to obtain updated external parameters; According to the updated external parameters, the historical disparity map corresponding to the previous moment is converted into an initial disparity map at the current moment.

3. The method according to claim 1, characterized in that The step of inputting the initial disparity map at the current moment and the binocular image at the current moment into the distance measurement model to obtain the target disparity map corresponding to the current moment includes: Perform feature extraction on the binocular image at the current moment to obtain binocular feature maps of different scales; According to the binocular feature maps of different scales, the initial disparity map at the current moment is updated in cascade at different scales to obtain the target disparity map corresponding to the current moment.

4. The method according to claim 1, characterized in that: The distance measurement model is pre-trained in the following manner: Acquire a training set, wherein the training set includes a sample binocular image and a corresponding disparity map label and an initial sample disparity map, wherein the sample binocular image includes a first image and a second image; Performing feature extraction on the first image to obtain multiple frames of first feature maps of different scales; Performing feature extraction on the second image to obtain multiple frames of second feature maps of different scales; According to the first feature map and the second feature map, the initial sample disparity map of the training set is updated multiple times at the same scale to obtain a first disparity map corresponding to each update, and the first disparity map obtained by the last update of the current scale is scale-transformed to serve as the initial sample disparity map of the next scale; Respectively performing scale transformation on each frame of the first disparity map at each scale to obtain a corresponding second disparity map with an original scale; The loss is calculated according to the disparity map label and all the second disparity maps, and the parameters of the ranging model are optimized by back propagation until the calculated loss reaches the training stop condition, thereby obtaining the ranging model.

5. The method according to claim 4, characterized in that The scaling of the first disparity map obtained by the last update of the current scale is used as the initial sample disparity map of the next scale, including: The first disparity map obtained by the last update of the current scale is upsampled to obtain the same first disparity map as the next level scale, and is used as the initial sample disparity map for the first disparity update of the next level scale.

6. The method according to claim 4, characterized in that The method of performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map to obtain a first disparity map corresponding to each update includes: At the same scale, the first disparity map obtained by the current disparity update is used as the initial sample disparity map for the next disparity update.

7. The method according to claim 6, characterized in that The method of performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map to obtain a first disparity map corresponding to each update includes: In a k-th disparity update at the same scale, obtaining a spatial correlation of the first feature map and the second feature map at a current scale, where k is a positive integer; According to the spatial correlation, a disparity search is performed on the initial sample disparity map of the k-th disparity update with a set radius to obtain a correlation index; splicing the correlation index, the first feature map of the current scale, and the initial sample disparity map of the k-th disparity update, and using the spliced ​​matrix as a hidden state; Performing a disparity update on the hidden state using a gated recurrent unit to obtain a disparity increment corresponding to the k-th disparity update; The disparity increment of the k-th disparity update is superimposed on the initial sample disparity map of the k-th disparity update to obtain a first disparity map corresponding to the k-th disparity update of the current scale.

8. The method according to claim 4, characterized in that The scaling of each frame of the first disparity map at each scale is respectively performed to obtain a corresponding second disparity map with an original scale, including: Performing multiple upsampling convolutions on a first feature map having the same scale as the first disparity map to generate a corresponding mask; According to the mask, the first disparity map is scaled in a sliding window and / or interpolation form to obtain a second disparity map of an original scale.

9. A binocular vision-based distance measuring device, applied to a mobile device equipped with a binocular camera, characterized in that: It includes a disparity conversion module, a disparity prediction module and a depth acquisition module; The disparity conversion module is used to obtain an initial disparity map at the current moment according to the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment; wherein the historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model; The disparity prediction module is used to input the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment; The depth acquisition module is used to obtain a target depth map corresponding to the current moment according to the target disparity map at the current moment; The distance measurement model is pre-trained by cascading different scales according to the sample binocular images in the training set and the corresponding disparity map labels and the initial sample disparity map.

10. A mobile device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 8.

11. The mobile device according to claim 10, characterized in that The mobile device is an aircraft or a flying car.

12. A computer-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of a vehicle, the processor is caused to execute the method according to any one of claims 1 to 8.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which implements the method according to any one of claims 1 to 8 when the computer program or the instruction is executed by a processor.

Images