Distance measurement method, device and mobile device based on binocular vision

Abstract

The present application relates to a binocular vision-based ranging method, device, and mobile device. The method is applied to a mobile device equipped with a binocular camera, and includes: obtaining an initial disparity map at the current moment based on the mobile device's last moment positioning data, current moment positioning data, and the historical disparity map corresponding to the last moment; the historical disparity map is predicted by a pre-trained ranging model based on the binocular image at the previous moment acquired by the binocular camera; the initial disparity map at the current moment and the binocular image at the current moment are input into the ranging model to obtain a target disparity map corresponding to the current moment; based on the target disparity map at the current moment, a target depth map corresponding to the current moment is obtained; the ranging model is pre-trained by cascading different scales based on sample binocular images in the training set, the corresponding disparity map labels, and the initial sample disparity map. The solution provided by the present application can accurately measure distance in real time over long distances.

Description

Technical Field

[0001] The present application relates to the field of binocular ranging technology, and in particular to a binocular vision-based ranging method, device and mobile device. Background Art

[0002] Binocular ranging technology uses two different cameras to capture the same scene from different angles, and uses the parallax between the two images to calculate the distance to the target object. Traditional binocular ranging algorithms traverse the pixels of the left image, use a matching algorithm to obtain the corresponding feature points on the right image, calculate the parallax of the feature points on the left and right images, and then determine the depth of the pixel points on the left image based on the relationship between the parallax and depth. However, binocular ranging technology is affected by factors such as the scene and ambient lighting. For example, a lack of sufficient visual features in the scene, or changes in ambient lighting angle or intensity can cause image quality degradation, making feature extraction difficult or even matching failures. This can lead to inaccurate or even invalid depth estimation results, especially in real-time ranging at long distances (e.g., over 300 meters).

[0003] Therefore, in the long-distance ranging scenarios of aircraft flight and takeoff and landing, how to use binocular ranging technology to accurately measure the distance to the target object in real time still needs to be improved. Summary of the Invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a binocular vision-based ranging method, device and mobile device.

[0005] In a first aspect, the present application provides a binocular vision-based ranging method, which is applied to a mobile device equipped with a binocular camera, comprising:

[0006] Obtaining an initial disparity map at the current moment based on the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment; wherein the historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model;

[0007] Inputting the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment;

[0008] According to the target disparity map at the current moment, the target depth map corresponding to the current moment is obtained;

[0009] The ranging model is pre-trained by cascading different scales based on the sample binocular images and corresponding disparity map labels and the initial sample disparity map in the training set.

[0010] In the present application, by inputting the initial disparity map and the binocular image at the current moment into a pre-trained ranging model, a target disparity map corresponding to the current moment is obtained, wherein the initial disparity map is determined based on the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment. With the help of the initial disparity map, the temporal characteristics of the scene can be fully utilized, thereby reducing the number of iteration rounds and the time consumption of the ranging model, improving the computational efficiency, and achieving the effect of real-time ranging. At the same time, in the ranging model, multiple cascade iterations of different scales can iterate the disparity from the lowest resolution to a higher resolution to obtain a globally unified disparity, thereby reducing the flying point problem caused by the model during the upsampling process and improving the accuracy of the disparity estimation. Finally, the target depth map corresponding to the current moment is obtained based on the target disparity map at the current moment, which can improve the accuracy of the target depth map, that is, achieve accurate long-distance real-time ranging.

[0011] In some implementations, obtaining the initial disparity map at the current moment based on the previous moment positioning data of the mobile device, the current moment positioning data, and the historical disparity map corresponding to the previous moment includes:

[0012] Updating the extrinsic parameters of the binocular camera according to the current positioning data and the previous positioning data of the mobile device to obtain updated extrinsic parameters;

[0013] According to the updated external parameters, the historical disparity map corresponding to the previous moment is converted into an initial disparity map at the current moment.

[0014] In some implementations, inputting the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment includes:

[0015] Perform feature extraction on the binocular image at the current moment to obtain binocular feature maps of different scales;

[0016] According to the binocular feature maps of different scales, the initial disparity map at the current moment is updated in cascade at different scales to obtain the target disparity map corresponding to the current moment.

[0017] In some implementations, the ranging model is pre-trained in the following manner:

[0018] Acquire a training set, where the training set includes sample binocular images and corresponding disparity map labels and an initial sample disparity map, where the sample binocular images include a first image and a second image;

[0019] Performing feature extraction on the first image to obtain multiple frames of first feature maps of different scales; performing feature extraction on the second image to obtain multiple frames of second feature maps of different scales;

[0020] performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map, obtaining a first disparity map corresponding to each update, and performing a scale transformation on the first disparity map obtained by the last update at the current scale as the initial sample disparity map at the next scale;

[0021] Performing scale transformation on each frame of the first disparity map at each scale to obtain a corresponding second disparity map with the original scale;

[0022] The loss is calculated according to the disparity map label and all the second disparity maps, and the parameters of the ranging model are optimized by back propagation until the calculated loss reaches the training stop condition, thereby obtaining the ranging model.

[0023] In some embodiments, the scaling of the first disparity map obtained by the last update of the current scale to use it as the initial sample disparity map of the next scale includes:

[0024] The first disparity map obtained by the last update of the current scale is upsampled to obtain the same first disparity map as the next level scale, and is used as the initial sample disparity map for the first disparity update of the next level scale.

[0025] In some embodiments, performing multiple disparity updates on the initial sample disparity map of the training set at the same scale based on the first feature map and the second feature map to obtain the first disparity map corresponding to each update includes:

[0026] At the same scale, the first disparity map obtained by the current disparity update is used as the initial sample disparity map for the next disparity update.

[0027] In some embodiments, performing multiple disparity updates on the initial sample disparity map of the training set at the same scale based on the first feature map and the second feature map to obtain the first disparity map corresponding to each update includes:

[0028] In a k-th disparity update at the same scale, obtaining a spatial correlation of the first feature map and the second feature map at the current scale, where k is a positive integer;

[0029] According to the spatial correlation, a disparity search is performed on the initial sample disparity map of the k-th disparity update with a set radius to obtain a correlation index;

[0030] splicing the correlation index, the first feature map of the current scale, and the initial sample disparity map of the k-th disparity update, and using the matrix obtained by splicing as a hidden state;

[0031] Performing a disparity update on the hidden state using a gated recurrent unit to obtain a disparity increment corresponding to the k-th disparity update;

[0032] The disparity increment of the k-th disparity update is superimposed on the initial sample disparity map of the k-th disparity update to obtain a first disparity map corresponding to the k-th disparity update at the current scale.

[0033] In some embodiments, performing scale transformation on each frame of the first disparity map at each scale to obtain a corresponding second disparity map having the original scale includes:

[0034] Performing multiple upsampling convolutions on a first feature map having the same scale as the first disparity map to generate a corresponding mask;

[0035] According to the mask, the first disparity map is scaled in a sliding window and / or interpolation form to obtain a second disparity map of the original scale.

[0036] A second aspect of the present application provides a binocular vision-based ranging device, which is applied to a mobile device equipped with a binocular camera and includes a disparity conversion module, a disparity prediction module, and a depth acquisition module;

[0037] The disparity conversion module is used to obtain an initial disparity map at the current moment based on the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment; wherein the historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model;

[0038] The disparity prediction module is used to input the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment;

[0039] The depth acquisition module is used to obtain a target depth map corresponding to the current moment according to the target disparity map at the current moment;

[0040] The ranging model is pre-trained by cascading different scales based on the sample binocular images and corresponding disparity map labels and the initial sample disparity map in the training set.

[0041] A third aspect of the present application provides a mobile device, including:

[0042] processor; and

[0043] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method described above.

[0044] In some embodiments, the mobile device is an aircraft or a flying car.

[0045] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of a vehicle, the processor is caused to execute the method described above.

[0046] In a fifth aspect, the present application provides a computer program product or a computer program, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described above.

[0047] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0049] Figure 1 This is a schematic flow chart of the binocular vision-based ranging method shown in this application;

[0050] Figure 2 is a flow chart of the training method of the ranging model shown in this application;

[0051] Figure 3 This is a structural diagram of the training method of the ranging model shown in this application;

[0052] Figure 4 This is another flowchart of the training method of the ranging model shown in this application;

[0053] Figure 5 This is another schematic flow chart of the binocular vision-based ranging method shown in this application;

[0054] Figure 6 This is another flowchart of the binocular vision-based ranging method shown in this application;

[0055] Figure 7 Schematic diagram of the structure of the binocular vision-based distance measuring device shown in this application;

[0056] Figure 8 It is a structural diagram of the mobile device shown in this application. DETAILED DESCRIPTION

[0057] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although the accompanying drawings illustrate embodiments of the present application, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0058] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0059] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0060] Binocular ranging technology is affected by factors such as the scene and ambient lighting. For example, a lack of sufficient visual features in the scene, or image quality degradation caused by changes in ambient lighting angle or intensity, can lead to difficulty in feature extraction or even matching failures, resulting in inaccurate or ineffective depth estimation. This is especially true for long-distance real-time ranging (e.g., over 300 meters). Therefore, in long-distance ranging scenarios such as aircraft flight and takeoff and landing, there is still room for improvement in how to accurately measure the distance to target objects in real time using binocular ranging technology.

[0061] To address the above issues, the present application provides a binocular vision-based ranging method that can achieve accurate long-distance real-time ranging.

[0062] The technical solution of this application is described in detail below with reference to the accompanying drawings.

[0063] Figure 1 It is a flow chart of the binocular vision-based ranging method shown in this application.

[0064] See also Figure 1 This application presents a binocular vision-based ranging method, which is applied to a mobile device equipped with a binocular camera, and includes:

[0065] S110, obtaining an initial disparity map at the current moment based on the mobile device's last moment positioning data, the current moment positioning data, and the historical disparity map corresponding to the last moment; wherein the historical disparity map is obtained by predicting the binocular image at the last moment acquired by the binocular camera using a pre-trained ranging model.

[0066] In some embodiments, the mobile device may be a flying device or an amphibious device with both flying and land capabilities, such as a flying car. These examples are provided here for illustrative purposes only and are not intended to be limiting. A binocular camera is an imaging device with two lenses, typically arranged in parallel with a certain distance (called a baseline). It simulates the visual principles of human binoculars, simultaneously capturing two images of the same scene through the two lenses. These two images have slightly different perspectives, similar to the different views seen by the left and right eyes. In this embodiment, a binocular camera refers to a left camera and a right camera installed on the same mobile device, capturing the same real scene from different perspectives.

[0067] Among them, the disparity map is used to record the difference information of the corresponding pixel positions between the images captured by the binocular camera under two different perspectives in the same scene, and the difference information is caused by the different installation positions of the left camera and the right camera in the binocular camera on the mobile device. To explain, the disparity map is a form of image representation, which displays the disparity information between the left and right images of the binocular camera in the form of a grayscale image or a color image; in the disparity map, the value of each pixel represents the disparity size of the corresponding position of the pixel in the left and right images. Assuming that the optical axes of the left camera and the right camera are parallel, each pixel in the disparity map represents the horizontal displacement of a point in the real scene in the two images. For example, for a point in the real scene, its coordinates in the image taken by the left camera are (X L , Y), the coordinates in the image taken by the right camera are (X R , Y), the disparity d can be defined as d=X L -X R .

[0068] In this step, when the aircraft is flying and moving, the positioning data at different times are different, so the external parameters of the binocular camera will naturally be different. Therefore, the new external parameters can be determined by the positioning data of two adjacent moments, and the historical disparity map corresponding to the previous moment is converted into the initial disparity map corresponding to the current moment, maintaining the temporal correlation. Among them, the historical disparity map corresponding to the previous moment is specifically obtained by a pre-trained ranging model based on the binocular image of the previous moment acquired by the binocular camera. That is, the historical disparity map corresponding to the previous moment is the target disparity map corresponding to the previous moment output by the ranging model. The embodiment of the present application converts the target disparity map corresponding to the previous moment into the initial disparity map of the current moment through the positioning data of the previous moment and the current moment, introduces the temporal characteristics, and with the help of the initial disparity map, the temporal characteristics of the passing scene can be fully utilized, thereby reducing the iteration rounds and the time consumption of the ranging model, and can solve the problem that the iteration rounds are too time-consuming and cannot meet the real-time ranging.

[0069] It should be noted that when the ranging model is first run, the initial disparity map at the current moment is the given disparity map, and the disparity value of this given disparity map can be zero. In subsequent runs of the ranging model, the initial disparity map at the current moment is converted from the target disparity map corresponding to the previous moment, which can reduce the number of iterations and thus meet the requirements of long-distance real-time ranging.

[0070] S120, inputting the initial disparity map at the current moment and the binocular image at the current moment into a ranging model to obtain a target disparity map corresponding to the current moment; wherein the ranging model is pre-trained by cascading different scales based on sample binocular images in a training set and corresponding disparity map labels and the initial sample disparity map.

[0071] In this step, the ranging model is obtained through pre-training. It will be appreciated that during the pre-training process, the ranging model uses a training set including an initial sample disparity map, sample binocular images, and corresponding disparity map labels. The ranging model performs cascade updates at different scales based on the initial sample disparity map and sample binocular images. Loss calculation is then performed based on the update results and the disparity map labels to train until convergence, resulting in a trained ranging model. In the ranging model, multiple cascade iterations at different scales can iterate disparity from low resolution to higher resolution to obtain a globally unified disparity, thereby reducing the flying point problem caused by the upsampling process and improving the accuracy of disparity estimation. In this embodiment, after obtaining the initial disparity map and binocular images at the current moment, these initial disparity map and binocular images at the current moment can be input into the ranging model to obtain the target disparity map corresponding to the current moment. This target disparity map corresponding to the current moment is processed by the ranging model, thereby reducing the flying point problem caused by upsampling the acquired binocular images during dynamic flight of the aircraft, thereby improving the accuracy of disparity estimation.

[0072] S130 , obtaining a target depth map corresponding to the current moment according to the target disparity map at the current moment.

[0073] Among them, the depth map describes the distance information of each pixel in the scene relative to the binocular camera in the form of a grayscale image or a color image, that is, the value of each pixel represents the depth value of the point in three-dimensional space. The larger the depth value, the farther the scene point corresponding to the pixel is from the binocular camera, and vice versa. The depth map is closely related to the disparity map. Each pixel in the disparity map represents the horizontal displacement of the target point in space between the two images, so each pixel in the depth map represents the depth value of the target point in space. Among them, the focal length of the camera is known to be f, the baseline of the left camera and the right camera is b, and the disparity between the left camera and the right camera is d=X L -X R , then, according to the binocular camera disparity solution formula, we can know the depth of the target point Z = (f·b) / d. Therefore, we can obtain the target depth map corresponding to the current moment based on the current target disparity map to accurately determine the distance to the target object (obstacle) in front.

[0074] In binocular odometry, deep learning approaches are based on end-to-end neural network models, directly mapping image input to depth map output. Most deep learning-based binocular odometry approaches, represented by RAFT (Recurrent All-Pairs Field Transforms for Optical Flow), use iterative disparity optimization to estimate depth. Practical scenarios require long-range odometry on high-resolution images, but as the number of iterations increases, the computational complexity increases, making it difficult to ensure real-time performance. Deep learning algorithms include convolutional neural networks (CNNs) and transformers. CNNs can capture both local and global information in an image, but their ability to capture global context is limited. Transformers offer global capture capabilities but are computationally complex, potentially inferior to CNNs in capturing local details and texture information. Given that binocular odometry requires the disparity between corresponding pixels in the left and right eyes, CNNs are still widely used in current neural networks for binocular odometry. However, using CNN on high-resolution images requires downsampling and upsampling the input image by dozens or even tens of times, which may lead to inaccurate ranging results and flying point problems.

[0075] In order to further improve the accuracy of long-distance real-time distance measurement, the present application also improves the training of the distance measurement model based on binocular vision. Figure 2 、 Figure 3 and Figure 4 The ranging model needs to be pre-trained. In some implementations, the training method of the ranging model of the present application includes:

[0076] S210 , obtaining a training set, where the training set includes sample binocular images and corresponding disparity map labels and an initial sample disparity map, and the sample binocular images include a first image and a second image.

[0077] The first image may be a left-eye image or a right-eye image, and the second image may be a right-eye image or a left-eye image. In this embodiment, the first image is a left-eye image, and the second image is a right-eye image.

[0078] In step S210, the data sources of the training set include collected data, public data sets, simulation data, etc. The collected data can be obtained by acquiring the left and right eye images through the onboard binocular camera, and the depth map in the image coordinate system is obtained through lidar, SLAM (Simultaneous Localization and Mapping) mapping, etc., and then the intrinsic parameters, extrinsic parameters and pixel depth of the left and right cameras are obtained based on the binocular camera calibration to obtain the disparity map of the left eye image. The disparity value of the initial sample disparity map is zero. The acquired training set is divided into a training set and a validation set, which are used for training and validation of the ranging model respectively. Before the data is input into the ranging model to be trained, the data in the training set needs to be preprocessed. This preprocessing can include cleaning the data according to the actual situation, such as removing irrelevant or erroneous data points.

[0079] S220 , performing feature extraction on the first image to obtain multiple frames of first feature maps of different scales; performing feature extraction on the second image to obtain multiple frames of second feature maps of different scales.

[0080] Among them, the embodiment of the present application can determine the scale of the first feature map and the second feature map according to the number of cascade layers. In this embodiment, for the sake of convenience, the first feature map is the left eye feature map and the second feature map is the right eye feature map. The number of cascade layers in the cascade structure of different scales is X layers, then the corresponding scale can be 1 / 2 X+1 , 1 / 2 X, ..., 1 / 8, 1 / 4, 1 / 2 and 1. Assuming that the number of cascade layers in the cascade structure of different scales is three, according to actual needs, the corresponding scales can be 1 / 16, 1 / 8, 1 / 4, or 1 / 8, 1 / 4, 1 / 2, or 1 / 32, 1 / 16, 1 / 8. Assuming that the number of layers is four, the corresponding scales can be 1 / 32, 1 / 16, 1 / 8, 1 / 4. In this embodiment, the number of cascade layers can be three, and the different scales correspond to 1 / 16, 1 / 8, and 1 / 4. After processing at different scales, considering that some fine details in the original image may be lost at small scales, while the original image at scale 1 / 1 can completely retain these details, then in the last update, the scale 1 / 1 is combined for updating, so that these important details can be incorporated into the processing results, thereby improving the accuracy of the prediction results.

[0081] In one embodiment of step S220, the input left-eye image I L and right eye image I R , the left eye feature map f with scales of 1 / 16, 1 / 8, 1 / 4, and 1 / 1 is extracted by the first 2D convolution module using shared network parameters. L1 and the right eye feature maps f with scales of 1 / 16, 1 / 8, and 1 / 4 respectively R1 The first 2D convolution module includes N 3*3 convolution layers, BN (Batch Normalization) layers, and RELU (Rectified Linear Unit) layers. By setting the stride of the first 2D convolution module to 2, a feature map that is downsampled by 2 is obtained. The corresponding feature map is obtained after four convolution layers with a stride of 2.

[0082] like Figure 3 As shown, in another embodiment of step S220, the input left eye image I L and right eye image I R , the left eye feature map f with scales of 1 / 16, 1 / 8, and 1 / 4 is extracted by the first 2D convolution module using shared network parameters L1 and the right eye feature maps f with scales of 1 / 16, 1 / 8, and 1 / 4 respectively R1 At the same time, the second 2D convolution module is used to extract the left eye feature map f of 1 / 16, 1 / 8, 1 / 4, and 1 / 1 respectively. L2 The second 2D convolutional module connects to the feature extraction layers of the first 2D convolutional module at different downsampling multiples. The feature map at scale 1 / 16 is the lowest resolution feature map. The second 2D convolutional module includes N 3*3 convolutional layers with a stride of 1, a batch normalization layer, and a relu layer.

[0083] Therefore, the second 2D convolution module can select the left-eye feature of the first 2D convolution module, and is also used to output a left-eye feature map of scale 1 / 1; the second 2D convolution module can also be connected to the feature extraction layer with different downsampling multiples in the first 2D convolution module, and is also used to output a left-eye feature map of 1 / 1. This embodiment preferably is another embodiment of the above-mentioned step S220.

[0084] S230, performing multiple disparity updates on the initial sample disparity map of the training set at the same scale based on the first feature map and the second feature map, obtaining a first disparity map corresponding to each update, and scaling the first disparity map obtained by the last update of the current scale as the initial sample disparity map of the next scale.

[0085] In step S230, multiple disparity updates are performed on the initial sample disparity maps of the training set at the same scale, and also at different scales. Regardless of whether the disparity updates are performed at the same scale or at different scales, the process for each disparity update is similar, with the first disparity map obtained from the previous disparity update being used directly or indirectly as the initial sample disparity map for the current disparity update.

[0086] In some embodiments, for updates between different scales, the first disparity map obtained from the last update of the current scale is scale-transformed and used as the initial sample disparity map for the next scale. In at least one embodiment, the update between different scales may include upsampling the first disparity map obtained from the last update of the current scale to obtain the same first disparity map as that of the next scale, and using this as the initial sample disparity map for the first disparity update at the next scale.

[0087] Assuming the current scale is 1 / 16 and the next level scale is 1 / 8, then the first disparity map obtained from the last disparity update of scale 1 / 16 is upsampled by 2 times through interpolation to obtain a disparity map of scale 1 / 8. The disparity map of scale 1 / 8 is used as the initial sample disparity map for the first disparity update of scale 1 / 8. It should be noted that the initial sample disparity map of the first disparity update of scale 1 / 16 is a disparity map with a given disparity value of zero, and its scale is the same as the left eye feature map f L2 same.

[0088] In some embodiments, updating between the same scales may include using a first disparity map obtained from a current disparity update as an initial sample disparity map for a next disparity update at the same scale.

[0089] Assuming that the current scale is 1 / 16, for multiple disparity updates at the scale 1 / 16, the first disparity map obtained in the last disparity update is used as the initial sample disparity map for the current disparity update.

[0090] Furthermore, the initial sample disparity map of the training set is updated with disparity multiple times at the same scale according to the first feature map and the second feature map to obtain the first disparity map corresponding to each update, including obtaining the spatial correlation of the current scale of the first feature map and the second feature map in the k-th disparity update at the same scale, where k is a positive integer; based on the spatial correlation, a disparity search is performed on the initial sample disparity map of the k-th disparity update with a set radius to obtain a correlation index; the correlation index, the first feature map of the current scale and the initial sample disparity map of the k-th disparity update are spliced, and the spliced ​​matrix is ​​used as a hidden state; the hidden state is updated with disparity using a gated recurrent unit to obtain a disparity increment corresponding to the k-th disparity update; the disparity increment of the k-th disparity update is superimposed on the initial sample disparity map of the k-th disparity update to obtain the first disparity map corresponding to the k-th disparity update of the current scale.

[0091] It should be noted that, except for the initial sample disparity map of the first disparity update of the ranging model, which is a given disparity map, other initial sample disparity maps used are directly or indirectly obtained from the first disparity map obtained in the last disparity update.

[0092] Among them, taking the first disparity update of the scale 1 / 16 corresponding to the lowest resolution as an example, first, the left eye feature map f is calculated L1 and the right eye feature map f R1 The corresponding scale is 1 / 16 of the spatial correlation volume. The Correlation Volume is used to characterize the correlation between the left eye feature map and the right eye feature map. First, the Correlation Volume is calculated on the 1 / 16 low-resolution binocular feature map to reduce the amount of calculation to obtain a rough overall disparity. Secondly, for each pixel coordinate on the given initial sample disparity map, a disparity search is performed in the Correlation Volume within a set radius r to obtain a Correlation Volume index with a channel number of 2r+1. Thirdly, the left eye feature map f L2 , the given initial sample disparity map, and the Correlation Volume index are spliced ​​into a matrix as the hidden state (HiddenState), which is input to the gated recurrent unit GRU for state update, and the updated hidden state is passed through 2~3 3*3 convolutional layers to obtain the disparity increment Δd. Finally, the disparity increment Δd is superimposed on the given initial sample disparity map to obtain the disparity map d1 of scale 1 / 16 corresponding to the first disparity update, that is, a disparity update is completed. After that, the disparity map of scale 1 / 16 obtained from the first disparity update is used as the initial sample disparity map for the second disparity update, and the above steps are repeated i times to obtain i disparity maps d1~d at scale 1 / 16. i, i can be a positive integer greater than 1.

[0093] The disparity map d is obtained at the i-th disparity update of scale 1 / 16. i After that, the disparity map d i The disparity map of scale 1 / 8 is obtained by upsampling by 2 times through interpolation, and the disparity map of scale 1 / 8 obtained here is used as the initial sample disparity map of scale 1 / 8 for the first disparity update, and then the disparity update is repeated j times. The difference is that the left eye feature map f is calculated. L1 and the right eye feature map f R1 The corresponding spatial correlation volume at scale 1 / 8, as well as other corresponding parts, can obtain j disparity maps d at scale 1 / 8. i+1 ~d i+j By updating the parallax at the second-lowest resolution of 1 / 8 scale, the parallax of medium details can be refined while preserving the overall parallax.

[0094] The disparity map d is obtained at the jth disparity update of scale 1 / 8 i+j After that, the disparity map d i+j The disparity map of scale 1 / 4 is obtained by upsampling by 2 times through interpolation, and the disparity map of scale 1 / 4 is used as the initial sample disparity map of scale 1 / 4 for the first disparity update. Then the disparity update is repeated p times to obtain p disparity maps d at scale 1 / 4. i+j+1 ~d i+j+p Among them, the sub-high-resolution disparity update at scale 1 / 4 can obtain finer disparity results and more accurate ranging. At the same time, the initial sample disparity map comes from the update results of scale 1 / 16 and scale 1 / 8, which can reduce the number of required update iterations.

[0095] It's important to note that a Correlation Volume is a data structure used in computer vision and machine learning, particularly in tasks like feature matching and pose estimation, to measure the correlation between two feature maps. Rather than a single numerical value, a Correlation Volume is a collection of multiple correlation values ​​that represent the degree of association between two feature maps at different locations and dimensions.

[0096] The correlation index, that is, the Correlation Volume index, belongs to the disparity dimension index. In the disparity dimension, the index is used to access the relevant information corresponding to different disparities. As mentioned earlier, a disparity search is performed in the Correlation Volume with a radius r to obtain an index of the channel number 2r+1. The disparity index here can range from 0 to 2r, and each index corresponds to a disparity value. For example, index 0 may correspond to the minimum disparity value (such as at the beginning of the disparity search range), index 1 corresponds to a slightly larger disparity value, and so on. Through this disparity dimension index, the correlation measure corresponding to different disparity values ​​at a specific spatial position can be obtained, which can be used to compare the correlation under different disparities and find the best disparity match.

[0097] In a gated recurrent unit, the hidden state is an abstract representation of sequence information, similar to the hidden state in a traditional recurrent neural network (RNN). It is used to store and transmit past information to help the model process and predict the current elements in the sequence.

[0098] S240 , performing scale transformation on each frame of the first disparity map at each scale to obtain a corresponding second disparity map with the original scale.

[0099] Among them, step S240 may include performing multiple upsampling convolutions on the first feature map of the same scale as the first disparity map to generate a corresponding mask; based on the mask, performing scale transformation on the first disparity map in the form of sliding window and / or interpolation to obtain a second disparity map of the original scale. It is supplemented that the above-mentioned first disparity map refers to the disparity map d at the corresponding scale obtained each time. z The second disparity map is the disparity map fd obtained by scaling the first disparity map to obtain the original scale 1 / 1 z , where z=1, 2,…, i+j+p.

[0100] It should be noted that step S230 and step S240 may be performed in any order or simultaneously.

[0101] Assuming that the steps are carried out in sequence, all disparity maps d1~d i+j+p After that, each disparity map is scaled and the disparity maps fd1~fd corresponding to scale 1 / 1 are obtained. i+j+p Assuming that the operations are performed simultaneously, after all first disparity maps at the same scale are obtained, scale conversion is performed on each first disparity map.

[0102] Taking the disparity map d1 as an example, the left eye feature map f at scale 1 / 16 L2Connect multiple layers of 3*3 convolution to obtain a mask with n*n*3*3 channels, where n is a set positive integer. The number of convolution layers here can be increased or decreased according to time consumption and performance requirements, and can be 2~3 layers; then use a sliding window method to calculate each disparity of the disparity map d1 with a set sliding window size to obtain a 1 / 4 downsampled disparity map, and then use bilinear interpolation to obtain the disparity map f of the original scale 1 / 1 d1 Similarly, with the disparity map d i+1 For example, the left eye feature map f at scale 1 / 8 L2 Connect multiple layers of 3*3 convolution to obtain the corresponding scale 1 / 1 disparity map f di+1 .

[0103] At the i+j+pth update, the disparity map f is obtained by the disparity map d1 d1 The difference is that in the left eye feature map f of 1 / 1 L2 Connect multiple layers of 3*3 convolution to obtain a mask with n*n channels, and use the original scale 1 / 1 mask to map the 1 / 4 disparity map d i+j+p Perform weighted upsampling and learn the left eye feature map f through mask L2 The characteristics of the weighted process can effectively reduce the flying point problem caused by the weighting process.

[0104] S250, calculating the loss based on the disparity map label and all second disparity maps, and optimizing the parameters of the ranging model through back propagation until the calculated loss reaches the training stop condition, thereby obtaining the ranging model.

[0105] like Figure 4 As shown, in step S250, the disparity maps fd1~fd i+j+p The predicted values ​​are y1, y2, ... y i+j+p , the true value of the disparity map label is y, where the disparity map label is obtained by solving the real depth map. Assume that the loss function is L=1 / (i+j+p)·[(y-y1)+(y-y2)+…+(yy i+j+p )]. Of course, this application may also adopt other loss functions, which are only given as examples and are not limiting.

[0106] During the forward propagation process from step S220 to step S240, the predicted value is obtained, the loss is calculated, and then a backpropagation is performed to iterate the network parameters of the ranging model. The training is considered to have converged until the calculated loss is below the set loss value, and the training is stopped to obtain a trained ranging model.

[0107] Among them, during the training process of the ranging model, the initial weights and initial biases of the ranging model can be randomly initialized or loaded by a pre-trained model. The input data is forward propagated through the ranging model, the loss is calculated, and then the gradient is calculated by backpropagation. Finally, the optimizer is used to update the network parameters of the ranging model, wherein the ranging model can also be an algorithm model. In this process, multiple cycles will be iterated until the ranging model can reach stability or meet the set training stop conditions. Commonly used loss functions for binocular disparity estimation include L1 loss, smooth L1 loss, etc., and optimizers include SGD, Adam, RMSprop, etc. The network weights in the ranging model are adjusted by the optimizer to minimize the loss function. The embodiment of the present application can be used or not according to actual needs. The performance of the ranging model can be periodically evaluated on the validation set to detect overfitting or insufficient training of the ranging model. Commonly used evaluation indicators include EPE, D1, etc. When the validation set indicators meet the requirements, training is stopped to obtain the final ranging model.

[0108] In some variations of the embodiments, the multi-scale cascade structure can be applied to different scales. For example, a first disparity map can be generated at a scale of 1 / 32 downsampling, and then downsampled to 1 / 16 to continue disparity updating.

[0109] It can be seen that during the training process of the embodiment of the present application, the update and iterative characteristics of the ranging model are fully utilized to design a multi-scale cascade structure. The initial sample disparity is determined from the feature map with the lowest resolution, and then input as the initial sample disparity to the feature map with a higher resolution than the lowest resolution for disparity update. This can obtain a globally unified disparity on the low-resolution feature map while reducing model time consumption. By learning the feature map of the original scale, the downsampled disparity map is weighted, reducing the flying point problem caused by the upsampling process.

[0110] The binocular vision-based ranging method of the present application will be described in more detail below with reference to the accompanying drawings.

[0111] like Figure 5 and 6 As shown, in some embodiments, the binocular vision-based ranging method of the present application is applied to an aircraft equipped with a binocular camera, including steps S310 to S330.

[0112] S310 , obtaining an initial disparity map at the current moment based on the aircraft's last moment positioning data, the current moment positioning data, and the historical disparity map corresponding to the last moment output by the ranging model.

[0113] Among them, the aircraft is in a moving state, and the positioning data includes the camera pose. The pose at the previous moment is different from the pose at the current moment. According to the dynamic change of the aircraft pose, the historical disparity map corresponding to the previous moment can be converted into the initial disparity map at the current moment, where the historical disparity map is the target disparity map corresponding to the previous moment.

[0114] Furthermore, step S310 may include updating the extrinsic parameters of the binocular camera according to the current positioning data of the aircraft and the positioning data of the previous moment to obtain updated extrinsic parameters; and converting the historical disparity map corresponding to the previous moment into the initial disparity map of the current moment according to the updated extrinsic parameters.

[0115] In this embodiment, the aircraft is in motion, and the camera pose at the current moment and the previous moment is obtained through the positioning means of the aircraft. The camera pose includes the corresponding position and attitude of the camera. The position is the three-dimensional coordinate of the camera in space, and the attitude is the orientation of the camera (such as pitch angle, yaw angle, roll angle). According to the pose at the current moment and the pose at the previous moment, the extrinsic parameters are solved to obtain the extrinsic parameters. In this embodiment, the extrinsic parameters can be used to convert the points in the camera coordinate system at the previous moment to the current moment, that is, to convert the historical disparity map corresponding to the previous moment (the target disparity map corresponding to the previous moment) into the initial disparity map at the current moment. Generally speaking, the extrinsic parameters are used to convert the points in the world coordinate system back to the camera coordinate system at the current moment. Obviously, the camera extrinsic parameters are indirectly involved in the conversion between adjacent moments.

[0116] In this embodiment, new extrinsic parameters are determined by the camera poses at two adjacent moments, and the target disparity map corresponding to the previous moment is converted into the initial disparity map corresponding to the current moment. The temporal correlation is maintained and the temporal characteristics are introduced. With the help of the initial disparity map, the temporal characteristics of the scene can be fully utilized, thereby reducing the number of iterations and the time consumption of the ranging model, which can solve the problem that the iterations are too long and cannot meet the real-time ranging requirements.

[0117] It should be noted that when the ranging model is first applied, the initial disparity map can be a given disparity map, and the disparity value of this given disparity map can be zero. In subsequent applications of the ranging model, the initial disparity map at the current moment is converted from the target disparity map corresponding to the previous moment, which can reduce the number of iterations and thus meet the requirements of long-distance real-time continuous ranging.

[0118] S320 , inputting the initial disparity map at the current moment and the binocular image at the current moment into a distance measurement model to obtain a target disparity map corresponding to the current moment.

[0119] The ranging model includes a multi-scale cascade structure. Multiple cascade iterations at different scales can iterate disparity from the lowest resolution to higher resolutions to obtain a globally unified disparity, thereby reducing the flying spot problem caused by the upsampling process and improving the accuracy of disparity estimation. In this embodiment, after obtaining the initial disparity map and the binocular image at the current moment, these initial disparity map and the binocular image at the current moment can be input into the ranging model to obtain the target disparity map corresponding to the current moment. This target disparity map corresponding to the current moment is processed by the ranging model, which also reduces the flying spot problem caused by upsampling the acquired binocular image during dynamic flight of the aircraft, thereby improving the accuracy of disparity estimation.

[0120] Furthermore, step S320 may include performing feature extraction on the binocular image at the current moment to obtain binocular feature maps of different scales; and performing cascade updates of different scales on the initial disparity map at the current moment according to the binocular feature maps of different scales to obtain a target disparity map corresponding to the current moment.

[0121] In this embodiment, performing cascade updates of different scales on the initial disparity map at the current moment may further include: performing multiple disparity updates on the initial disparity map according to the extracted features at the same scale.

[0122] Among them, whether the disparity update is at the same scale or at different scales, the update process is similar. The difference is that at the same scale, the first disparity map obtained by the current disparity update is used as the initial sample disparity map for the next disparity update; at different scales, the first disparity map obtained by the last update of the current scale is upsampled to obtain the same first disparity map as the next level scale, and used as the initial sample disparity map for the first disparity update at the next level scale.

[0123] Furthermore, step S320 may include performing feature extraction on the left and right eye images to obtain multiple frames of left eye feature maps of different scales and multiple frames of right eye feature maps of different scales; performing multiple disparity updates on the initial disparity map at the same scale based on the left eye feature map and the right eye feature map to obtain a disparity map corresponding to each update, and scaling the disparity map obtained by the last update of the current scale as the initial disparity map of the next scale, where the scale of the initial disparity map of the next scale is the same as that of the next scale; scaling each frame of the disparity map at each scale to obtain a corresponding disparity map with the original scale, and outputting the disparity map with the original scale obtained by the last update, which is the target disparity map.

[0124] Furthermore, in the m-th disparity update at the same scale, the spatial correlation of the binocular feature map at the current scale is obtained, where m is a positive integer; based on the spatial correlation, a disparity search is performed on the initial disparity map of the m-th disparity update with a set radius to obtain a correlation index; the correlation index, the left-eye feature map of the current scale, and the initial disparity map of the m-th disparity update are spliced, and the spliced ​​matrix is ​​used as the hidden state; the hidden state is disparity updated using a gated recurrent unit to obtain a disparity increment corresponding to the m-th disparity update; the disparity increment of the m-th disparity update is superimposed on the initial disparity map of the m-th disparity update to obtain the disparity map corresponding to the m-th disparity update of the current scale.

[0125] The embodiments of the present application can be used for continuous ranging. For continuous binocular image data, the initial disparity map used by the ranging model can be converted from the target disparity map corresponding to the previous moment. For non-continuous binocular images, the initial disparity map used by the ranging model can use an initial disparity map with a given disparity value of zero.

[0126] S330 , obtaining a target depth map corresponding to the current moment according to the target disparity map at the current moment.

[0127] Obviously, based on the target depth map, the distance between the obstacle and the aircraft can be obtained. In addition, the target disparity map at the current moment is used to convert it into the initial disparity map at the next moment. This cycle of measurement achieves the purpose of continuous distance measurement.

[0128] As can be seen, the embodiments of the present application fully utilize the disparity update characteristics of the ranging model and the timing characteristics of actual usage scenarios. Using positioning data, the target disparity map output at the previous moment is converted into a coordinate system and then input into the ranging model as the initial disparity map at the current moment. This reduces the number of iterations of the ranging model, shortens the running time of the ranging model, and enables real-time continuous ranging. Furthermore, the multi-scale cascade approach allows disparity updates from the lowest-resolution feature map to higher-resolution feature maps, reducing the flying point problem caused by the upsampling process.

[0129] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a binocular vision-based ranging device and corresponding embodiments.

[0130] Figure 7 It is a structural schematic diagram of the binocular vision-based ranging device shown in this application.

[0131] See also Figure 7 The binocular vision-based ranging device shown in this application is applied to a mobile device equipped with a binocular camera, and includes a disparity conversion module 710, a disparity prediction module 720, and a depth acquisition module 730;

[0132] The disparity conversion module 710 is used to obtain an initial disparity map at the current moment based on the mobile device's previous moment positioning data, current moment positioning data, and the historical disparity map corresponding to the previous moment. The historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model.

[0133] The disparity prediction module 720 is used to input the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment;

[0134] The depth acquisition module 730 is used to obtain the target depth map corresponding to the current moment according to the target disparity map at the current moment;

[0135] Among them, the ranging model is pre-trained by cascading different scales based on the sample binocular images and corresponding disparity map labels and initial sample disparity maps in the training set.

[0136] Furthermore, the disparity conversion module 710 can be used to update the external parameters of the binocular camera based on the current positioning data of the mobile device and the positioning data of the previous moment to obtain updated external parameters; based on the updated external parameters, the historical disparity map corresponding to the previous moment is converted into the initial disparity map of the current moment.

[0137] Furthermore, the disparity prediction module 720 can be used to extract features from the binocular image at the current moment to obtain binocular feature maps of different scales; based on the binocular feature maps of different scales, the initial disparity map at the current moment is cascaded and updated at different scales to obtain the target disparity map corresponding to the current moment.

[0138] Furthermore, the distance measurement model involved in the above-mentioned disparity prediction module 720 is generated according to the following distance measurement model training device:

[0139] A training set acquisition unit is used to acquire a training set, the training set including a sample binocular image and a corresponding disparity map label and an initial sample disparity map, the sample binocular image including a first image and a second image;

[0140] A feature extraction unit is used to extract features from the first image to obtain multiple frames of first feature maps of different scales; and to extract features from the second image to obtain multiple frames of second feature maps of different scales;

[0141] a disparity updating unit configured to perform multiple disparity updates on the initial sample disparity maps of the training set at the same scale based on the first feature map, obtain a first disparity map corresponding to each update, and scale the first disparity map obtained by the last update at the current scale as the initial sample disparity map at the next scale;

[0142] a scale transformation unit, configured to perform scale transformation on each frame of the first disparity map at each scale to obtain a corresponding second disparity map of the original scale;

[0143] The parameter optimization unit is used to calculate the loss based on the disparity map label and all second disparity maps, and optimize the parameters of the ranging model through back propagation until the calculated loss reaches the training stop condition, thereby obtaining the ranging model.

[0144] Furthermore, in the disparity update unit, the first disparity map obtained by the last update of the current scale is scale-transformed and used as the initial sample disparity map of the next scale, including: upsampling the first disparity map obtained by the last update of the current scale to obtain the same first disparity map as the next scale, and using it as the initial sample disparity map for the first disparity update of the next scale.

[0145] Furthermore, in the disparity update unit, the initial sample disparity map of the training set is updated multiple times at the same scale according to the first feature map to obtain the first disparity map corresponding to each update, including: at the same scale, the first disparity map obtained by the current disparity update is used as the initial sample disparity map for the next disparity update.

[0146] Furthermore, in the disparity update unit, the initial sample disparity map of the training set is disparity updated multiple times at the same scale according to the first feature map to obtain the first disparity map corresponding to each update, including obtaining the spatial correlation of the current scale of the first feature map and the second feature map in the k-th disparity update at the same scale, where k is a positive integer; based on the spatial correlation, a disparity search is performed on the initial sample disparity map of the k-th disparity update with a set radius to obtain a correlation index; the correlation index, the first feature map of the current scale and the initial sample disparity map of the k-th disparity update are spliced, and the spliced ​​matrix is ​​used as a hidden state; the hidden state is disparity updated using a gated recurrent unit to obtain a disparity increment corresponding to the k-th disparity update; the disparity increment of the k-th disparity update is superimposed on the initial sample disparity map of the k-th disparity update to obtain the first disparity map corresponding to the k-th disparity update of the current scale.

[0147] Furthermore, the scale transformation unit is used to perform multiple upsampling convolutions on the first feature map of the same scale as the first disparity map to generate a corresponding mask; according to the mask, the first disparity map is scaled in the form of a sliding window and / or interpolation to obtain a second disparity map of the original scale.

[0148] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0149] Figure 8 It is a structural diagram of the mobile device shown in this application.

[0150] See also Figure 8 , the mobile device 800 includes a memory 810 and a processor 820 .

[0151] The processor 820 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0152] Memory 810 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. ROM may store static data or instructions required by processor 820 or other computer modules. Permanent storage may be a readable and writable storage device. Permanent storage may be a non-volatile storage device that maintains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device utilizes a mass storage device (e.g., a magnetic or optical disk, flash memory). In other embodiments, the permanent storage device may be a removable storage device (e.g., a floppy disk, optical drive). System memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory (DRAM). System memory may store some or all instructions and data required by the processor during operation. Furthermore, memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), as well as magnetic disks and / or optical disks. In some embodiments, the memory 810 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0153] The memory 810 stores executable codes. When the executable codes are processed by the processor 820 , the processor 820 may execute part or all of the above-mentioned methods.

[0154] In some optional embodiments, the mobile device may be an aircraft or a flying car, etc.

[0155] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0156] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of a server (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0157] The embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the methods described in the above embodiments.

[0158] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0159] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0160] The embodiments of the present application have been described above. The above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A binocular vision-based ranging method, applied to a mobile device equipped with a binocular camera, characterized in that: include: Obtaining an initial disparity map at the current moment based on the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment; wherein the historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model; Inputting the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment; According to the target disparity map at the current moment, the target depth map corresponding to the current moment is obtained; The ranging model is pre-trained by cascading different scales based on the sample binocular images and corresponding disparity map labels and the initial sample disparity map in the training set.

2. The method according to claim 1, characterized in that The obtaining of the initial disparity map at the current moment according to the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment includes: Updating the extrinsic parameters of the binocular camera according to the current positioning data and the previous positioning data of the mobile device to obtain updated extrinsic parameters; According to the updated external parameters, the historical disparity map corresponding to the previous moment is converted into an initial disparity map at the current moment.

3. The method according to claim 1, characterized in that The step of inputting the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment includes: Perform feature extraction on the binocular image at the current moment to obtain binocular feature maps of different scales; According to the binocular feature maps of different scales, the initial disparity map at the current moment is updated in cascade at different scales to obtain the target disparity map corresponding to the current moment.

4. The method according to claim 1, wherein The distance measurement model is pre-trained in the following way: Acquire a training set, where the training set includes sample binocular images and corresponding disparity map labels and an initial sample disparity map, where the sample binocular images include a first image and a second image; Performing feature extraction on the first image to obtain multiple frames of first feature maps of different scales; Performing feature extraction on the second image to obtain multiple frames of second feature maps at different scales; performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map, obtaining a first disparity map corresponding to each update, and performing a scale transformation on the first disparity map obtained by the last update at the current scale as the initial sample disparity map at the next scale; Performing scale transformation on each frame of the first disparity map at each scale to obtain a corresponding second disparity map with the original scale; The loss is calculated according to the disparity map label and all the second disparity maps, and the parameters of the ranging model are optimized by back propagation until the calculated loss reaches the training stop condition, thereby obtaining the ranging model.

5. The method according to claim 4, characterized in that The scaling of the first disparity map obtained by the last update of the current scale is performed to use the first disparity map as the initial sample disparity map of the next scale, including: The first disparity map obtained by the last update of the current scale is upsampled to obtain the same first disparity map as the next level scale, and is used as the initial sample disparity map for the first disparity update of the next level scale.

6. The method according to claim 4, characterized in that The performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map to obtain a first disparity map corresponding to each update includes: At the same scale, the first disparity map obtained by the current disparity update is used as the initial sample disparity map for the next disparity update.

7. The method according to claim 6, characterized in that The performing multiple disparity updates on the initial sample disparity map of the training set at the same scale according to the first feature map and the second feature map to obtain a first disparity map corresponding to each update includes: In a k-th disparity update at the same scale, obtaining a spatial correlation of the first feature map and the second feature map at the current scale, where k is a positive integer; According to the spatial correlation, a disparity search is performed on the initial sample disparity map of the k-th disparity update with a set radius to obtain a correlation index; splicing the correlation index, the first feature map of the current scale, and the initial sample disparity map of the k-th disparity update, and using the matrix obtained by splicing as a hidden state; Performing a disparity update on the hidden state using a gated recurrent unit to obtain a disparity increment corresponding to the k-th disparity update; The disparity increment of the k-th disparity update is superimposed on the initial sample disparity map of the k-th disparity update to obtain a first disparity map corresponding to the k-th disparity update at the current scale.

8. The method according to claim 4, characterized in that The scaling of each frame of the first disparity map at each scale is performed to obtain a corresponding second disparity map having the original scale, including: Performing multiple upsampling convolutions on a first feature map having the same scale as the first disparity map to generate a corresponding mask; According to the mask, the first disparity map is scaled in a sliding window and / or interpolation form to obtain a second disparity map of the original scale.

9. A binocular vision-based distance measurement device, applied to a mobile device equipped with a binocular camera, characterized in that: Includes disparity conversion module, disparity prediction module and depth acquisition module; The disparity conversion module is used to obtain an initial disparity map at the current moment based on the positioning data of the mobile device at the previous moment, the positioning data at the current moment, and the historical disparity map corresponding to the previous moment; wherein the historical disparity map is obtained by predicting the binocular image at the previous moment acquired by the binocular camera using a pre-trained ranging model; The disparity prediction module is used to input the initial disparity map at the current moment and the binocular image at the current moment into the ranging model to obtain the target disparity map corresponding to the current moment; The depth acquisition module is used to obtain a target depth map corresponding to the current moment according to the target disparity map at the current moment; The ranging model is pre-trained by cascading different scales based on the sample binocular images and corresponding disparity map labels and the initial sample disparity map in the training set.

10. A mobile device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 8.

11. The mobile device according to claim 10, wherein: The mobile device is an aircraft or a flying car.

12. A computer-readable storage medium, characterized in that Executable codes are stored thereon, and when the executable codes are executed by a processor of a vehicle, the processor is caused to perform the method according to any one of claims 1 to 8.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which implements the method according to any one of claims 1 to 8 when the computer program or the instruction is executed by a processor.

Images