Depth estimation method, device and equipment based on binocular camera

Abstract

The present disclosure relates to the technical field of image recognition, and provides a depth estimation method, device and equipment based on a binocular camera, which comprises the following steps: inputting a first image obtained by a main camera into a first light neural network to obtain a first feature vector, and inputting the first feature vector into a depth estimation network to obtain an initial depth estimation result; inputting a second image obtained by an auxiliary camera into a second light neural network to obtain a second feature vector; inputting the first feature vector, the second feature vector and a predetermined camera parameter into a cost calculation network to obtain a cost vector; inputting the first image into a third light neural network to obtain a third feature vector; and inputting the initial depth estimation result, the cost vector and the third feature vector into an iterative prediction network to obtain a target depth estimation result. The technical scheme provided by one or more embodiments of the present disclosure can stably provide a high-precision absolute depth estimation result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image recognition, in particular to a depth estimation method, device and equipment based on a binocular camera. BACKGROUND

[0002] Depth estimation is an important technology often needed in the fields of AR interaction, robot obstacle avoidance, three-dimensional reconstruction, etc., which aims to assign a real physical distance value to each pixel in an image. Only with an accurate distance, can a virtual object be placed steadily on a desktop, a mechanical arm be accurately grasped, or multiple frames of images be spliced into a three-dimensional model with consistent scales. The depth corresponding to the real world meter unit is called absolute depth in the industry, which is an indispensable rigid requirement for measurement, control and mapping tasks.

[0003] At present, depth estimation technology based on monocular cameras is more popular, and the core reason is that the hardware threshold is low. A mobile phone or a vehicle-mounted camera can be implemented without additional sensors. Due to the low deployment cost, manufacturers are willing to use it as the "default option" to solve the "yes or no" problem first. However, the depth estimation technology based on monocular cameras can only rely on the texture, perspective and semantic prior of the image itself to "guess" the depth, and there is a lack of observable fixed mapping between the output result and the meter scale of the real world, resulting in an absolute depth error often exceeding 10%. More seriously, the same picture may be interpreted into completely different depth ranges in different scenes, making the values across scenes and devices not directly comparable or used for measurement.

[0004] Therefore, when the task is upgraded from "visual effect" to "measurement accuracy", the depth estimation scheme based on monocular cameras is often forced to introduce additional calibration objects or post-processing steps, but it is still difficult to provide repeatable and traceable absolute depth. SUMMARY

[0005] Therefore, the present disclosure one or more embodiments provides a depth estimation method, device and equipment based on a binocular camera, which can stably provide high-precision absolute depth estimation results.

[0006] In a first aspect, the disclosure provides a depth estimation method based on a binocular camera, the method comprising: inputting a first image obtained by a main camera into a first lightweight neural network to perform instance feature extraction to obtain a first feature vector, and inputting the first feature vector into a depth estimation network to obtain an initial depth estimation result; inputting a second image obtained by an auxiliary camera into a second lightweight neural network to perform instance feature extraction to obtain a second feature vector; inputting the first feature vector, the second feature vector, and a predetermined camera parameter into a cost calculation network to obtain a cost vector, the cost vector being used to represent a matching degree of the first feature vector and the second feature vector; inputting the first image into a third lightweight neural network to perform context feature extraction to obtain a third feature vector; and inputting the initial depth estimation result, the cost vector, and the third feature vector into an iterative prediction network to obtain a target depth estimation result.

[0007] In a second aspect, the disclosure provides a depth estimation device based on a binocular camera, the device comprising: a first image processing unit configured to input a first image obtained by a main camera into a first lightweight neural network to perform instance feature extraction to obtain a first feature vector, and input the first feature vector into a depth estimation network to obtain an initial depth estimation result; a second image processing unit configured to input a second image obtained by an auxiliary camera into a second lightweight neural network to perform instance feature extraction to obtain a second feature vector; an image matching unit configured to input the first feature vector, the second feature vector, and a predetermined camera parameter into a cost calculation network to obtain a cost vector, the cost vector being used to represent a matching degree of the first feature vector and the second feature vector; a third image processing unit configured to input the first image into a third lightweight neural network to perform context feature extraction to obtain a third feature vector; and a depth prediction unit configured to input the initial depth estimation result, the cost vector, and the third feature vector into an iterative prediction network to obtain a target depth estimation result.

[0008] In a third aspect, the disclosure provides an electronic device, the electronic device comprising a memory and a processor, the memory being configured to store a computer program, the computer program being configured to be executed by the processor to implement the depth estimation method based on the binocular camera.

[0009] In a fourth aspect, the disclosure provides a computer readable storage medium, the computer readable storage medium being configured to store a computer program, the computer program being configured to be executed by a processor to implement the depth estimation method based on the binocular camera.

[0010] This disclosure provides a technical solution through one or more embodiments. First, it uses two lightweight neural networks to extract instance-level features from the left and right images captured by two cameras. This preserves key information such as texture, edges, and semantics of each image while minimizing subsequent computational load. Then, the two sets of feature vectors, along with camera intrinsic and extrinsic parameters, are fed into a cost calculation network. Under geometric constraints, the matching metric between feature vectors is quantified into a cost vector. This preserves the disparity information of the left and right cameras while avoiding the computational explosion problem caused by pixel-by-pixel image matching. Next, contextual features are extracted from the first image obtained from the main camera, which can compensate for global semantic cues that may be lost in weak texture or occluded areas. Finally, the initial depth estimation result, cost vector, and contextual feature vector are fed into an iterative prediction network. This allows for continuous residual correction, achieving iterative optimization and gradually converging the target depth estimation result to the ideal accuracy range.

[0011] This disclosure provides a technical solution with one or more embodiments, where each module has a clear division of labor. The first and second lightweight neural networks are responsible for "seeing image information clearly," the cost calculation network is responsible for "aligning image information," the third lightweight neural network is responsible for "completing image information," and the iterative network is responsible for "refining image information." With the combined action of these multiple methods, more accurate absolute depth information can be obtained.

[0012] This disclosure provides a technical solution with one or more embodiments that explicitly incorporates the geometric relationship between the two cameras into the cost vector. This allows the overall network to move beyond guessing depth from a single image in a black-box manner, instead utilizing parallax stereo verification like the human eye. Consequently, in scenes with fixed baselines and known camera calibration, it can achieve an absolute depth error far lower than that of traditional monocular methods. Simultaneously, all heavy computations are broken down into lightweight convolutional and iterable modules, reducing equipment requirements and making the solution both highly accurate and feasible for practical application. Attached Figure Description

[0013] The features and advantages of the embodiments of this disclosure will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the present disclosure in any way. In the drawings:

[0014] Figure 1 A schematic diagram illustrating the steps of a depth estimation method based on a binocular camera in one embodiment of this disclosure is shown.

[0015] Figure 2 A flowchart illustrating a depth estimation method based on a binocular camera in one embodiment of this disclosure is shown.

[0016] Figure 3 A schematic diagram of the functional units of a depth estimation device based on a binocular camera is shown in one embodiment of this disclosure;

[0017] Figure 4 A schematic diagram of the structure of an electronic device according to one embodiment of the present disclosure is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] Please see Figure 1 The present disclosure provides a depth estimation method based on a binocular camera, which may include the following steps.

[0020] S1: Input the first image obtained by the main camera into the first lightweight neural network to extract instance features, obtain the first feature vector, and input the first feature vector into the depth estimation network to obtain the initial depth estimation result.

[0021] In this embodiment, the first lightweight neural network will perform instance-level feature extraction on the first image captured by the main camera, such as extracting key information such as texture, edges, and semantics of the first image.

[0022] In this embodiment, the depth estimation network is not specifically limited and can refer to various depth estimation networks in monocular depth estimation technology, as long as the initial depth estimation result can be obtained for subsequent calculations.

[0023] In some implementations, the depth estimation network preferably includes: a feature processing submodule, used to receive a first feature vector and perform at least one of channel compression, spatial alignment and scale fusion on the first feature vector to generate a depth estimation fusion feature; an inverse depth regression submodule, used to generate an inverse depth feature map based on the depth estimation fusion feature; and a depth transformation submodule, used to convert the inverse depth feature map into an initial depth estimation result based on a preset transformation function.

[0024] Specifically, the feature processing submodule in the depth estimation network module can receive feature maps from the backbone network and perform channel compression, spatial alignment, or scale fusion on these feature maps to generate the fused features required for depth prediction. The inverse depth regression submodule in the depth estimation network module, connected after the feature processing submodule, can generate an inverse depth representation based on the fused features. The inverse depth regression submodule can include at least one sequence convolution computation unit and at least one prediction unit, which can output an inverse depth feature map for at least one channel. The depth transformation submodule in the depth estimation network module can convert the inverse depth feature map into actual depth or maintain it in inverse depth format based on a preset function for use in subsequent tasks.

[0025] S2: Input the second image obtained by the auxiliary camera into the second lightweight neural network to extract instance features and obtain the second feature vector.

[0026] In this embodiment, the second lightweight neural network will perform instance-level feature extraction on the second image captured by the auxiliary camera, such as extracting key information such as texture, edges, and semantics of the second image.

[0027] In this embodiment, the first lightweight neural network and the second lightweight neural network not only have the same network structure (such as the target network structure) but also the same network weights. Thus, the difference between the first feature vector and the second feature vector is mainly due to the parallax information from the left and right cameras.

[0028] In some implementations, both the first lightweight neural network and the second lightweight neural network are target network structures used to perform the target task of "instance feature extraction". The target network structure may specifically include: a feature extraction front-end submodule for performing shallow feature extraction on the input image to obtain an initial feature map; a multi-level residual feature extraction submodule for performing deep feature extraction and feature fusion on the initial feature map to obtain a two-dimensional feature map; a global feature aggregation submodule for performing spatial dimension aggregation operations on the two-dimensional feature map to obtain a target feature vector; and an output submodule for generating an output result corresponding to the target task based on the target feature vector.

[0029] Specifically, the feature extraction front-end module in the target network structure can perform initial feature extraction on the input image. The feature extraction front-end module can include convolutional layers and pooling layers, which are used to perform convolution operations on the input image with a preset stride to obtain an initial feature map, and to perform downsampling processing on the initial feature map, respectively.

[0030] The multi-level residual feature extraction submodule in the target network structure is connected after the feature extraction front-end submodule. The multi-level residual feature extraction submodule comprises multiple (e.g., four) sequentially connected feature extraction levels, with the number of output channels increasing progressively for each level. Each feature extraction level includes at least two sequentially connected convolution-normalization-activation structures and at least one bypass connection structure. The convolution-normalization-activation structure performs downsampling between adjacent feature extraction levels, and the bypass connection structure fuses the input features of its respective feature extraction level with the output of the corresponding convolution-normalization-activation structure.

[0031] The global feature aggregation submodule in the target network structure performs spatial dimension aggregation operations on the two-dimensional feature map output from the last feature extraction stage to obtain the target feature vector. The output submodule in the target network structure generates the output result of the target task based on the target feature vector.

[0032] S3: Input the first feature vector, the second feature vector, and the predetermined camera parameters into the cost calculation network to obtain the cost vector, which is used to characterize the matching degree between the first feature vector and the second feature vector.

[0033] In this embodiment, the camera parameters include camera intrinsic and extrinsic parameters. By feeding two sets of feature vectors along with the camera intrinsic and extrinsic parameters into the cost calculation network, the matching degree between the feature vectors can be quantified into a cost vector under geometric constraints. This preserves the parallax information of the left and right cameras while avoiding the drastic increase in computational load caused by pixel-by-pixel image matching.

[0034] In some implementations, the cost calculation network includes: a feature alignment submodule, configured to determine the disparity range based on the camera parameters, and within the disparity range, align the first feature vector with the second feature vector along a specified direction and with different displacements to generate multiple sets of left and right feature pairs; a cost construction submodule, configured to construct a multi-disparity dimension cost representation based on the aligned left and right feature pairs; and a cost stacking submodule, configured to stack the cost values ​​obtained under each disparity displacement along the disparity dimension to form the cost vector.

[0035] Specifically, the feature alignment submodule in the cost calculation network can align a graph feature vector (i.e., the first feature vector) with another feature graph vector (i.e., the second feature vector) along a specified direction with different displacements, based on a preset disparity range.

[0036] The cost construction submodule in the cost computation network can construct a multi-disparity dimension cost representation based on aligned left and right feature pairs. This submodule can include computational units for performing similarity or disparity calculations, which can generate a corresponding cost value for each preset disparity displacement.

[0037] The cost stacking submodule in the cost computation network stacks the cost values ​​obtained under each disparity shift along the disparity dimension to form a two-dimensional cost volume. The two-dimensional cost volume is a two-dimensional structure (typically height × disparity number) obtained by stacking along the disparity dimension at each pixel location. It represents the matching cost value of each pixel under different disparities. The cost vector is a local cost feature extracted from this two-dimensional cost volume for subsequent matching or regression.

[0038] S4: Input the first image into the third lightweight neural network to extract contextual features and obtain the third feature vector.

[0039] In this embodiment, contextual features are extracted from the first image obtained from the main camera, which can compensate for global semantic cues that may be lost in the cost vector in weak texture and occluded areas, providing richer detailed information for the subsequent iterative optimization process and improving the accuracy of depth estimation.

[0040] In this embodiment, the third lightweight neural network can adopt the same target network structure as the first lightweight neural network, but the target task needs to be set as "contextual feature extraction". Clearly, the network weights of the third lightweight neural network and the first lightweight neural network are different.

[0041] S5: Input the initial depth estimation result, the cost vector, and the third feature vector into the iterative prediction network to obtain the target depth estimation result.

[0042] In this embodiment, the initial depth estimation result, cost vector, and context feature vector (i.e., the third feature vector) are fed into the iterative prediction network together, which can continuously perform residual correction to achieve iterative optimization and thus gradually converge the target depth estimation result to the ideal accuracy range.

[0043] In some implementations, the iterative prediction network includes: a cyclic update submodule, used to acquire the current input features and the previous hidden state, and determine the current hidden state based on the current input features and the previous hidden state; a depth increment submodule, used to determine the current depth increment based on the current hidden state; and a depth prediction submodule, used to fuse the current depth increment with the previous depth estimation result, update the previous depth estimation result, and obtain the current depth estimation result; wherein the current input features include the previous depth estimation result, the cost vector, and the third feature vector, the previous hidden state or the current hidden state is used to characterize the network optimization state of the iterative prediction network, and the initial value of the previous depth estimation result is the initial depth estimation result.

[0044] In some implementations, the cyclic update submodule includes: an update gate for determining an update ratio, the update ratio representing how much information in the previous hidden state is retained in the current hidden state; and a reset gate for determining a reset ratio, the reset ratio representing the degree of suppression of information unrelated to the current input feature in the previous hidden state.

[0045] Specifically, the recurrent update submodule is the core of the iterative prediction network (similar to GRU, gated recurrent unit), used to update the hidden state, thereby capturing and fusing current input features with historical information. It controls the flow of information through update and reset gates, generating a hidden state containing contextual and temporal information, providing the foundation for depth increment prediction. The input to the recurrent update submodule includes the input features of the current iteration (typically including the third feature vector, cost vector, and current depth estimate) and the hidden state from the previous time step (in the first iteration, the hidden state may be initialized to zero or an initial state). The output of the recurrent update submodule is the updated hidden state, which is then passed to the depth increment submodule and the recurrent update submodule for the next iteration.

[0046] The depth increment submodule can predict the depth increment (i.e., the adjustment amount of the depth estimate) based on the hidden state output by the iterative update submodule. The depth increment represents the depth change for each pixel and is used to fine-tune the current depth estimate.

[0047] The depth prediction submodule fuses the depth increment output by the depth increment submodule with the depth estimate from the previous iteration to obtain an updated depth estimate. This fusion is typically achieved through addition or weighted operations, progressively optimizing the depth map.

[0048] In each iteration, the loop update submodule first receives the input features and the previous hidden state, and outputs the new hidden state. The depth increment submodule uses the new hidden state to predict the depth increment. The depth prediction submodule fuses the depth increment with the previous depth estimate and outputs the new depth estimate. The new depth estimate and the hidden state are passed to the next iteration, and this process is repeated until convergence. The entire process ensures the gradual optimization of the depth estimate, where the hidden state carries historical information, and the depth estimate is continuously refined through incremental adjustments.

[0049] Please see Figure 2 The depth estimation method based on a binocular camera provided in one embodiment of this disclosure can be implemented by the following process.

[0050] Step 1: Input the first image obtained by the main camera into the first lightweight neural network resnet18-2lf, obtain the first feature vector V1 output by the network, and input V1 into the depth estimation network H1 to obtain the initial depth estimation result D1.

[0051] Step 2: Input the second image obtained by the auxiliary camera into the second lightweight neural network resnet18-2lf, and obtain the second feature vector V2 output by the network.

[0052] Step 3: Input V1, V2, camera intrinsic parameter K1, and camera extrinsic parameter P1 into the cost calculation network COS1 to obtain the cost vector CSV1.

[0053] Step 4: Input the first image obtained by the main camera into the third lightweight neural network resnet18-2lc to obtain the third feature vector C1 output by the network.

[0054] Step 5: Input D1, CSV1, and C1 into the iterative prediction network UDD1 to obtain the final target depth estimation result D2.

[0055] In this embodiment, the lightweight neural networks resnet18-2lf and resnet18-2lc are used to extract features. The characteristic of these networks is that they have fewer parameters than typical networks. Lightweighting means that the network has fewer parameters. resnet18-2lf and resnet18-2lc have the same network structure, but the network outputs are used in two different places.

[0056] In this embodiment, the camera intrinsic parameter K1 and the camera extrinsic parameter P1 are shared by both cameras.

[0057] In this embodiment, the specific meaning of the cost vector CSV1 is the output of the cost calculation network COS1, which represents the matching result vector obtained by matching two feature vectors V1 and V2.

[0058] In this embodiment, the depth estimation network H1 predicts an initial depth estimation result based on the feature vector of the input image. The iterative prediction network UDD1 iteratively optimizes the initial depth prediction result and finally outputs the optimized target depth prediction result. Optionally, the iterative prediction network UDD1 uses a recurrent neural network to iteratively optimize the depth prediction result.

[0059] This disclosure provides a technical solution through one or more embodiments. First, it uses two lightweight neural networks to extract instance-level features from the left and right images captured by two cameras. This preserves key information such as texture, edges, and semantics of each image while minimizing subsequent computational load. Then, the two sets of feature vectors, along with camera intrinsic and extrinsic parameters, are fed into a cost calculation network. Under geometric constraints, the matching metric between feature vectors is quantified into a cost vector. This preserves the disparity information of the left and right cameras while avoiding the computational explosion problem caused by pixel-by-pixel image matching. Next, contextual features are extracted from the first image obtained from the main camera, which can compensate for global semantic cues that may be lost in weak texture or occluded areas. Finally, the initial depth estimation result, cost vector, and contextual feature vector are fed into an iterative prediction network. This allows for continuous residual correction, achieving iterative optimization and gradually converging the target depth estimation result to the ideal accuracy range.

[0060] This disclosure provides a technical solution with one or more embodiments, where each module has a clear division of labor. The first and second lightweight neural networks are responsible for "seeing image information clearly," the cost calculation network is responsible for "aligning image information," the third lightweight neural network is responsible for "completing image information," and the iterative network is responsible for "refining image information." With the combined action of these multiple methods, more accurate absolute depth information can be obtained.

[0061] This disclosure provides a technical solution with one or more embodiments that explicitly incorporates the geometric relationship between the two cameras into the cost vector. This allows the overall network to move beyond guessing depth from a single image in a black-box manner, instead utilizing parallax stereo verification like the human eye. Consequently, in scenes with fixed baselines and known camera calibration, it can achieve an absolute depth error far lower than that of traditional monocular methods. Simultaneously, all heavy computations are broken down into lightweight convolutional and iterable modules, reducing equipment requirements and making the solution both highly accurate and feasible for practical application.

[0062] Please see Figure 3 This disclosure also provides a depth estimation device based on a binocular camera, the device comprising:

[0063] The first image processing unit 100 is used to input the first image obtained by the main camera into the first lightweight neural network, perform instance feature extraction to obtain a first feature vector, and input the first feature vector into the depth estimation network to obtain an initial depth estimation result.

[0064] The second image processing unit 200 is used to input the second image obtained by the auxiliary camera into the second lightweight neural network, perform instance feature extraction, and obtain the second feature vector.

[0065] Image matching unit 300 is used to input the first feature vector, the second feature vector and predetermined camera parameters into the cost calculation network to obtain a cost vector, wherein the cost vector is used to characterize the matching degree between the first feature vector and the second feature vector;

[0066] The third image processing unit 400 is used to input the first image into a third lightweight neural network to extract contextual features and obtain a third feature vector.

[0067] The depth prediction unit 500 is used to input the initial depth estimation result, the cost vector, and the third feature vector into the iterative prediction network to obtain the target depth estimation result.

[0068] The various units described in this embodiment can be implemented by a computer chip or by a product with a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0069] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0070] In this embodiment, the specific explanations of the first lightweight neural network, the second lightweight neural network, the third lightweight neural network, the depth estimation network, the cost calculation network, and the iterative prediction network can be found in the corresponding parts of the method embodiments above, and will not be repeated here.

[0071] Please see Figure 4 This disclosure also provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program, which, when executed by the processor, implements the above-described depth estimation method based on a binocular camera.

[0072] This disclosure also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described depth estimation method based on a binocular camera.

[0073] The processor can be a central processing unit (CPU). It can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0074] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above-described embodiments.

[0075] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0077] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of apparatus, devices, and storage media are basically similar to method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0078] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0079] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A depth estimation method based on a binocular camera, characterized in that, The method includes: The first image obtained by the main camera is input into the first lightweight neural network to extract instance features and obtain the first feature vector. The first feature vector is then input into the depth estimation network to obtain the initial depth estimation result. The second image obtained by the auxiliary camera is input into the second lightweight neural network to extract instance features and obtain the second feature vector; The first feature vector, the second feature vector, and the predetermined camera parameters are input into the cost calculation network to obtain the cost vector, which is used to characterize the matching degree between the first feature vector and the second feature vector. The first image is input into a third lightweight neural network to extract contextual features and obtain a third feature vector. The initial depth estimation result, the cost vector, and the third feature vector are input into the iterative prediction network to obtain the target depth estimation result. The first lightweight neural network, the second lightweight neural network, and the third lightweight neural network all have the target network structure. The network weights of the first lightweight neural network and the second lightweight neural network are the same, while the network weights of the first lightweight neural network and the third lightweight neural network are different. The target network structure is used to perform a target task. The target network structure includes: a feature extraction front-end submodule, used to perform shallow feature extraction on the input image to obtain an initial feature map; a multi-level residual feature extraction submodule, used to perform deep feature extraction and feature fusion on the initial feature map to obtain a two-dimensional feature map; a global feature aggregation submodule, used to perform spatial dimension aggregation operations on the two-dimensional feature map to obtain a target feature vector; and an output submodule, used to generate an output result corresponding to the target task based on the target feature vector.

2. The method according to claim 1, characterized in that, The depth estimation network includes: The feature processing submodule is used to receive the first feature vector and perform at least one of channel compression, spatial alignment and scale fusion on the first feature vector to generate depth estimation fusion features; The inverse depth regression submodule is used to generate an inverse depth feature map based on the depth estimation fusion features. The depth transformation submodule is used to convert the inverse depth feature map into the initial depth estimation result based on a preset transformation function.

3. The method according to claim 1, characterized in that, The cost calculation network includes: The feature alignment submodule is used to determine the disparity range based on the camera parameters, and within the disparity range, align the first feature vector with the second feature vector along a specified direction and with different displacements to generate multiple sets of left and right feature pairs; The cost construction submodule is used to construct a cost representation with multiple disparity dimensions based on the aligned left and right feature pairs; The cost stacking submodule is used to stack the cost values ​​obtained under each disparity displacement along the disparity dimension to form the cost vector.

4. The method according to claim 1, characterized in that, The iterative prediction network includes: The loop update submodule is used to obtain the current input features and the previous hidden state, and determine the current hidden state based on the current input features and the previous hidden state; The depth increment submodule is used to determine the current depth increment based on the current hiding state; The depth prediction submodule is used to fuse the current depth increment with the previous depth estimation result, update the previous depth estimation result, and obtain the current depth estimation result; The current input features include the preceding depth estimation result, the cost vector, and the third feature vector. The preceding hidden state or the current hidden state is used to characterize the network optimization state of the iterative prediction network. The initial value of the preceding depth estimation result is the initial depth estimation result.

5. The method according to claim 4, characterized in that, The cyclic update submodule includes: An update gate is used to determine the update ratio, which represents how much information from the previous hidden state is retained in the current hidden state; A reset gate is used to determine a reset ratio, which represents the degree of suppression of information unrelated to the current input features in the preceding hidden state.

6. A depth estimation device based on a binocular camera, characterized in that, The device includes: The first image processing unit is used to input the first image obtained by the main camera into the first lightweight neural network, perform instance feature extraction to obtain a first feature vector, and input the first feature vector into the depth estimation network to obtain an initial depth estimation result; The second image processing unit is used to input the second image obtained by the auxiliary camera into the second lightweight neural network, perform instance feature extraction, and obtain the second feature vector. An image matching unit is used to input the first feature vector, the second feature vector, and predetermined camera parameters into a cost calculation network to obtain a cost vector, wherein the cost vector is used to characterize the matching degree between the first feature vector and the second feature vector. The third image processing unit is used to input the first image into the third lightweight neural network, perform context feature extraction, and obtain the third feature vector. The depth prediction unit is used to input the initial depth estimation result, the cost vector, and the third feature vector into the iterative prediction network to obtain the target depth estimation result; The first lightweight neural network, the second lightweight neural network, and the third lightweight neural network all have the target network structure. The network weights of the first lightweight neural network and the second lightweight neural network are the same, while the network weights of the first lightweight neural network and the third lightweight neural network are different. The target network structure is used to perform a target task. The target network structure includes: a feature extraction front-end submodule, used to perform shallow feature extraction on the input image to obtain an initial feature map; a multi-level residual feature extraction submodule, used to perform deep feature extraction and feature fusion on the initial feature map to obtain a two-dimensional feature map; a global feature aggregation submodule, used to perform spatial dimension aggregation operations on the two-dimensional feature map to obtain a target feature vector; and an output submodule, used to generate an output result corresponding to the target task based on the target feature vector.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images