Depth estimation method, training method of depth estimation model, and device and system thereof

Abstract

This application provides a depth estimation method, a training method for a depth estimation model, and related apparatus and system. The method includes: acquiring a first color image; inputting the first color image into a first depth estimation model to obtain a first intermediate depth image; and inputting the first color image and the first intermediate depth image into a second depth estimation model to obtain a first target depth image. The second depth estimation model is trained based on the color image and its corresponding target depth image, and the first depth estimation model is trained based on the color image and its corresponding intermediate depth image. According to the technical solution of this application, obtaining the intermediate depth image corresponding to the color image from the color image, and obtaining the target depth image corresponding to the color image based on the color image and its corresponding intermediate depth image, can fully utilize the prior information of the color image, which helps to improve the accuracy of the target depth estimation image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a depth estimation method, a method for training a depth estimation model, and an apparatus and system thereof. Background Technology

[0002] Currently, unsupervised depth estimation algorithms are among the mainstream algorithms for intelligent driving (ID) and artificial intelligence (AI). These algorithms can use neural network models to estimate depth information from images captured by color cameras and are widely used because they do not require labeled training sets.

[0003] However, current unsupervised depth estimation algorithms do not make good use of the prior information of color images, resulting in poor quality of the estimated depth images. Furthermore, although current unsupervised depth estimation algorithms have eliminated the limitation of training labels to some extent, thus limiting training costs, once the algorithm model is deployed, encountering scenarios not encountered during the training phase can also lead to poor quality of the estimated depth images.

[0004] Therefore, improving the quality of depth estimation images is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a depth estimation method, a training method for a depth estimation model, and a device and system thereof, which helps to improve the quality of depth-estimated images.

[0006] In a first aspect, a depth estimation method is provided, comprising: acquiring a first color image; inputting the first color image into a first depth estimation model to obtain a first intermediate depth image; inputting the first color image and the first intermediate depth image into a second depth estimation model to obtain a first target depth image, wherein the second depth estimation model is trained based on the color image and the target depth image corresponding to the color image, and the first depth estimation model is trained based on the color image and the intermediate depth image corresponding to the color image.

[0007] According to the technical solution of this application, an intermediate depth image corresponding to the color image is obtained through the color image, and a target depth image corresponding to the color image is obtained based on the color image and the intermediate depth image corresponding to the color image. This can make full use of the prior information of the color image and help improve the accuracy of the target depth estimation image.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the above-mentioned inputting the first color image and the first intermediate depth image into the second depth estimation model to obtain the first target depth image includes: obtaining a common edge image based on the first color image and the first intermediate depth image, the common edge image being used to represent the common edge portion of the first color image and the first intermediate depth image; and obtaining the first target depth image based on the common edge image and the first intermediate depth image.

[0009] According to the technical solution of this application, a common edge image is obtained through a color image and an intermediate depth image corresponding to the color image, and a target depth image is obtained based on the common edge image and the intermediate depth image corresponding to the color image. This can make full use of the prior information of the color image to obtain the common edge image, and effectively utilize the favorable common edge parts in the color image and the intermediate depth image corresponding to the color image, thereby helping to improve the accuracy of the target depth estimation image.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, obtaining a common edge image based on the first color image and the first intermediate depth image includes: stitching the first color image and the first intermediate depth image together to obtain an input matrix; performing dimensionality reduction processing on the input matrix to obtain a first matrix and a second matrix; and obtaining a common edge image based on the first matrix and the second matrix.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the above-mentioned inputting the first color image into the first depth estimation model to obtain the first intermediate depth image includes: obtaining the depth feature vector of the first color image based on the first color image, including a first depth feature vector and a second depth feature vector; obtaining a first target feature vector based on the first depth feature vector and a first weight value; obtaining a second target feature vector based on the second depth feature vector and the second weight value; and obtaining the first intermediate depth image based on the first target feature vector and the second target feature vector.

[0012] According to the technical solution of this application, a target vector is obtained based on the depth feature vector and the weight value corresponding to the depth feature vector, thereby obtaining an intermediate depth image, which helps to improve the accuracy of the depth image.

[0013] Optionally, the above color image may include a left view or a right view.

[0014] Optionally, the aforementioned color images are obtained using one or more cameras located on the terminal device.

[0015] Optionally, the camera mentioned above is a stereo camera.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the above method further includes: calculating gradient information of a first depth estimation model and a second depth estimation model based on a first color image and a first target depth image; sending the gradient information to a first device, wherein the gradient information is used by the first device to train and obtain an updated first depth estimation model and an updated second depth estimation model.

[0017] According to the technical solution of this application, gradient information is calculated based on a color image and a target depth image corresponding to the color image, and the gradient information is sent to a first device for the first device to train and update the depth estimation model. This can solve the problem of difficulty in upgrading the model after deployment and help improve the accuracy of the model.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the above method also includes:

[0019] Receive updated first depth estimation model and updated second depth estimation model from the first device.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the above method also includes:

[0021] The updated first depth estimation model and the updated second depth estimation model are obtained by training based on gradient information.

[0022] According to the technical solution of this application, gradient information is calculated based on a color image and a target depth image corresponding to the color image, and the depth estimation model is trained and updated based on the gradient information. This can solve the problem of difficulty in upgrading the model after deployment and help improve the accuracy of the model.

[0023] Secondly, a method for training a depth estimation model is provided, characterized by comprising: processing a color image using a first depth estimation model to obtain an intermediate depth image of the color image; processing the color image and the intermediate depth image of the color image using a second depth estimation model to obtain a target depth image of the color image; training the first depth estimation model based on the color image and the intermediate depth image of the color image to obtain an updated first depth estimation model; and training the second depth estimation model based on the color image and the target depth image of the color image to obtain an updated second depth estimation model.

[0024] According to the technical solution of this application, a first depth estimation model is obtained by training a color image, and a second depth estimation model is obtained by training a color image and an intermediate depth image corresponding to the color image. The trained depth estimation model can make full use of the prior information of the color image, which helps to improve the accuracy of the target depth estimation image.

[0025] In conjunction with the second aspect, in some implementations of the second aspect, the updated second depth estimation model is specifically used to obtain a common edge image based on the first color image and the first intermediate depth image, the common edge image being used to represent the common edge portion of the first color image and the first intermediate depth image; and to obtain a first target depth image based on the common edge image and the first intermediate depth image.

[0026] In conjunction with the second aspect, in some implementations of the second aspect, obtaining a common edge image based on the first color image and the first intermediate depth image includes: concatenating the first color image and the first intermediate depth image to obtain an input matrix; performing dimensionality reduction processing on the input matrix to obtain a first matrix and a second matrix; and obtaining a common edge image based on the first matrix and the second matrix.

[0027] In conjunction with the second aspect, in some implementations of the second aspect, the updated first depth estimation model is specifically used to obtain the depth feature vector of the first color image based on the first color image, including a first depth feature vector and a second depth feature vector; obtain a first target feature vector based on the first depth feature vector and a first weight value; obtain a second target feature vector based on the second depth feature vector and the second weight value; and obtain a first intermediate depth image based on the first target feature vector and the second target feature vector.

[0028] Optionally, the above color image may include a left view or a right view.

[0029] Optionally, the aforementioned color images are obtained using one or more cameras located on the terminal device.

[0030] Optionally, the camera mentioned above is a stereo camera.

[0031] In conjunction with the second aspect, in some implementations of the second aspect, the above training method further includes: receiving gradient information of the first depth estimation model and the second depth estimation model, wherein the gradient information is calculated by the second device based on the first color image and the first target depth image; and training an updated first depth estimation model and an updated second depth estimation model based on the gradient information.

[0032] According to the technical solution of this application, the first device can train and update the depth estimation model based on the gradient information calculated by the terminal device based on the color image and the target depth image corresponding to the color image. This can solve the problem of difficulty in upgrading the model after deployment and help improve the accuracy of the model.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the above training method further includes: sending an updated first depth estimation model and an updated second depth estimation model.

[0034] Thirdly, a depth estimation apparatus is provided, comprising: an acquisition unit for acquiring a first color image; and a processing unit for inputting the first color image into a first depth estimation model to obtain a first intermediate depth image, and inputting the first color image and the first intermediate depth image into a second depth estimation model to obtain a first target depth image, wherein the second depth estimation model is trained based on the color image and the target depth image corresponding to the color image, and the first depth estimation model is trained based on the color image and the intermediate depth image corresponding to the color image.

[0035] In conjunction with the third aspect, in some implementations of the third aspect, the aforementioned processing unit is specifically used to obtain a common edge image based on the first color image and the first intermediate depth image, wherein the common edge image is used to represent the common edge portion of the first color image and the first intermediate depth image, and to obtain a first target depth image based on the common edge image and the first intermediate depth image.

[0036] In conjunction with the third aspect, in some implementations of the third aspect, the aforementioned processing unit is specifically used to stitch together the first color image and the first intermediate depth image to obtain an input matrix, perform dimensionality reduction processing on the input matrix to obtain a first matrix and a second matrix, and obtain a common edge image based on the first matrix and the second matrix.

[0037] In conjunction with the third aspect, in some implementations of the third aspect, the aforementioned processing unit is specifically used to obtain a depth feature vector of the first color image based on the first color image, including a first depth feature vector and a second depth feature vector; obtain a first target feature vector based on the first depth feature vector and a first weight value; obtain a second target feature vector based on the second depth feature vector and the second weight value; and obtain a first intermediate depth image based on the first target feature vector and the second target feature vector.

[0038] Optionally, the above color image may include a left view or a right view.

[0039] Optionally, the aforementioned color images are obtained using one or more cameras located on the terminal device.

[0040] Optionally, the camera mentioned above is a stereo camera.

[0041] In conjunction with the third aspect, in some implementations of the third aspect, the processing unit is further configured to calculate gradient information of the first depth estimation model and the second depth estimation model based on the first color image and the first target depth image; the apparatus further includes a sending unit configured to send the gradient information to the first device, the gradient information being used by the first device to train and obtain an updated first depth estimation model and an updated second depth estimation model.

[0042] In conjunction with the third aspect, in some implementations of the third aspect, the acquisition unit is also used to receive an updated first depth estimation model and an updated second depth estimation model from the first device.

[0043] Fourthly, a training apparatus for a depth estimation model is provided, comprising: an acquisition unit, configured to process a color image using a first depth estimation model to acquire an intermediate depth image of the color image, and to process the color image and the intermediate depth image of the color image using a second depth estimation model to acquire a target depth image of the color image; and a training unit, configured to train the first depth estimation model based on the color image and the intermediate depth image of the color image to obtain an updated first depth estimation model, and to train the second depth estimation model based on the color image and the target depth image of the color image to obtain an updated second depth estimation model.

[0044] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the updated second depth estimation model is specifically used to obtain a common edge image based on the first color image and the first intermediate depth image, the common edge image being used to represent the common edge portion of the first color image and the first intermediate depth image, and to obtain a first target depth image based on the common edge image and the first intermediate depth image.

[0045] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the training unit is specifically used to stitch together the first color image and the first intermediate depth image to obtain an input matrix, perform dimensionality reduction processing on the input matrix to obtain a first matrix and a second matrix, and obtain a common edge image based on the first matrix and the second matrix.

[0046] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the updated first depth estimation model is specifically used to obtain the depth feature vector of the first color image based on the first color image, including the first depth feature vector and the second depth feature vector, to obtain the first target feature vector based on the first depth feature vector and the first weight value, to obtain the second target feature vector based on the second depth feature vector and the second weight value, and to obtain the first intermediate depth image based on the first target feature vector and the second target feature vector.

[0047] Optionally, the color image may include a left view or a right view.

[0048] Optionally, the color image is captured using one or more cameras set in the terminal device.

[0049] Optional, the camera is a stereo camera.

[0050] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the acquisition unit is further configured to receive gradient information of the first depth estimation model and the second depth estimation model, wherein the gradient information is calculated by the second device based on the first color image and the first target depth image; the training unit is further configured to train and obtain updated first depth estimation model and updated second depth estimation model based on the gradient information.

[0051] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the training device further includes a sending unit for sending updated first depth estimation models and second depth estimation models.

[0052] Fifthly, a depth estimation apparatus is provided, characterized in that the apparatus includes a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method in any implementation of the first aspect.

[0053] Optionally, the device can be installed in various devices or systems that require depth estimation, such as vehicle terminals, mobile terminals, high-precision map systems, and intelligent transportation systems.

[0054] Optionally, the device can also be a chip.

[0055] In a sixth aspect, a training apparatus for a depth estimation model is provided, characterized in that the apparatus includes a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the training method in any implementation of the second aspect.

[0056] Optionally, the training device can be a host computer, server, cloud device, or other device capable of model training.

[0057] Optionally, the training device can also be a chip.

[0058] A seventh aspect provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing any implementation of the first or second aspect.

[0059] Eighthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any one of the implementations of the first or second aspect described above.

[0060] It should be noted that the above-mentioned computer program code can be stored in whole or in part on the first storage medium, wherein the first storage medium can be packaged together with the processor or packaged separately from the processor. This application embodiment does not specifically limit this.

[0061] Ninth aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface and executing the method in any one of the implementations of the first or second aspect described above.

[0062] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in any of the implementations of the first aspect. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of an artificial intelligence main framework according to an embodiment of this application.

[0064] Figure 2 This is an example application scenario of the depth estimation method in this application.

[0065] Figure 3 This is another application scenario of the depth estimation method in this application.

[0066] Figure 4 A schematic diagram of a system architecture according to an embodiment of this application.

[0067] Figure 5 This is an example of a schematic flowchart of the depth estimation method in this application.

[0068] Figure 6 This is an example of the input-output diagram for the depth estimation method in this application.

[0069] Figure 7 This is a schematic structural diagram of the first depth estimation model in this application.

[0070] Figure 8 This is a schematic diagram of an example of the improved multi-scale residual module (MSB) of this application.

[0071] Figure 9 This is an example of a schematic flowchart of the second depth estimation model in this application.

[0072] Figure 10 This is an example of a schematic flowchart illustrating the depth estimation model training method of this application.

[0073] Figure 11 This is an example of an interactive diagram illustrating the depth estimation model training method of this application.

[0074] Figure 12 This is a schematic block diagram of an example depth estimation device of this application.

[0075] Figure 13This is a schematic structural diagram of an example of the hardware structure of the depth estimation device of this application.

[0076] Figure 14 This is a schematic block diagram of an example of a depth estimation model training device of this application.

[0077] Figure 15 This is a schematic structural diagram of an example of the hardware structure of the depth estimation model training device of this application. Detailed Implementation

[0078] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0079] This application relates to neural networks. In order to better understand the methods of this application, the relevant terms and concepts of neural networks will be introduced below.

[0080] (1) Neural network (NN)

[0081] Neural networks can be composed of neural units, which can refer to units represented by x. s An arithmetic unit that takes an intercept of 1 as input can output an example as shown in the following formula:

[0082]

[0083] Where s = 1, 2, ..., n, n is a natural number greater than 1, representing the number of layers in the neural network, W s For x s The weights, also known as the parameters or coefficients of a neural network, x s Let 'b' be the input to the neural network, and 'f' be the bias of the neural unit. 'f' represents the activation function of the neural unit, which performs a non-linear transformation on the features in the neural network, converting the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer; the activation function can be the sigmoid function. A neural network is formed by connecting multiple individual neural units, meaning the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from that local receptive field, which can be a region composed of several neural units.

[0084] (2) Deep neural network (DNN)

[0085] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with multiple hidden layers. Based on the position of different layers, DNNs can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that each neuron in the i-th layer is connected to a neuron in the (i+1)-th layer.

[0086] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following non-linear relationship expression: in, It is the input vector. It is the output vector. is the offset vector, W is the weight, also called coefficient or parameter; the weight can be in the form of a weight matrix, and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained after simple operations. Because DNNs have many layers, the weights W and offset vectors... The number is also relatively large. The meaning of the weights in each layer of a DNN (for ease of description, we'll call them coefficients) is as follows: Taking the coefficient W as an example, assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0087] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0088] The input layer has no weights W. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more weights a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning weights, and its ultimate goal is to obtain the weights of all layers in the trained deep neural network (e.g., a weight matrix formed by the coefficients W of multiple layers).

[0089] (3) Convolutional Neural Network (CNN)

[0090] A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0091] (4) Loss function

[0092] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weights of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where weights are pre-configured for each layer.) For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, pre-defining "how to compare the difference between the predicted and target values" is the core of the process—the loss function or objective function. These are crucial equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0093] (5) Back propagation (BP)

[0094] Neural networks can employ backpropagation algorithms to correct the initial weight values ​​during training, thereby minimizing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then used to update the weights in the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process designed to obtain the optimal weights, such as the weight matrix, of the neural network model.

[0095] In traditional methods, depth estimation models are often trained using color image data. However, traditional methods neglect the use of common edge regions between color and depth images. As a result, the depth estimation images obtained by this method are of poor quality.

[0096] Furthermore, although current unsupervised depth estimation algorithms have eliminated the limitations of training labels to some extent and limited training costs, the quality of depth-estimated images can still be poor when encountering scenarios not encountered during the training phase after the algorithm model is deployed.

[0097] To address the aforementioned issues, this application proposes a depth estimation method. In this method, the depth estimation model includes a common edge extraction module capable of extracting common edges between the color image and the depth image, thus leveraging prior information from the color image to improve the estimation accuracy of the depth image. Furthermore, during the training of the depth estimation model, federated learning and unsupervised learning methods are combined to achieve periodic updates after the algorithm model is deployed, further enhancing the quality of the depth image output by the model.

[0098] The solutions in this application can be applied to various depth estimation scenarios such as image recognition, image processing, high-precision maps, intelligent driving, intelligent transportation, and autonomous driving.

[0099] Figure 1 This is a schematic diagram of an artificial intelligence main framework according to an embodiment of this application. The main framework describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0100] The above-mentioned artificial intelligence framework will be elaborated in detail from two dimensions: the "intelligent information chain" (horizontal axis) and the "information technology (IT) value chain" (vertical axis).

[0101] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0102] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0103] (1) Infrastructure:

[0104] Infrastructure provides computing power to support artificial intelligence systems, enables them to communicate with the outside world, and provides support through basic platforms.

[0105] Infrastructure can communicate with the outside world through sensors, and its computing power can be provided by smart chips.

[0106] The intelligent chip here can be a hardware acceleration chip such as a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0107] The basic platform of the infrastructure can include distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks.

[0108] For example, for infrastructure, data can be acquired through sensors and external communication, and then this data can be provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0109] (2) Data:

[0110] The data at the layer above the infrastructure is used to represent data sources in the field of artificial intelligence. This data involves at least one type of information, such as graphics, images, voice, and text. This data varies across different application areas and can have different forms of representation. For example, in the context of the Internet of Things (IoT), the content of the data is related to specific IoT connected terminals and may include sensor data such as force, displacement, liquid level, temperature, or humidity.

[0111] In the embodiments of this application, the data is, for example, a color image.

[0112] (3) Data processing:

[0113] The aforementioned data processing typically includes data training, machine learning, deep learning, search, reasoning, and decision-making.

[0114] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0115] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0116] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0117] (4) General abilities:

[0118] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0119] (5) Smart products and industry applications:

[0120] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, or intelligent terminals, etc.

[0121] Figure 2 This is a specific application scenario of the depth estimation method provided in the embodiments of this application. In this application scenario, a vehicle 220 and a cloud service system 210 may be included, and the vehicle 220 and the cloud service system 210 may communicate via a network.

[0122] Some or all of the functions of vehicle 220 are controlled by computing platform 221. Computing platform 221 may include at least one processor 222, which can execute instructions 224 stored in a non-transitory computer-readable medium such as memory 223.

[0123] In some embodiments, the computing platform 221 may also be multiple computing devices that control individual components or subsystems of the vehicle 220 in a distributed manner. The processor 222 may be any conventional processor, such as a central processing unit (CPU). Alternatively, the processor 222 may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0124] In addition to instructions 224, memory 223 may also store data such as road maps, route information, vehicle position, direction, speed, and other vehicle data, as well as other information. This information can be used by vehicle 220 and computing platform 221 during operation of vehicle 220 in autonomous, semi-autonomous, and / or manual modes.

[0125] It should be understood that Figure 2 The structure of the vehicle should not be construed as a limitation on the embodiments of this application.

[0126] Optionally, the vehicle 220 may include one or more different types of vehicles, or one or more different types of transport vehicles or movable objects that operate or move on land (e.g., highways, roads, railways, etc.), water (e.g., waterways, rivers, oceans, etc.), or in space. For example, vehicles may include automobiles, bicycles, motorcycles, trains, subways, airplanes, ships, aircraft, robots, or other types of transport vehicles or movable objects, etc., and this application embodiment does not limit this.

[0127] In addition, such as Figure 2 The application scenarios shown may also include cloud servers. In this embodiment, the cloud server can perform perception fusion and computational inference based on the gradient information uploaded by the vehicle and other information in the cloud server (such as information of other vehicles, road condition information, etc.) to train and update the network gradient, thereby improving the accuracy of the depth estimation model.

[0128] In one embodiment, the cloud server can also be implemented using a virtual machine.

[0129] The embodiments of this application can also be applied to many fields of artificial intelligence, such as image recognition, image processing, high-precision mapping, intelligent driving, intelligent transportation, and autonomous driving. Specifically, they are applied to the branches of these artificial intelligence fields that require depth estimation. For example, in the field of high-precision mapping, depth estimation can obtain richer information, thereby providing more accurate map information. As another example, in the field of intelligent driving, depth estimation can further determine the road conditions faced by the vehicle, thereby assisting the driver in making correct driving operations based on the actual situation.

[0130] The following is a brief introduction to the application scenarios of intelligent driving.

[0131] In intelligent driving scenarios, road condition information can be obtained by performing depth estimation on the input color image, thereby assisting the driver in making correct decisions. For example... Figure 3As shown, when the color images captured by the camera are input into the depth estimation model, current road condition information can be obtained, such as traffic light information, other vehicle information, and pedestrian information. This road condition information is then input into the driving decision module. Based on this road condition information, the driving decision module further determines what action to take. For example, if the current traffic light is red, the driving decision module will issue a stop instruction. Furthermore, in some products, the driving decision module can also initiate automatic braking. As another example, if the current road condition information indicates the presence of a pedestrian crossing ahead, the driving decision module will issue a deceleration instruction. In other words, Figure 3 It improves the safety of intelligent driving by using depth estimation, effectively preventing drivers from making mistakes due to poor observation.

[0132] Figure 4 This is a schematic diagram of a system architecture according to an embodiment of this application, which can be used to train neural network models, such as image recognition models and depth estimation models. Figure 4 As shown, the data acquisition device 460 is used to acquire training data. For the method of this embodiment, when training the first depth estimation model, the training data may include a color image; when training the second depth estimation model, the training data may include a color image and the first depth estimation map.

[0133] After collecting the training data, the data acquisition device 460 stores the training data in the database 430, and the training device 420 trains the target model / rule 401 based on the training data maintained in the database 430. "A / B" describes the relationship between related objects, indicating that there can be three kinds of relationships. For example, A / B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone.

[0134] The following describes how the training device 420 obtains the target model / rule 401 based on training data. In one scenario, the training device 420 processes the input original image and compares the output image with the original image until the difference between the output image and the original image is less than a certain threshold, thus completing the training of the target model / rule 401. In this case, the depth estimation model of this embodiment can be trained, that is, a trained basic neural network (i.e., the second depth estimation model) can be obtained, so that the basic neural network can be further used to train the first depth estimation model. In another scenario, the training device 420 processes the input color image and the first depth image until the accuracy of the category output by the training device 420 is greater than or equal to a certain threshold, thus completing the training of the target model / rule 401. In this case, the depth estimation model of this embodiment can be trained, that is, a depth estimation model is further trained based on the aforementioned basic neural network.

[0135] The target model / rule 401 described above can be used to implement the method of the embodiments of this application. Specifically, the target model / rule 401 in the embodiments of this application can be a neural network. It should be noted that in practical applications, the training data maintained in the database 430 may not all come from the data acquisition device 460; it may also be received from other devices. Furthermore, it should be noted that the training device 420 may not necessarily train the target model / rule 401 entirely based on the training data maintained in the database 430; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0136] The target model / rule 401 trained using training device 420 can be applied to different systems or devices, such as... Figure 4 The execution device 410 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, in-vehicle terminal, etc., or it can be a server or cloud service. Figure 4 In this embodiment, the execution device 410 is configured with an input / output (I / O) interface 412 for data interaction with external devices. Users can input data to the I / O interface 412 through the client device 440. The input data may include a color image input by the client device.

[0137] The preprocessing module 413 and the preprocessing module 414 are used to preprocess the input data (such as a color image) received by the I / O interface 412. In this embodiment, the preprocessing module 413 and the preprocessing module 414 may be omitted (or only one of them may be used), and the calculation module 411 may be used directly to process the input data.

[0138] During the preprocessing of input data by the execution device 410, or during the calculation module 411 of the execution device 410 performing calculations and other related processes, the execution device 410 can call data, code, etc. in the data storage system 450 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 450.

[0139] Finally, the I / O interface 412 returns the processing result to the client device 440, thereby providing it to the user.

[0140] It is worth noting that the training device 420 can generate corresponding target models / rules 401 based on different training data for different objectives or tasks. The corresponding target models / rules 401 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0141] Figure 5 This is an example of a schematic flowchart of the depth estimation method in this application.

[0142] S510, acquire the first color image.

[0143] Optionally, the first color image is a left view or a right view.

[0144] Optionally, the first color image may be captured using one or more cameras mounted on a second device, which may be a vehicle, an airplane, a mobile phone, or other possible mobile devices, and this application does not limit them.

[0145] Alternatively, the aforementioned camera can be a stereo camera.

[0146] S520, the first color image is input into the first depth estimation model to obtain the first intermediate depth image.

[0147] The first depth estimation model is trained based on the color image and the corresponding intermediate depth image.

[0148] In the embodiments of this application, the relationship between the color image and the first color image is not limited. For example, the first color image can be one type of color image, or the first color image does not belong to any of the above-mentioned color images, that is, the first color image is not data in the model training dataset. As introduced in the background art of this application, during the use of the model, scenarios that have not appeared in the model training may be encountered.

[0149] Specifically, the above-mentioned input of the first color image into the first depth estimation model to obtain the first intermediate depth image may include: obtaining the depth feature vector of the first color image based on the first color image, including a first depth feature vector and a second depth feature vector; obtaining a first target feature vector based on the first depth feature vector and a first weight value; obtaining a second target feature vector based on the second depth feature vector and the second weight value; and obtaining the first intermediate depth image based on the first target feature vector and the second target feature vector.

[0150] Thus, according to the technical solution of this application, the target vector is obtained based on the depth feature vector and the weight value corresponding to the depth feature vector, thereby obtaining an intermediate depth image, which helps to improve the accuracy of the depth image.

[0151] S530, input the first color image and the first intermediate depth image into the second depth estimation model to obtain the first target depth image.

[0152] The second depth estimation model is trained based on the color image and the target depth image corresponding to the color image.

[0153] Specifically, the above-mentioned inputting the first color image and the first intermediate depth image into the second depth estimation model to obtain the first target depth image may include: obtaining a common edge image based on the first color image and the first intermediate depth image, the common edge image being used to represent the common edge portion of the first color image and the first intermediate depth image; and obtaining the first target depth image based on the common edge image and the first intermediate depth image.

[0154] Specifically, obtaining the common edge image based on the first color image and the first intermediate depth image can include: concatenating the first color image and the first intermediate depth image to obtain an input matrix; performing dimensionality reduction processing on the input matrix to obtain a first matrix and a second matrix; and obtaining the common edge image based on the first matrix and the second matrix.

[0155] Thus, according to the technical solution of this application, a common edge image is obtained through a color image and an intermediate depth image corresponding to the color image, and a target depth image is obtained based on the common edge image and the intermediate depth image corresponding to the color image. This fully utilizes the prior information of the color image to obtain the common edge image and effectively utilizes the favorable common edge portion in the color image and the intermediate depth image corresponding to the color image, thereby helping to improve the accuracy of the target depth estimation image.

[0156] Optionally, the above method further includes: calculating gradient information of a first depth estimation model and a second depth estimation model based on a first color image and a first target depth image; sending the gradient information to a first device, wherein the gradient information is used by the first device to train and obtain an updated first depth estimation model and an updated second depth estimation model.

[0157] Optionally, the above method may further include: receiving an updated first depth estimation model and an updated second depth estimation model from a first device.

[0158] Thus, according to the technical solution of this application, gradient information is calculated based on the color image and the target depth image corresponding to the color image, and the gradient information is sent to the first device for the first device to train and update the depth estimation model. This can solve the problem of difficulty in upgrading the model after deployment and help improve the accuracy of the model.

[0159] Optionally, the above method further includes: training an updated first depth estimation model and an updated second depth estimation model based on gradient information.

[0160] Thus, according to the technical solution of this application, gradient information is calculated based on the color image and the target depth image corresponding to the color image, and the depth estimation model is trained and updated based on the gradient information. This can solve the problem of difficulty in upgrading the model after deployment and help improve the accuracy of the model.

[0161] According to the technical solution of this application, an intermediate depth image corresponding to the color image is obtained through the color image, and a target depth image corresponding to the color image is obtained based on the color image and the intermediate depth image corresponding to the color image. This can make full use of the prior information of the color image and help improve the accuracy of the target depth estimation image.

[0162] Figure 6 This is an example of the input-output diagram for the depth estimation method in this application.

[0163] like Figure 6 As shown, in this embodiment of the application, the first color image 610 is used as input to the first depth estimation model to obtain the first intermediate depth image 620, and the first intermediate depth image 620 and the first color image 610 are used as input to the second depth estimation model to obtain the first target depth image 630.

[0164] The first color image 610 is... Figure 5 The first color image acquired in S510, by way of example and not limitation, is a right view taken by a binocular camera mounted on the vehicle.

[0165] The first color image 610 is input into the first depth estimation model, i.e., the process is executed. Figure 5 As described in S520, a first depth image 620 is obtained. This first depth estimation model is trained based on the color image and its corresponding intermediate depth image.

[0166] As one possible implementation, the structure of this first depth estimation model can be as follows: Figure 7 As shown, the first depth estimation model may include an encoder and a decoder. By way of example and not limitation, the first depth estimation model may perform three downsampling operations on the first color image 610 at the encoder and three upsampling operations at the decoder to obtain a first intermediate depth image 620. The three downsampling operations at the encoder can be performed by a four-stage convolutional network, for example, a black bar X... i (i = 1, 2, 3) and X4. Convolutional network X i(i = 1, 2, 3) consists of n (n = 4, 5, 8) improved multi-scale residual modules (MSBs) and a 3*3 convolutional layer with a stride of 2. The values ​​of i and n correspond one-to-one. For example, convolutional network X1 (i = 1) consists of 4 (n = 4) improved MSBs and a 3*3 convolutional layer with a stride of 2; X4 consists of 8 MSBs. Correspondingly, at the decoding end, three upsampling operations are performed. The first depth estimation model can include three deconvolutional networks and one convolutional network, such as the white bars, Y... i (i = 1, 2, 3) and Y4. Deconvolutional network Y i (i = 1, 2, 3) consists of a single deconvolutional layer with a kernel size of 3, while the convolutional network Y4 consists of 3*3 convolutional layers.

[0167] In this embodiment of the application, the first depth estimation model can extract the depth feature vector X from the first color image 610. n-1 and the depth feature vector X n-1 After MSB processing, the target feature vector X is obtained. n .

[0168] The following uses X n-1 Taking ∈h*w*c as an example, combined with Figure 8 This section provides a detailed introduction to the MSB processing procedure.

[0169] Input the depth feature vector X of the first depth estimation model n-1 ∈h*w*c, we can first follow the formula (1) below:

[0170] feature 3*3 feature 5*5 =Split(X) n-1 (1)

[0171] After the splitting operation, the depth feature vector X is... n-1 Divide the passage in two to obtain X. split-1 ∈h*w*1 / 2c and X split-2 ∈h*w*1 / 2c. Let X... split-1 ∈h*w*1 / 2c and X split-2 ∈h*w*1 / 2c respectively input their respective multi-scale feature pathways, according to the following formulas (2) and (3):

[0172] feature 3*3 =relu(bn(ω) 1*3 (relu(bn(ω 3*1 (X split_1 (2)

[0173] feature 5*5 =relu(bn(ω) 1*5 (relu(bn(ω 5*1 (X split_2 (3)

[0174] Obtain the first depth feature vector. 3*3 ∈h*w*1 / 2c and the second deep feature vector 5*5 ∈h*w*1 / 2c. In formula (2), represents convolution, represents the normalization layer, and represents the activation layer (rectified linear unit). Simultaneously, X... split-1 ∈h*w*1 / 2c and X split-2 ∈h*w*1 / 2c respectively input their respective weight paths, according to the following formulas (4) and (5):

[0175] pool 3*3 =Sigmoid(ω) 1*1 (relu(ω 1*1 (avgpool(X split_1 (4)

[0176] pool 5*5 =Sigmoid(ω) 1*1 (relu(ω 1*1 (avgpool(X split_2 (5)

[0177] The first and second weight values ​​are obtained, which can be used to distinguish the importance of features from different channels. Then, the first deep feature vector is... 3*3 ∈h*w*1 / 2c and the second deep feature vector 5*5 ∈h*w*1 / 2c and the corresponding weight values ​​are determined according to the following formulas (6) and (7):

[0178] feature_attention 3*3 =feature 3*3 *pool 3*3 (6)

[0179] feature_attention 5*5 =feature 5*5 *pool 5*5 (7)

[0180] Obtain the first target deep feature vector feature_attention3*3 ∈h*w*1 / 2c and the second target depth feature vector feature_attention 5*5 ∈h*w*1 / 2c. Then follow the formula (8) below:

[0181] msb=concat(feature_attention 3*3 feature_attention 5*5 (8)

[0182] feature_attention 3*3 and feature_attention 5*5 Concatenate along the channel dimension to obtain msb∈h*w*c. Finally, input deep feature vector X... n-1 ∈h*w*c and msb∈h*w*c according to formula (9):

[0183] X n =msb+X n-1 (9)

[0184] Perform an element-wise addition operation to obtain the output X of the MSB module. n ∈h*w*c.

[0185] The following is a brief introduction to the processing procedures of the encoding and decoding ends of the first depth estimation model in this application. This part can be referred to existing technologies.

[0186] At the encoding end, the first color image 610 is first input into a convolutional layer with a kernel size of 3*3 to extract initial features. init ∈H*W*16, after being encoded through layers X1 to X4, as shown in formulas (10) to (17) below:

[0187] feature init =relu(bn(ω) 3*3 (I))) (10)

[0188] X1 = msb(feature) init (11)

[0189] Down1 = relu(bn(ω) 3*3,stride＝2 (X1))) (12)

[0190] X2 = msb(Down1) (13)

[0191] Down2=relu(bn(ω 3*3,stride＝2 (X2))) (14)

[0192] X3 = msb(Down2) (15)

[0193] Down3=relu(bn(ω 3*3,stride＝2 (X3))) (16)

[0194] X4 = msb(Down3) (17)

[0195] For ease of description, the first color image is uniformly referred to as I in the formula; and in formulas (11), (13), (15) and (17), the number of MSBs is 4, 5, 8 and 8 respectively.

[0196] At the decoding end, the features corresponding to those at the encoding end are concatenated to the corresponding stage, and upsampling is performed through deconvolution, as shown in formulas (18) to (22):

[0197] Up1=relu(bn(Cvtω 3*3,stride＝2 (X4))) (18)

[0198] Up2=relu(bn(Cvtω 3*3,stride＝2 (concat(Up1,X3))) (19)

[0199] Up3=relu(bn(Cvtω 3*3,stride＝2 (concat(Up2,X2))) (20)

[0200] Up4=relu(bn(Cvtω 3*3,stride＝2 (concat(Up3,X1))) (21)

[0201] depth corase =ω 3*3 (Up4) (22)

[0202] Finally, a 3x3 convolution with one channel is used to output the depth estimation map for the encoding and decoding stages. corase ∈H*W*1, which is the first intermediate depth image.

[0203] After the above processing, it is complete. Figure 5 As shown in S520, the first color image is input into the first depth estimation model to obtain the first intermediate depth image 620.

[0204] According to the technical solution of this application, a target vector is obtained based on the depth feature vector and the weight value corresponding to the depth feature vector, thereby obtaining an intermediate depth image, which helps to improve the accuracy of the depth image.

[0205] However, as described in the background section of this application, the prior information of the color image is not well utilized in the above process, resulting in poor quality of the depth estimation image. Therefore, this application further provides a second depth estimation model, which helps to further improve the accuracy of the depth estimation image.

[0206] like Figure 6 As shown, after obtaining the first intermediate depth image 620, this application can perform step S530, that is, use the second depth estimation model to process the first color image 610 and the first intermediate depth image 620 to obtain the first target depth image 630.

[0207] The specific process of obtaining the first target depth image 630 from the first color image 610 and the first intermediate depth image 620 can be as follows: Figure 9 As shown (the first color image in) Figure 6 Winning bid number 610, in Figure 9 The reference number is 910; the first intermediate depth image is in Figure 6 Winning bid number 620, in Figure 9 The reference number is 920; the first target depth image is in Figure 6 Winning bid number 630, in Figure 9 (Issue number 930):

[0208] First, input the first color image 610 (I∈H*W*3) and the first intermediate depth image (depth) corase According to formula (23): ∈H*W*1)

[0209] input = relu(bn(ω) 3*3 (concat(image,depth corase )))) (twenty three)

[0210] The components are concatenated together and then subjected to a convolutional layer to obtain the input matrix input∈H*W*16.

[0211] Then, the input matrix input∈H*W*16 is input into the channel attention module and the spatial attention module for processing, according to formulas (24) and (25):

[0212] attention_c=concat(maxpool(input),avgpool(input)) (24)

[0213] attention_s=ω 1*1 (concat(relu(bn(ω 3*3(relu(bn(ω 1*1 (input)))))),relu(bn(ω 5*5 (relu(bn(ω 1*1 (input)))))))) (25

[0214] We obtain the first matrix attention_c∈1*1*32 and the second matrix attention_s∈H*W*1. Then, according to formula (26):

[0215] attention_map=sigmoid(attention_s*attention_c) (26)

[0216] The common edge image attention_map∈H*W*1 is obtained from the first matrix attention_c∈1*1*32 and the second matrix attention_s∈H*W*1, where sigmoid is the activation function.

[0217] Finally, according to formula (27):

[0218] depth refine =depth corase *attention_map (27)

[0219] From the first intermediate depth image (depth) corase The final refined depth image is obtained by combining the common edge image (∈H*W*1) and the attention map (∈H*W*1). refine ∈H*W*1, which is the depth image of the first target.

[0220] Thus, according to the technical solution of this application, a common edge image is obtained through a color image and an intermediate depth image corresponding to the color image, and a target depth image is obtained based on the common edge image and the intermediate depth image corresponding to the color image. This fully utilizes the prior information of the color image to obtain the common edge image and effectively utilizes the favorable common edge portion in the color image and the intermediate depth image corresponding to the color image, thereby helping to improve the accuracy of the target depth estimation image.

[0221] Figure 10 This is an example of a schematic flowchart illustrating the depth estimation model training method of this application.

[0222] S1010, The first depth estimation model is used to process the color image to obtain the intermediate depth image of the color image; the second depth estimation model is used to process the color image and the intermediate depth image of the color image to obtain the target depth image of the color image.

[0223] Optionally, the above color image may include a left view or a right view.

[0224] Optionally, the aforementioned color images are obtained using one or more cameras located on the terminal device.

[0225] Optionally, the camera mentioned above is a stereo camera.

[0226] S1020: Based on the color image and the intermediate depth image of the color image, train the first depth estimation model to obtain the updated first depth estimation model; based on the color image and the target depth image of the color image, train the second depth estimation model to obtain the updated second depth estimation model.

[0227] Specifically, the updated first depth estimation model is used to obtain the depth feature vector of the first color image based on the first color image, including the first depth feature vector and the second depth feature vector; to obtain the first target feature vector based on the first depth feature vector and the first weight value; to obtain the second target feature vector based on the second depth feature vector and the second weight value; and to obtain the first intermediate depth image based on the first target feature vector and the second target feature vector.

[0228] According to the technical solution of this application, a first depth estimation model is obtained by training a color image, and a second depth estimation model is obtained by training a color image and an intermediate depth image corresponding to the color image. The trained depth estimation model can make full use of the prior information of the color image, which helps to improve the accuracy of the target depth estimation image.

[0229] Specifically, the updated second depth estimation model is used to obtain a common edge image based on the first color image and the first intermediate depth image, the common edge image being used to represent the common edge portion of the first color image and the first intermediate depth image; and to obtain a first target depth image based on the common edge image and the first intermediate depth image.

[0230] Optionally, obtaining the common edge image based on the first color image and the first intermediate depth image includes: concatenating the first color image and the first intermediate depth image to obtain an input matrix; performing dimensionality reduction processing on the input matrix to obtain a first matrix and a second matrix; and obtaining the common edge image based on the first matrix and the second matrix.

[0231] Optionally, the first device may refer to the training of the first depth estimation model and the second depth estimation model. Figure 4 The training methods shown in the document will not be elaborated upon here.

[0232] Thus, the depth estimation model trained according to the technical solution of this application can obtain a common edge image from the color image and the intermediate depth image corresponding to the color image, and obtain the target depth image from the common edge image and the intermediate depth image corresponding to the color image. It can make full use of the prior information of the color image to obtain the common edge image and effectively utilize the favorable common edge parts in the color image and the intermediate depth image corresponding to the color image, thereby helping to improve the accuracy of the target depth estimation image.

[0233] In this embodiment of the application, the training method for the depth estimation model further includes interaction between the first device and the terminal device, such as... Figure 11 As shown, the first device sends the trained depth estimation model to the second device (e.g., a terminal device). During the model usage process, the second device can back-calculate gradient information and feed it back to the first device for the first device to train an updated depth estimation model.

[0234] S1110, the second device calculates the gradient information of the first depth estimation model and the second depth estimation model based on the first color image and the first target depth image.

[0235] Specifically, during the use of the model, the second device will encounter numerous scenarios not present in the training dataset. Taking a car equipped with a computing platform as an example, while the car is in motion, the binocular camera inside can capture a first color image not present in the training dataset. This first color image can then be used by the first and second depth estimation models deployed on the computing platform to infer the depth image of the first target. Furthermore, the computing platform can back-calculate the network gradient to obtain gradient information.

[0236] S1120, the second device sends gradient information of the first depth estimation model and the second depth estimation model to the first device. This gradient information is used by the first device to train and obtain updated first depth estimation models and second depth estimation models.

[0237] S1130, the first device trains and obtains an updated first depth estimation model and an updated second depth estimation model based on gradient information.

[0238] S1140, the second device receives the updated first depth estimation model and the updated second depth estimation model from the first device.

[0239] According to the technical solution of this application, the first device can train an updated depth estimation model based on the gradient information calculated by the second device from the color image and the target depth image corresponding to the color image. This can solve the problem of difficulty in upgrading the model after deployment and help improve the accuracy of the model.

[0240] Optionally, under certain conditions, the second device can execute S1150 to train and obtain an updated first depth estimation model and an updated second depth estimation model based on gradient information.

[0241] In this way, the second device can train and update the depth estimation model based on the gradient information calculated from the color image and the target depth image corresponding to the color image. This can solve the problem of difficulty in upgrading the model after deployment and help improve the accuracy of the model.

[0242] The above, combined with Figures 5 to 11 This application provides a detailed description of the depth estimation method and the training method for the depth estimation model. It is understood that each device, in order to achieve the above functions, includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0243] The following, combined with Figures 12 to 15 This application provides a detailed description of the depth estimation apparatus and the training apparatus for the depth estimation model provided in the embodiments. It should be understood that the descriptions of the apparatus embodiments correspond to the descriptions of the method embodiments; therefore, any content not described in detail can be found in the above method embodiments. For the sake of brevity, some content is not repeated.

[0244] Figure 12 This is a schematic structural diagram of an example of the depth estimation device of this application. Figure 12 The depth estimation device 1200 shown includes an acquisition unit 1210, a processing unit 1220, and a transceiver unit 1230.

[0245] The acquisition unit 1210, processing unit 1220, and transceiver unit 1230 can be used to execute the depth estimation method of the embodiments of this application. The processing unit 1220 is capable of implementing... Figures 5 to 9 The functions of the first depth estimation model and the second depth estimation model.

[0246] For example, the acquisition unit 1210 is used to acquire a first color image; the processing unit 1220 is used to input the first color image into a first depth estimation model to obtain a first intermediate depth image, and input the first color image and the first intermediate depth image into a second depth estimation model to obtain a first target depth image; the transceiver unit 1230 is used to send gradient information to the first device, and the gradient information is used by the first device to train and obtain an updated first depth estimation model and an updated second depth estimation model.

[0247] It should be understood that the processing unit 1220 in the above-described device 1200 may be equivalent to the processor 1320 in the device 1300 described below.

[0248] Figure 13 This is a schematic diagram of the hardware structure of the depth estimation device provided in the embodiments of this application. Figure 13 The depth estimation device 1300 shown (which may specifically be a computer device) includes a memory 1310, a processor 1320, a communication interface 1330, and a bus 1340. The memory 1310, the processor 1320, and the communication interface 1330 are interconnected via the bus 1340.

[0249] Optionally, the communication interface can also enable communication with other devices, such as communication with the first device.

[0250] The memory 1310 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1310 may store a program, and when the program stored in the memory 1310 is executed by the processor 1320, the processor 1320 and the communication interface 1330 are used to execute the various steps of the depth estimation method of the embodiments of this application.

[0251] The processor 1320 may be a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to achieve the functions required by the units in the depth estimation apparatus of this application embodiment, or to execute the depth estimation method of the method embodiment of this application.

[0252] The processor 1320 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the depth estimation method of this application can be completed by the integrated logic circuitry in the hardware of the processor 1320 or by instructions in software form. The processor 1320 can also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1310. The processor 1320 reads the information in the memory 1310 and, in conjunction with its hardware, performs the functions required by the units included in the depth estimation device of this application embodiment, or executes the depth estimation method of the method embodiment of this application.

[0253] The communication interface 1330 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 1300 and other devices or communication networks. For example, the first depth estimation model and / or the second depth estimation model described above can be obtained through the communication interface 1330.

[0254] Bus 1340 may include a pathway for transmitting information between various components of device 1300 (e.g., memory 1310, processor 1320, communication interface 1330).

[0255] Figure 14 This is a schematic block diagram of a training apparatus for a depth estimation model according to an embodiment of this application. Figure 14 The training device 1400 for the depth estimation model shown includes an acquisition unit 1410, a training unit 1420, and a transceiver unit 1430.

[0256] The acquisition unit 1410, training unit 1420, and transceiver unit 1430 can be used to execute the training method of the depth estimation model of the embodiments of this application.

[0257] For example, the acquisition unit 1410 is used to process the color image using a first depth estimation model to obtain an intermediate depth image of the color image, and to process the color image and the intermediate depth image of the color image using a second depth estimation model to obtain a target depth image of the color image; the training unit 1420 is used to train the first depth estimation model based on the color image and the intermediate depth image of the color image to obtain an updated first depth estimation model, and to train the second depth estimation model based on the color image and the target depth image of the color image to obtain an updated second depth estimation model; the transceiver unit 1430 is used to receive gradient information of the first depth estimation model and the second depth estimation model, the gradient information being calculated by the second device based on the first color image and the first target depth image, and to send the updated first depth estimation model and the updated second depth estimation model to the second device.

[0258] It should be understood that the training unit 1420 in the above-described device 1400 may be equivalent to the processor 1520 in the device 1500 described below.

[0259] Figure 15 This is a schematic diagram of the hardware structure of the training device for the depth estimation model provided in this application embodiment. Figure 15 The training device 1500 for the depth estimation model shown (specifically, the device 1500 can be a computer device) includes a memory 1510, a processor 1520, a communication interface 1530, and a bus 1540. The memory 1510, processor 1520, and communication interface 1530 are interconnected via the bus 1540.

[0260] Optionally, the communication interface can also enable communication with other devices, such as a second device.

[0261] The memory 1510 may be a ROM, a static storage device, a dynamic storage device, or RAM. The memory 1510 may store a program, and when the program stored in the memory 1510 is executed by the processor 1520, the processor 1520 and the communication interface 1530 are used to execute the various steps of the training method of the depth estimation model of the embodiments of this application.

[0262] The processor 1520 may be a CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute related programs to implement the functions required by the units in the depth estimation model training device of the present application embodiment, or to execute the depth estimation model training method of the present application method embodiment.

[0263] The processor 1520 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the training method for the depth estimation model of this application can be completed by the integrated logic circuits in the hardware of the processor 1520 or by instructions in software form. The aforementioned processor 1520 can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1510. The processor 1520 reads the information in the memory 1510 and, in conjunction with its hardware, performs the functions required by the units included in the depth estimation model training device of the present application embodiment, or executes the depth estimation model training method of the method embodiment of the present application.

[0264] The communication interface 1530 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 1500 and other devices or communication networks. For example, the first training data described above can be obtained through the communication interface 1530.

[0265] Bus 1540 may include a pathway for transmitting information between various components of device 1500 (e.g., memory 1510, processor 1520, communication interface 1530).

[0266] It should be noted that, although Figure 13 The device 1300 shown Figure 15 The illustrated device 1500 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, devices 1300 and 1500 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 1300 and 1500 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that devices 1300 and 1500 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 13 , Figure 15 All the devices shown.

[0267] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different devices to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0268] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0269] In the several embodiments provided in this application, it should be understood that the disclosed systems, methods, and apparatuses can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0270] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0271] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0272] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: a Universal Serial Bus flash disk (USB flash disk, UFD), which can also be simply referred to as a USB flash drive, portable hard drive, ROM, RAM, magnetic disk, or optical disk, and other media capable of storing program code.

[0273] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of depth estimation, the method comprising: The method comprises: obtaining a first color image; inputting the first color image into a first depth estimation model to obtain a first intermediate depth image, the first depth estimation model being trained according to a color image and an intermediate depth image corresponding to the color image; inputting the first color image and the first intermediate depth image into a second depth estimation model to obtain a first target depth image, the second depth estimation model being trained according to the color image and a target depth image corresponding to the color image; the inputting the first color image and the first intermediate depth image into the second depth estimation model to obtain the first target depth image comprises: obtaining a common edge image according to the first color image and the first intermediate depth image, the common edge image being used to represent a common edge part of the first color image and the first intermediate depth image; obtaining the first target depth image according to the common edge image and the first intermediate depth image.

2. The method of claim 1, wherein, The obtaining the common edge image according to the first color image and the first intermediate depth image comprises: splicing the first color image and the first intermediate depth image to obtain an input matrix; performing dimension reduction processing on the input matrix to obtain a first matrix and a second matrix; obtaining the common edge image according to the first matrix and the second matrix.

3. The method according to claim 1 or 2, characterized in that, The inputting the first color image into the first depth estimation model to obtain the first intermediate depth image comprises: obtaining a depth feature vector of the first color image according to the first color image, the depth feature vector comprising a first depth feature vector and a second depth feature vector; obtaining a first target feature vector according to the first depth feature vector and a first weight value, and obtaining a second target feature vector according to the second depth feature vector and a second weight value; obtaining the first intermediate depth image according to the first target feature vector and the second target feature vector.

4. The method according to claim 1 or 2, characterized in that, The color image comprises a left view or a right view.

5. The method according to claim 1 or 2, characterized in that, The color image is obtained by using one or more cameras arranged on a terminal device.

6. The method of claim 5, wherein, The camera is a binocular camera.

7. The method according to claim 1 or 2, characterized in that, The method further comprises: calculating gradient information of the first depth estimation model and the second depth estimation model according to the first color image and the first target depth image; sending the gradient information to a first device, the gradient information being used by the first device to train an updated first depth estimation model and an updated second depth estimation model.

8. The method of claim 7, wherein, The method further comprises: receiving the updated first depth estimation model and the updated second depth estimation model from the first device.

9. The method of claim 7, wherein, The method further comprises: training the updated first depth estimation model and the updated second depth estimation model according to the gradient information.

10. A method for training a depth estimation model, comprising: The method comprises: processing a color image using a first depth estimation model to obtain an intermediate depth image of the color image; processing the color image and the intermediate depth image of the color image using a second depth estimation model to obtain a target depth image of the color image; training the first depth estimation model according to the color image and the intermediate depth image of the color image to obtain an updated first depth estimation model; training the second depth estimation model according to the color image and the target depth image of the color image to obtain an updated second depth estimation model; wherein the updated second depth estimation model is specifically used for, obtaining a common edge image according to the first color image and the first intermediate depth image, the common edge image being used for representing a common edge part of the first color image and the first intermediate depth image; obtaining a first target depth image according to the common edge image and the first intermediate depth image.

11. The training method of claim 10, wherein, The obtaining of the common edge image according to the first color image and the first intermediate depth image comprises: splicing the first color image and the first intermediate depth image to obtain an input matrix; performing dimension reduction processing on the input matrix to obtain a first matrix and a second matrix; obtaining the common edge image according to the first matrix and the second matrix.

12. The training method according to claim 10 or 11, characterized in that, The updated first depth estimation model is specifically used for, obtaining a depth feature vector of the first color image according to the first color image, the depth feature vector comprising a first depth feature vector and a second depth feature vector; obtaining a first target feature vector according to the first depth feature vector and a first weight value, and obtaining a second target feature vector according to the second depth feature vector and a second weight value; obtaining the first intermediate depth image according to the first target feature vector and the second target feature vector.

13. The training method according to claim 10 or 11, characterized in that, The color image comprises a left view or a right view.

14. The training method according to claim 10 or 11, characterized in that, The color image is obtained by using one or more cameras arranged on a terminal device.

15. The training method of claim 14, wherein, The camera is a binocular camera.

16. The training method of claim 10 or 11, wherein, The training method further comprises: receiving gradient information of the first depth estimation model and the second depth estimation model, the gradient information being calculated according to the first color image and the first target depth image; training the updated first depth estimation model and the updated second depth estimation model according to the gradient information.

17. The training method of claim 16, wherein, The training method further comprises: sending the updated first depth estimation model and the updated second depth estimation model.

18. An apparatus for depth estimation, the apparatus comprising: The apparatus comprises a processor and a data interface, the processor reading instructions stored on a memory through the data interface to execute the method of any one of claims 1 to 9. 19.A device for training a depth estimation model, comprising: The apparatus comprises a processor and a data interface, the processor reading instructions stored on a memory through the data interface to execute the method of any one of claims 10 to 17.

20. A computer-readable storage medium, characterized in that, The computer readable medium stores program codes for execution by a device, the program codes comprising instructions for executing the method of any one of claims 1 to 9 or any one of claims 10 to 17.

21. A computer program product, characterised in that, When the computer program is executed on a computer, the computer is caused to execute the method of any one of claims 1 to 9 or any one of claims 10 to 17.

Images