Training method and device of depth estimation model, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a method and device for training a depth estimation model, electronic equipment and a storage medium, and relate to the fields of intelligent transportation, games, artificial intelligence, computer vision and cloud technology. The method comprises: performing feature extraction on a first image in each training sample by an initial depth estimation model in a neural network model to obtain at least two scales of first feature maps of the first image, fusing the first feature maps of the first image to obtain a target feature map of the first image, and obtaining predicted depth information of the first image based on the target feature map. Based on the predicted depth information and the real depth information of each first image, a training loss is determined, and a depth estimation model in the neural network model that meets a preset condition is taken as a trained depth estimation model. The trained depth estimation model can fuse the multi-resolution features obtained by each feature extraction to obtain more accurate depth information of the first image.

Description

Technical Field

[0001] This application relates to the fields of intelligent transportation, gaming, artificial intelligence, computer vision, and cloud technology. Specifically, this application relates to a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for a depth estimation model. Background Technology

[0002] With the development of science and technology, more and more fields need to process images to obtain relevant information from them.

[0003] Currently, image information is restored directly based on the feature maps obtained from each feature extraction of the image. This leads to the loss of some information in the image, resulting in the relevant information in the obtained image not being able to well express the information of the image itself and failing to meet actual needs. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training a depth estimation model. The depth estimation model obtained through this training method can fully integrate multi-resolution features obtained from image feature extraction, thereby better extracting depth information from the image. The specific technical solution is as follows:

[0005] According to one aspect of the embodiments of this application, a method for training a depth estimation model is provided, the method comprising:

[0006] Acquire multiple training samples, each training sample including a first image and the true depth information of the first image;

[0007] The neural network model is iteratively trained based on multiple training samples until the total training loss of the neural network model meets a preset condition. The neural network model includes an initial depth estimation model, and the depth estimation model that meets the preset condition is used as the trained depth estimation model. The total training loss is determined in the following way:

[0008] For each first image, features are extracted from the first image using an initial depth estimation model to obtain first feature maps of at least two scales corresponding to the first image; the target feature map corresponding to the first image is obtained by fusing the first feature maps corresponding to the first image; and the first depth information of the first image is obtained based on the target feature map.

[0009] The first training loss is determined based on the first depth information and the true depth information corresponding to each first image.

[0010] According to another aspect of the embodiments of this application, a training apparatus for a depth estimation model is provided, the apparatus comprising:

[0011] The training sample acquisition module is used to acquire multiple training samples, each training sample including a first image and the true depth information of the first image;

[0012] The depth estimation model training module is used to iteratively train the neural network model based on multiple training samples until the total training loss of the neural network model meets a preset condition. The neural network model includes an initial depth estimation model, and the depth estimation model that meets the preset condition is used as the trained depth estimation model. The total training loss is determined in the following way:

[0013] For each first image, features are extracted from the first image using an initial depth estimation model to obtain first feature maps of at least two scales corresponding to the first image; the target feature map corresponding to the first image is obtained by fusing the first feature maps; and first depth information is obtained based on the target feature map.

[0014] The total training loss is determined based on the first depth information and the true depth information corresponding to each first image.

[0015] Optionally, when the depth estimation model training module obtains the target feature map corresponding to the first image by fusing the first feature maps corresponding to the first image, it is specifically used for:

[0016] The first feature map corresponding to the first image is used as the initial feature map set. The first operation is performed a set number of times. The second feature map obtained from the last first operation is used as the target feature map.

[0017] The depth estimation model training module, when performing the first operation, is specifically used for:

[0018] Each feature map in the feature map set, except for the feature map with the smallest scale, is taken as a feature map to be processed. For each feature map to be processed, the feature map to be processed is fused with a feature map with a smaller scale to obtain a second feature map with the same scale as the feature map to be processed. The second feature map corresponding to each feature map to be processed is taken as the feature map set corresponding to the next first operation.

[0019] Optionally, for each first image, the depth estimation model training module is further used to:

[0020] For at least one feature map in each of the second feature maps obtained in the first operation, the second depth information of the first image corresponding to the feature map is obtained based on each feature map in the first operation.

[0021] The second training loss is determined based on the real depth information and the second depth information corresponding to each first image;

[0022] The depth estimation model training module, when determining the total training loss based on the first depth information and the true depth information corresponding to each first image, is specifically used for:

[0023] Based on the first depth information and the true depth information corresponding to each first image, the first training loss is determined;

[0024] The total training loss is obtained based on the first training loss and the second training loss.

[0025] Optionally, each training sample may further include a second image that is temporally consecutive to the first image in the training sample, and the neural network model may further include an initial pose estimation model. During the training of the neural network model, the depth estimation model training module is also used for:

[0026] For each first image, the first image and the corresponding second image are input into the initial pose estimation model to obtain the pose information of the first image;

[0027] For each first image, image reconstruction is performed based on the pose information and first depth information of the first image to obtain the reconstructed image corresponding to the first image;

[0028] The third training loss is determined based on the first difference between each first image and its corresponding reconstructed image;

[0029] The depth estimation model training module, when determining the total training loss based on the first depth information and the true depth information corresponding to each first image, is specifically used for:

[0030] Based on the first depth information and the true depth information corresponding to each first image, the first training loss is determined;

[0031] The total training loss is determined based on the first training loss and the third training loss.

[0032] Optionally, the depth estimation model training module is also used for:

[0033] For a first image, determine a second difference between the first image and the corresponding second image;

[0034] The weight values ​​of the third training loss are determined based on the magnitude relationship between the first difference and the second difference corresponding to each first image.

[0035] When determining the total training loss based on the first and third training losses, the depth estimation model training module is specifically used for:

[0036] Based on the weight values ​​of the third training loss, the first and third training losses are weighted and summed to obtain the total training loss.

[0037] Optionally, for each first image, the depth estimation model training module is also used to:

[0038] For the feature map with the smallest scale in the feature map set, residual feature extraction is performed on the feature map to obtain a second feature map with the same scale as the feature map.

[0039] For each feature map to be processed, the depth estimation model training module, when fusing the feature map to be processed with a feature map of a smaller scale to obtain a second feature map with the same scale as the original feature map, specifically uses the following:

[0040] Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed. The third feature map is then concatenated with the feature map to be processed with a smaller scale to obtain the fourth feature map corresponding to the feature map to be processed. Feature extraction is performed on the fourth feature map to obtain the second feature map with the same scale as the feature map to be processed.

[0041] Optionally, for each first image, the number of operations is set to at least three. For the third first operation and each subsequent first operation, when the depth estimation model training module fuses the feature map to be processed with a feature map of a smaller scale to obtain a second feature map with a scale corresponding to the feature map to be processed, it is specifically used for:

[0042] Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed;

[0043] Determine the second feature map to be processed in the feature map set corresponding to at least one first operation prior to this first operation, whose scale corresponds to the feature map to be processed;

[0044] Residual features are extracted from each second feature map to be processed to obtain the fourth feature map corresponding to each second feature map to be processed.

[0045] Based on the correlation between the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed, the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed are determined respectively.

[0046] Based on the weights corresponding to the third feature map, each of the fourth feature maps, and the feature maps with a scale smaller than the feature map to be processed, the third feature map, each of the fourth feature maps, and the feature maps with a scale smaller than the feature map to be processed are fused to obtain a second feature map with a scale corresponding to the feature map to be processed.

[0047] Optionally, for each first image, the depth estimation model training module, when using the second feature map obtained from the last first operation as the target feature map, specifically uses:

[0048] For each of at least two scales, the second feature map obtained after the last first operation corresponding to that scale is determined as the target feature map for that scale;

[0049] For each first image, the depth estimation model training module, when obtaining the first depth information of the first image based on the target feature map, is specifically used for:

[0050] First depth information is obtained based on each feature map in at least one feature map of the target feature map at each scale.

[0051] According to another aspect of the embodiments of this application, a depth information determination method is provided, the method comprising:

[0052] Obtain the image to be processed;

[0053] The image to be processed is input into a trained depth estimation model, and the depth information of the image to be processed is obtained through the model.

[0054] The trained depth estimation model is obtained by training the neural network model using the methods described above.

[0055] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0056] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0057] According to another aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0058] The beneficial effects of the technical solutions provided in this application are:

[0059] During the training of the depth estimation model, for each training sample's first image, feature extraction is performed on the first image to obtain at least two scales of first feature maps, i.e., at least two resolutions of first feature maps. By fusing the first feature maps corresponding to the first image, a target feature map of the first image is obtained. This ensures that the target feature map can include information from the first feature maps at multiple resolutions, allowing the first depth information obtained based on the target feature map to well represent the depth information in the first image. Based on the first depth information and the true depth information corresponding to each first image, the total training loss of the neural network model is determined. The neural network model is then iteratively trained based on multiple training samples until the total training loss of the neural network model meets a preset condition, resulting in a well-trained depth estimation model. This well-trained depth estimation model can effectively extract the depth information of the input image (i.e., the image to be processed).

[0060] Furthermore, when determining the depth information of the image to be processed using the trained depth estimation model, more accurate depth information of the image to be processed can be obtained by fusing the multi-resolution features obtained from each feature extraction of the image to be processed, thus avoiding the loss of some information of the image in the process of determining the depth information. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0062] Figure 1 A flowchart illustrating the training method of the depth estimation model provided in an embodiment of this application is shown;

[0063] Figure 2 This illustration shows a schematic diagram of obtaining first depth information according to an embodiment of this application;

[0064] Figure 3 This illustration shows a schematic diagram of obtaining a second feature map according to an embodiment of this application;

[0065] Figure 4 This illustration shows another schematic diagram of obtaining a second feature map provided by an embodiment of this application;

[0066] Figure 5 A schematic diagram of the initial depth estimation model provided in an embodiment of this application is shown;

[0067] Figure 6 A structural diagram of the neural network model provided in an embodiment of this application is shown;

[0068] Figure 7 A schematic diagram of the initial attitude estimation model in an embodiment of this application is shown;

[0069] Figure 8 This illustration shows a schematic diagram of depth information obtained from images in a dataset in an application scenario applicable to the embodiments of this application;

[0070] Figure 9 A flowchart of the depth information determination method provided in an embodiment of this application is shown;

[0071] Figure 10 A schematic diagram of a depth information determination system provided in an embodiment of this application is shown;

[0072] Figure 11 The diagram illustrates the signaling interaction of the depth information determination method in the application scenario to which this application embodiment applies;

[0073] Figure 12 A schematic diagram of a depth information determination method applicable to the scenarios described in this application is shown.

[0074] Figure 13 This illustration shows a schematic diagram of the structure of a training device for a depth estimation model provided in an embodiment of this application;

[0075] Figure 14 A schematic diagram of the structure of an electronic device to which this application embodiment applies is shown. Detailed Implementation

[0076] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0077] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0078] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0079] The following explains the terminology and related technologies involved in this application:

[0080] Self-supervised learning refers to a model learning common rules or latent knowledge from unlabeled data.

[0081] Encoder-Decoder: The encoder itself is a series of convolutional networks. This network mainly consists of convolutional layers, pooling layers, and batch normalization (BN) layers. Convolutional layers are responsible for acquiring local image features, pooling layers downsample the image and pass scale-invariant features to the next layer, and BN mainly normalizes the distribution of the training images, accelerating learning. In summary, the encoder classifies and analyzes low-level local pixel values ​​of the image to obtain high-order semantic information. The decoder upsamples the scaled-down feature image and then performs convolution on the upsampled image to refine the geometry of the object and compensate for the loss of detail caused by the pooling layers in the encoder scaling down the object.

[0082] Monocular depth estimation: Obtaining depth information (relative to the distance from the shooting source) for each pixel in a single image.

[0083] Monocular depth estimation model: The input of the model is an image, and the output is the depth information corresponding to each pixel. For the human eye, due to a large amount of prior knowledge, a large amount of depth information can be extracted from the image information obtained by one eye. Therefore, the monocular depth estimation model not only needs to learn objective depth information from two-dimensional images, but also needs to extract some empirical information. The latter is more sensitive to the image acquisition equipment and the acquisition scene of the dataset.

[0084] The KITTI dataset is currently the largest international dataset for evaluating computer vision algorithms in autonomous driving scenarios. This data is used to evaluate the performance of computer vision techniques such as stereo imaging, optical flow, visual odometry, and 3-dimensional object detection in in-vehicle environments. KITTI includes real-world image data collected in urban, rural, and highway scenes, with up to 15 vehicles and 30 pedestrians per image, and varying degrees of occlusion and truncation.

[0085] The intrinsic parameter matrix of the image acquisition device is used to transform the 3D coordinates of the image acquired by the image acquisition device to 2D homogeneous image coordinates.

[0086] 3D point cloud: A point cloud is a dataset of points in a coordinate system, including various information such as coordinates, color, and intensity. 3D point cloud is a method of representing the three-dimensional world using point clouds. It can be imagined as atomizing a three-dimensional object, using multiple points to represent an object. 3D modeling using 3D point clouds is not only fast but also has high accuracy and more precise details.

[0087] L1 paradigm loss function: also known as minimum absolute error, it minimizes the sum of the target value and the absolute value.

[0088] SSIM loss function: compares the similarity of two images from three dimensions: brightness, contrast, and structure.

[0089] 6-DOF: DOF (degree of freedom) refers to the three rotation angles (Euler angles) and position-related information in three dimensions: up / down, front / back, and left / right.

[0090] FPN (Feature Pyramid Network): First proposed in a CVPR 2017 paper for object detection tasks. The main network uses ResNet, and top-level features are fused with bottom-level features through upsampling. Moreover, each layer makes independent predictions, thereby fusing multi-scale information.

[0091] Attention technology is essentially inspired by the human visual attention mechanism. When perceiving things, humans generally don't look at a scene from beginning to end, but rather observe and pay attention to specific parts as needed. Furthermore, when they discover that something they want to observe frequently appears in a certain part of a scene, they learn from it and will focus their attention on that part when similar scenes appear again. Therefore, attention is equivalent to assigning different weight coefficients to each element of the extracted features as a basis for determining their importance, and then fusing the features based on these weight coefficients.

[0092] In related technologies, image information recovery is performed directly based on feature maps obtained from feature extraction of the image each time. This leads to the loss of some image information, resulting in the obtained image information not being able to accurately represent the image itself and failing to meet practical needs. Therefore, this application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training a depth estimation model. By fusing multi-resolution features obtained from feature extraction of the image to be processed, more accurate depth information of the image to be processed can be obtained, avoiding the loss of some image information during the process of determining depth information.

[0093] The execution entity of the depth estimation model training method provided in this application embodiment can be a depth estimation model training device. This depth estimation model training device can be any computer device, including but not limited to a terminal or server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal (also referred to as a user terminal or user equipment (UE)) can be a smartphone, tablet, laptop, desktop computer, intelligent voice interaction device (e.g., smart speaker), wearable electronic device (e.g., smartwatch), in-vehicle terminal, smart home appliance (e.g., smart TV), AR (Augmented Reality) / VR (Virtual Reality) device, etc., but is not limited thereto. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. The depth estimation model training method can be implemented by a processor calling computer-readable instructions stored in memory.

[0094] The training method for the depth estimation model provided in this application can be implemented based on artificial intelligence (AI) technology. For example, a neural network model can be constructed based on AI technology, and the constructed neural network model can be trained based on multiple training samples to obtain a trained depth estimation model. AI is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain optimal results.

[0095] The aforementioned depth estimation model can also be a neural network model based on computer vision (CV) technology. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition and measurement, and further performs image processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection.

[0096] The training method for the depth estimation model provided in this application embodiment can be implemented based on cloud technology. For example, the data computation involved in training the depth estimation model can be carried out using cloud computing. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize data computation, storage, processing, and sharing.

[0097] The depth estimation model trained using the method provided in this application can also be implemented in the field of intelligent transportation and applied to autonomous driving or traffic applications. For example, the trained depth estimation model can be used to determine the depth information of vehicle images captured by a driving camera, thereby determining the distance between the vehicle to which the driving camera belongs and the vehicles in the vehicle image, and avoiding surrounding vehicles of the currently driven vehicle. Currently, the depth estimation model trained using the method provided in this application can also be used in the gaming field. For example, by using this depth estimation model to determine the depth information of images in the current game scene, the distance between other game characters and the current game character can be obtained, thereby avoiding or rescuing other game characters.

[0098] Optionally, the training method for the depth estimation model provided in this application embodiment can also be implemented based on blockchain technology. Specifically, the data used in the training process of the depth estimation model, such as multiple training samples and the second feature map obtained in each first operation during training, can be stored on the blockchain.

[0099] It should be noted that, in the optional embodiments of this application, the object information (e.g., the acquired image to be processed includes a certain object) and other related data require the permission or consent of the object when the above embodiments of this application are applied to specific products or technologies. Furthermore, the collection, use, and processing of the related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to an object, this data must be obtained with the object's authorization and consent, and in accordance with the relevant laws, regulations, and standards of the country and region.

[0100] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0101] Figure 1 A flowchart illustrating the training method of the depth estimation model provided in an embodiment of this application is shown. Figure 1 As shown, the method includes:

[0102] S10: Obtain multiple training samples, each training sample including a first image and the true depth information of the first image;

[0103] S20: Iteratively train the neural network model based on multiple training samples until the total training loss of the neural network model meets a preset condition. The neural network model includes an initial depth estimation model, and the depth estimation model that meets the preset condition is used as the trained depth estimation model. The total training loss is determined in the following way:

[0104] For each first image, features are extracted from the first image using an initial depth estimation model to obtain first feature maps of at least two scales corresponding to the first image; the target feature map corresponding to the first image is obtained by fusing the first feature maps corresponding to the first image; and the first depth information of the first image is obtained based on the target feature map.

[0105] The total training loss is determined based on the first depth information and the true depth information corresponding to each first image.

[0106] The trained depth estimation model can be trained by a model training device, which can be the entity that executes the training method of the depth estimation model, or it can be an electronic device that is different from the current entity, such as a cloud server.

[0107] By training this depth estimation model, more accurate depth information for each first image can be obtained by fusing the multi-resolution features extracted from each feature extraction of the first image, avoiding the loss of some information from the first image during the depth information determination process. This allows for precise determination of the depth information of the image to be processed using the trained depth estimation model.

[0108] Optionally, a subset of images from any image dataset can be selected as the first images, and multiple training samples can be generated based on the selected first images and their depth information. For example, the dataset could be the KITTI dataset, specifically the KITTI2015 dataset.

[0109] This application does not limit the image quality of each first image; the two images can be the same or different. Image quality can be characterized by one or more image metrics, and this application does not limit this. Specifically, image metrics may include, but are not limited to, resolution, sharpness, and image scale. For example, the scale of the first image can be 128*128, 256*256, etc. It should be understood that the higher the image scale, the higher the corresponding resolution and sharpness.

[0110] Of course, to ensure that the depth information obtained from the depth estimation model in this embodiment is of a fixed scale, each first image can be preprocessed before training the neural network model based on each training sample, and the size of each first image can be adjusted to a fixed scale. For example, the scale of each first image can be adjusted to 196*640.

[0111] The initial depth estimation model is a neural network model based on FPN. Specifically, it can be an encoder-decoder structure. The encoder, or backbone network, can be implemented using ResNet (Residual Neural Network), specifically ResNet-18. The decoder can also be implemented using FPN.

[0112] When obtaining the first depth information through the initial depth estimation model, the encoder part of the initial depth estimation model can be used to extract features from the first image to obtain first feature maps of at least two scales corresponding to the first image. The decoder part of the initial depth estimation model can then fuse the first feature maps corresponding to the first image to obtain the target feature map corresponding to the first image. Based on the target feature map, the first depth information is obtained.

[0113] In the embodiments of this application, for each first image, a first feature extraction can be performed on the first image to obtain a first feature map with the same scale as the first image, and feature extraction can be performed based on the first feature map to obtain a first feature map with a scale of 1 / 2 of the first image, and so on, to obtain first feature maps of at least two scales of the first image.

[0114] This application embodiment does not limit the specific number of scale types for each obtained first feature map. These can be set based on empirical or experimental values, or adjusted according to actual conditions. Taking a scale type of 5 for each obtained first feature map as an example, for a first image, after feature extraction, the scales of the obtained first feature maps are: the scale of the first image, 1 / 2 of the scale of the first image, 1 / 4 of the scale of the first image, 1 / 8 of the scale of the first image, and 1 / 16 of the scale of the first image.

[0115] In this implementation, when fusing the first feature maps corresponding to the first image to obtain the target feature map corresponding to the first image, any number of first feature maps can be fused, or the first feature maps can be fused sequentially in pairs, following the order of their size from smallest to largest. For example, the smallest and second smallest first feature maps can be fused first, then the fused result can be fused with the third smallest first feature map, and so on, until the largest first feature map is fused to obtain the target feature map corresponding to the first image.

[0116] When fusing the first feature maps, since the scales of the first feature maps are inconsistent, feature extraction can be performed on the first feature maps with smaller scales firstly, such as upsampling feature extraction, so that the scales of the processed first feature maps are the same, and then the processed first feature maps can be fused.

[0117] In this embodiment, the target feature map is a feature map that can represent all the information of the first image. Further feature extraction can be performed on the target feature map to obtain specific information about the first image; this embodiment does not limit this. For example, further depth feature extraction can be performed on the target feature map to obtain the depth information of the first image. As another example, further pose feature extraction can be performed on the target feature map to obtain the pose information of the first image, etc.

[0118] For an image, its depth information can be a matrix formed by the depth values ​​of each pixel in the image. For example, if the image is any image acquired by an image acquisition device, its depth information can be a matrix formed by the distances between each pixel in the image and the image acquisition device.

[0119] When obtaining the first depth information based on the target feature map, the target feature map can be input into a specific depth feature extraction network that can perform further depth feature extraction. Based on this specific depth feature extraction network, the first depth information is obtained. Specifically, Figure 2A schematic diagram illustrating the acquisition of first depth information provided in an embodiment of this application is shown. For example... Figure 2 As shown, this specific deep feature extraction network can include two convolutional layers with a kernel size of 3*3 and a softmax (normalization exponent) layer. The softmax layer can be composed of a sigmoid (an activation function) layer.

[0120] Alternatively, the total training loss can be determined based on the L1 paradigm loss function (inverse depth smoothing loss function):

[0121]

[0122] in, d represents the average normalized inverse depth information of each pixel in an image, to prevent the depth information obtained by predicting the image from being shrunk. t This represents the depth information of a pixel in the image. This represents the average depth information of all pixels in the image. t This represents the first image. This represents the gradient value of the average normalized inverse depth information corresponding to each pixel in the first image in the dimension containing the first depth information. This represents the gradient value of the average normalized inverse depth information corresponding to each pixel in the first image in the dimension where the true depth information is located.

[0123] If the number of target feature maps is one, the first depth information can be directly obtained based on this target feature map, and the first training loss can be determined based on the first depth information and the true depth information corresponding to each first image. If the number of target feature maps is greater than one, for each first image, the depth information corresponding to each target feature map can be obtained separately, the average value of the depth information corresponding to each target feature map can be used as the first depth information, and the first training loss can be determined based on the first depth information and the true depth information corresponding to each first image. Alternatively, after obtaining the depth information corresponding to each target feature map of a first image, the depth loss corresponding to each target feature map of the first image can be determined based on the first depth information and the true depth information, the average value of the depth loss corresponding to each target feature map of the first image can be determined, and the first training loss can be determined based on the average value of the depth loss corresponding to each first image.

[0124] The preset conditions may include, but are not limited to, the convergence of the loss function of the neural network model, the total training loss being less than or equal to a preset value, or the accuracy of the obtained first depth information meeting the requirements. In this embodiment, the specific value of the preset value is not limited; it can be an empirical value, an experimental value, or a value determined based on actual circumstances.

[0125] To improve the fusion effect of each first feature map corresponding to each first image, the embodiments of this application also provide the following optional implementation methods:

[0126] Optionally, by fusing the first feature maps corresponding to the first image, a target feature map corresponding to the first image is obtained, including:

[0127] Using the first feature maps corresponding to the first image as an initial feature map set, performing a first operation a predetermined number of times, and using the second feature map obtained from the last first operation as the target feature map; the first operation includes:

[0128] Each feature map in the feature map set, except for the feature map with the smallest scale, is taken as a feature map to be processed. For each feature map to be processed, the feature map to be processed is fused with a feature map with a smaller scale to obtain a second feature map with the same scale as the feature map to be processed. The second feature map corresponding to each feature map to be processed is taken as the feature map set corresponding to the next first operation.

[0129] This application embodiment does not limit the number of times the first operation is performed. It can be set based on empirical or experimental values, or adjusted according to actual conditions. For example, the number of times can be set to 2.

[0130] Optionally, during the execution of the first operation, the first operation can be performed on each feature map in the feature map set corresponding to the first operation in ascending order.

[0131] When performing the first operation, for the feature map with the smallest scale in the feature map set corresponding to the first operation, the feature map can be directly used as the second feature map corresponding to the feature map, or residual feature extraction can be performed on the feature map to obtain the second feature map corresponding to the feature map.

[0132] Residual feature extraction is performed on a feature map, which means extracting features from the feature map again and then performing a residual concatenation with the original feature map. The feature map after residual concatenation is the same as the feature map obtained by residual feature extraction. The scale of the feature map obtained by residual feature extraction is the same as the scale of the original feature map.

[0133] The first operation can be performed on each feature map in the feature map set in ascending order of scale. For each feature map to be processed, since it is not necessary to process feature maps with a scale smaller than the feature map to be processed, or the first operation has already been performed on feature maps with a scale smaller than the feature map to be processed, the feature map to be processed can be directly fused with the second feature map corresponding to the feature map with a scale smaller than the feature map to be processed.

[0134] The second feature map obtained from the last first operation can be either the second feature map corresponding to the first feature map with the largest scale, or the second feature map obtained from the last first operation corresponding to the first feature map at each scale. Based on the above process of performing the first operation, it can be seen that, regardless of which scale the second feature map corresponds to, the second feature map obtained from the last first operation can include all the first feature maps with scales smaller than or equal to the first feature map, as well as relevant information from each second feature map. That is, it can include information from each feature map with a scale smaller than or equal to the second feature map during each first operation, which can well represent the information of the first image.

[0135] Considering that the first feature map with the smallest scale and the second feature map corresponding to the first feature map contain less information about the first image, and that the first feature map with the smallest scale and the second feature map corresponding to the first feature map have a weak ability to express information, in this embodiment of the application, the second feature map obtained by the last first operation can also be the second feature map obtained by the last first operation corresponding to the first feature map of each scale, except for the second feature map with the smallest scale.

[0136] Of course, when performing the first operation each time, for each feature map to be processed in the feature map set corresponding to the first operation, when fusing the feature map to be processed with a feature map with a smaller scale, the feature map to be processed can be fused directly with the feature map with a smaller scale, or residual features can be extracted from the feature map to be processed first, and then the feature map to be processed after residual feature extraction can be fused with the feature map with a smaller scale in the feature map set corresponding to the first operation.

[0137] Optionally, for a feature map to be processed, the corresponding feature map with a scale smaller than the feature map to be processed can be a feature map with a scale smaller than the feature map to be processed and whose scale is adjacent to the feature map to be processed, or it can be at least two feature maps with a scale smaller than the feature map to be processed.

[0138] For each feature map to be processed, since the scale of the feature map to be processed is inconsistent with the scale of the feature map with a smaller scale, feature extraction can be performed on the feature map with a smaller scale than the feature map to be processed, such as upsampling feature extraction, so that the scale of the processed feature map is the same as the scale of the feature map to be processed. Then the feature map to be processed and the processed feature map are fused to obtain a second feature map with the same scale as the feature map to be processed.

[0139] Since each time a first operation is performed on each feature map to be processed, the resulting second feature map with a larger scale can include information from the second feature maps with a scale smaller than or equal to the feature map to be processed in each of the previous first operations, that is, each feature map to be processed has already fused information from each feature map with a resolution smaller than or equal to the feature map to be processed, in order to reduce the amount of computation, the embodiments of this application also provide the following optional implementation methods:

[0140] Optionally, for each first image, the second feature map obtained from the last first operation is used as the target feature map, including:

[0141] For each of at least two scales, the second feature map obtained from the last first operation corresponding to that scale is determined as the target feature map for that scale;

[0142] For each first image, based on the target feature map, the first depth information of that first image is obtained, including:

[0143] First depth information is obtained based on each feature map in at least one feature map of the target feature map at each scale.

[0144] In this implementation, since larger-scale feature maps contain more information, the first depth information can be directly obtained based on the larger-scale target feature map among the target feature maps at each scale. Alternatively, the depth information corresponding to the target feature map at each scale can be obtained first, and then the average value of the depth information corresponding to the target feature map at each scale can be determined as the first depth information. Then, the first training loss can be determined based on the first depth information and the true depth information of each first image.

[0145] By using the above method, the second feature map obtained from the last first operation corresponding to each scale is determined as the target feature map for that scale. In subsequent operations, it is not necessary to obtain the second feature map of the current scale again, which can reduce the amount of computation without affecting the information in the obtained feature map.

[0146] Optionally, for each first image, the first operation further includes:

[0147] For the feature map with the smallest scale in the feature map set, residual feature extraction is performed on the feature map to obtain a second feature map with the same scale as the feature map.

[0148] For each feature map to be processed, a second feature map with the same scale as the original feature map is obtained by fusing it with feature maps of smaller scale. This second feature map may include:

[0149] Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed. The third feature map is then concatenated with the feature map to be processed with a smaller scale to obtain the fourth feature map corresponding to the feature map to be processed. Feature extraction is performed on the fourth feature map to obtain the second feature map with the same scale as the feature map to be processed.

[0150] In this implementation, by performing residual feature extraction on each feature map in the feature map set during the first operation, and then further processing the feature maps at each scale after residual feature extraction to obtain the second feature map corresponding to that scale, it is possible to prevent overfitting of the initial depth model, which could lead to problems such as data resource consumption.

[0151] Figure 3 This illustration shows a schematic diagram of obtaining a second feature map according to an embodiment of this application. Taking a feature map to be processed as an example, Figure 3 As shown, when fusing a feature map to be processed, we can first upsample and extract features from feature maps in the feature map set whose scale is smaller than that of the feature map to be processed, to obtain... Figure 3 The feature map shown by the right diagonal line is then subjected to residual feature extraction to obtain... Figure 3 The feature map shown in the blank area is concatenated with the feature map obtained by upsampling feature extraction and the feature map obtained by residual feature extraction. The concatenated feature map is then input into a 3*3 convolutional layer to perform downsampling feature extraction on the concatenated feature map, thereby obtaining the second feature map corresponding to the current feature map to be processed.

[0152] Optionally, for each first image, the number of operations is set to at least three. For the third first operation and each subsequent first operation, a second feature map corresponding to the scale of the feature map to be processed is obtained by fusing the feature map to be processed with a feature map with a smaller scale. The method further includes:

[0153] Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed;

[0154] Determine the second feature map to be processed in the feature map set corresponding to at least one first operation prior to this first operation, whose scale corresponds to the feature map to be processed;

[0155] Residual features are extracted from each second feature map to be processed to obtain the fourth feature map corresponding to each second feature map to be processed.

[0156] Based on the correlation between the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed, the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed are determined respectively.

[0157] Based on the weights corresponding to the third feature map, each of the fourth feature maps, and the feature maps with a scale smaller than the feature map to be processed, the third feature map, each of the fourth feature maps, and the feature maps with a scale smaller than the feature map to be processed are fused to obtain a second feature map with a scale corresponding to the feature map to be processed.

[0158] In this implementation, the scale and the number of second feature maps corresponding to the feature map to be processed can be determined according to the actual situation. The closer the number of each second feature map to be processed is to the number of all first operations before the first operation, the better the effect of the scale and the second feature map corresponding to the feature map to be processed will be.

[0159] After obtaining the third feature map corresponding to the feature map to be processed, each of the fourth feature maps corresponding to the feature map to be processed, and the feature map to be processed and the feature map with a smaller scale, the weights corresponding to the third feature map, each of the fourth feature maps, and the feature map to be processed and the feature map with a smaller scale can be determined by a convolutional layer based on attention technology.

[0160] Before determining the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed, we can first perform upsampling feature extraction on the feature maps with a scale smaller than the feature map to be processed. This makes the upsampled feature maps have the same scale as the third and fourth feature maps. Therefore, based on the third feature map, each of the fourth feature maps, and the upsampled feature maps, we can obtain the weights corresponding to the third feature map, each of the fourth feature maps, and the upsampled feature maps, respectively. The weights corresponding to the upsampled feature maps are the same as the weights corresponding to the feature maps with a scale smaller than the feature map to be processed.

[0161] Optionally, the third feature map, each of the fourth feature maps, and the feature map to be processed along with the feature maps to be processed with smaller scales can be weighted and concatenated based on their respective weights to achieve fusion. After fusing the third feature map, each of the fourth feature maps, and the feature map to be processed along with the feature maps to be processed with smaller scales, the fused feature map can be directly determined as the second feature map with the scale corresponding to the feature map to be processed. Alternatively, the fused feature map can be input into a convolutional layer with a 1*1 kernel to obtain the second feature map with the scale corresponding to the feature map to be processed.

[0162] Figure 4 This illustration shows another schematic diagram of obtaining a second feature map according to an embodiment of this application. Taking a feature map to be processed as an example, such as... Figure 4 As shown, the third feature map corresponding to the feature map to be processed is 1 ( Figure 4 The number of feature maps filled with diagonal lines in the middle and right sides is 1. Figure 4 In the case of a feature map without filler, a feature map can be extracted by upsampling the third feature map, each of the fourth feature maps, and the feature map to be processed with a feature map of a smaller scale than the feature map to be processed. Figure 4 The feature map filled with midpoints is input into a convolutional layer based on attention technology. Figure 4 In the Att convolutional layer, weights corresponding to the third feature map, each of the fourth feature maps, and the feature map after upsampling feature extraction are obtained respectively. Based on the weights corresponding to the third feature map, each of the fourth feature maps, and the feature map after upsampling feature extraction, the third feature map, each of the fourth feature maps, and the feature map after upsampling feature extraction are weighted and concatenated. The fused feature map is then input into a convolutional layer with a 1*1 kernel. Through this convolutional layer, a second feature map with a scale corresponding to the feature map to be processed can be obtained.

[0163] In this way, because in each first operation, the second feature map corresponding to each feature map to be processed can fuse the information of the feature map to be processed and the second feature map corresponding to the feature map with a smaller scale than the feature map to be processed in the first operation, for each first operation, by fusing the feature map to be processed with the second feature map obtained in at least one operation before the first operation to obtain a second feature map with the same scale as the feature map to be processed, the second feature map corresponding to the feature map to be processed can fuse the information of the feature map to be processed, the feature map with a smaller scale in the first operation, and the second feature map with the same scale as the feature map to be processed obtained in at least one operation before the first operation, thus better representing the information in the first image.

[0164] Since the second feature map obtained after each first operation can include information from all feature maps with a scale smaller than or equal to the second feature map during the current first operation, it can well represent the information of the first image. The more times the first operation is performed, the more accurate the information included in the resulting second feature map becomes. The ability of the second feature map to represent information after each first operation is related to the network layer that performed the first operation. To specifically train each part of the initial depth estimation model, this application also provides the following optional implementation methods:

[0165] Optionally, for each first image, the number of times is set to at least twice, and the method may further include:

[0166] For at least one feature map in each of the second feature maps obtained in the first operation, the second depth information of the first image corresponding to the feature map is obtained based on each feature map in the first operation.

[0167] The second training loss is determined based on the real depth information and the second depth information corresponding to each first image;

[0168] Based on the first depth information and the true depth information corresponding to each first image, the total training loss is determined, including:

[0169] Based on the first depth information and the true depth information corresponding to each first image, the first training loss is determined;

[0170] The total training loss is obtained based on the first training loss and the second training loss.

[0171] In this implementation, the second depth information of the first image corresponding to each feature map can be obtained based on each feature map in each second feature map obtained in the first operation.

[0172] Of course, considering that the first feature map with the smallest scale and the second feature map corresponding to it obtained by the first operation contain relatively little information about the first image, the ability of the first feature map with the smallest scale and the second feature map corresponding to it to express information is relatively weak. In this embodiment of the application, the second depth information of the first image corresponding to each feature map other than the second feature map with the smallest scale can also be obtained based on each of the second feature maps obtained by the first operation.

[0173] After determining the second depth information corresponding to each first image, the total training loss can be determined based on the L1 paradigm loss function as described above. The first training loss is determined based on the first depth information and the true depth information corresponding to each first image, and the second training loss is determined based on the true depth information and the second depth information corresponding to each first image.

[0174] The total training loss can be obtained by directly summing the first training loss and the second training loss, or by pre-setting weights for the first and second training losses respectively, and then weighting and summing the first and second training losses according to these weights to obtain the total training loss. The weights for the first and second training losses can be the same or different; this embodiment does not impose any restrictions on this. For example, the weights for both the first and second training losses can be set to 10. -3 .

[0175] Figure 5 A schematic diagram of the initial depth estimation model provided in an embodiment of this application is shown. Figure 5 In the diagram, each box represents a network module in the initial depth estimation model. Solid lines represent downsampling feature extraction, dashed lines represent residual feature extraction, dotted lines represent upsampling feature extraction, and spaced lines represent the input. For example... Figure 5 As shown, by inputting the first image into the initial depth estimation model, the network module F can be used sequentially. 00 F 10 F 20 F 30 and F 40Feature extraction is performed on the first image to obtain first feature maps corresponding to five scales: a first scale (the same scale as the first image), a second scale (1 / 2 the scale of the first image), a third scale (1 / 4 the scale of the first image), a fourth scale (1 / 8 the scale of the first image), and a fifth scale (1 / 16 the scale of the first image). After obtaining the first feature maps, each first feature map can be used as a feature map to be processed. This first operation is performed five times. Specifically, it can be achieved through the network module F. 01 F 11 F 21 F 31 and F 41 To achieve the first operation, the network module F can be used. 02 F 12 and F 22 To achieve the second operation of the first operation, the network module F can be used. 03 and F 13 To achieve the third first operation, the network module F can be used. 04 To achieve the fourth first operation, the network module F can be used. 05 F 15 F 25 F 35 and F 45 Perform the fifth step of the first operation. Considering that the feature map at the smallest scale (1 / 16 of the scale of the first image) contains relatively little information, such as... Figure 5 As shown, when extracting the first depth information, the second feature map with the smallest scale obtained from the last first operation is not directly processed; similarly, when extracting the second depth information, the second feature map with the smallest scale obtained from the first operation is not directly processed. That is, the second feature map obtained from the last first operation (i.e., the second feature map obtained through network module F) is processed separately. 05 F 15 F 25 and F 35 The output feature map is used as the target feature map. A specific depth feature extraction network module is used to extract features from each target feature map, resulting in four first depth information values. Then, the specific depth feature extraction network module is used to extract features from each of the second feature maps obtained from the first operation (i.e., those extracted by network module F). 01 F 11 F 21 and F 31 Feature extraction is performed on the output feature map to obtain four second-depth information.

[0176] It should be noted that, in Figure 5 The initial depth estimation model shown can contain only a specific depth feature extraction network, by using network module F 05 F15 F 25 F 35 F 01 F 11 F 21 and F 31 By cascading it with this specific depth feature extraction network, the process of determining the first depth information and the second depth information described above can be achieved.

[0177] Among them, through network module F 01 F 11 F 21 F 31 and F 41 The specific process for achieving the first operation is as follows:

[0178] via network module F 41 Residual feature extraction is performed on the first feature map at the fifth scale to obtain a second feature map corresponding to the first feature map at the fifth scale. This is then processed through network module F. 31 Residual feature extraction is performed on the first feature map at the fourth scale, and upsampling feature extraction is performed on the second feature map at the fifth scale. The feature map after residual feature extraction is then concatenated with the feature map after upsampling feature extraction. Figure 3 The 3x3 convolutional layer shown performs downsampling feature extraction on the concatenated feature map, resulting in a second feature map at the fourth scale. This process is repeated sequentially through network module F. 21 This yields a second feature map with a scale of the third scale; through network module F... 11 This yields a second feature map with a scale of the second scale; through network module F... 01 This yields a second feature map with a scale of the first scale.

[0179] The second feature maps obtained after the first operation are used as new feature maps to be processed, and then processed by network module F. 02 F 12 and F 22 The specific process for implementing the second first operation is as follows:

[0180] via network module F 22 Residual feature extraction is performed on the second feature map at the third scale obtained after the first operation, and upsampling feature extraction is performed on the second feature map at the fourth scale obtained after the first operation, such as... Figure 3 As shown, the feature map after residual feature extraction is concatenated with the feature map after upsampling feature extraction. The concatenated feature map is then passed through a 3x3 convolutional layer for downsampling feature extraction, resulting in a second feature map with a third scale after the second first operation. This process is repeated sequentially through network module F. 12The second feature map, obtained after the second first operation, is at the second scale; through network module F 02 The second feature map obtained after the second first operation is the one with the first scale.

[0181] The second feature maps obtained after the second first operation are used as new feature maps to be processed, and then processed by network module F. 03 and F 13 The specific process for implementing the third first operation is as follows:

[0182] via network module F 13 Residual feature extraction is performed on the second feature maps at the second scale obtained after the first operation and the second operation, respectively. Upsampling feature extraction is performed on the second feature map at the third scale obtained after the second operation, such as... Figure 4 As shown, the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature are input into a convolutional layer implemented using attention technology. This yields weight values ​​corresponding to the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature. Based on these weight values, the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature are weighted and concatenated. The weighted concatenated feature map is then input into a convolutional layer with a 1*1 kernel, resulting in a second feature map with the second scale obtained after the third first operation. This process is repeated sequentially through network module F. 03 The second feature map obtained after the third first operation is the first scale.

[0183] The second feature maps obtained after the third first operation are used as new feature maps to be processed, and then processed by network module F. 04 The specific process of implementing the fourth first operation is as follows:

[0184] via network module F 04 Residual feature extraction is performed on the second feature maps at the first scale obtained after the first, second, and third operations, respectively. Upsampling feature extraction is performed on the second feature map at the second scale obtained after the third operation. Figure 4 As shown, based on the weight values ​​of the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature, the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature are weighted and concatenated. The weighted concatenated feature map is then input into a convolutional layer with a kernel of 1*1 to obtain a second feature map with the first scale after the fourth first operation.

[0185] The second feature map, corresponding to the smallest scale in each first operation, is taken as the feature map to be processed, and then processed by network module F. 05 F15 F 25 F 35 and F 45 The specific process for achieving the fifth first operation is as follows:

[0186] via network module F 45 Residual feature extraction is performed on the second feature map at the fifth scale obtained after the first operation, resulting in the second feature map at the fifth scale obtained after the fifth operation. This is achieved through network module F. 35 Residual features are extracted from the second feature map at the fourth scale obtained after the first operation, and then applied to network module F. 45 The output second feature map at the fifth scale is upsampled for feature extraction. The feature map after residual feature extraction is concatenated with the feature map after upsampling. The concatenated feature map is then processed... Figure 3 The convolutional kernel shown is a 3x3 convolutional layer used for downsampling feature extraction, resulting in a second feature map with a scale of the fourth scale after the fifth first operation. This is then processed through network module F. 25 Residual feature extraction is performed on the second feature maps at the third scale obtained after the first operation and the second operation, respectively, for network module F. 35 The output second feature map, at a scale of fourth, is upsampled for feature extraction, such as... Figure 4 As shown, based on the weight values ​​of the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature, a weighted concatenation is performed on the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature. The weighted concatenated feature map is then input into a convolutional layer with a 1*1 kernel, resulting in a second feature map with a scale of the third scale obtained after the fifth first operation. This is achieved through network module F. 15 Residual feature extraction is performed on the second feature maps at the second scale obtained after the first, second, and third operations, respectively, for network module F. 25 The output second feature map, at a scale of third, is upsampled for feature extraction, such as... Figure 4 As shown, based on the weight values ​​of the feature maps extracted from each residual feature and the feature maps extracted from the upsampled features, a weighted concatenation is performed on the feature maps extracted from each residual feature and the feature maps extracted from the upsampled features. The weighted concatenated feature map is then input into a convolutional layer with a 1*1 kernel, resulting in a second feature map with the second scale obtained after the fifth first operation. This is achieved through network module F. 05 Residual feature extraction is performed on the second feature maps of the first scale obtained after the first, second, third, and fourth operations, respectively, for network module F. 15 The output, at a scale of two, is used to perform upsampling feature extraction on the second feature map, such as... Figure 4 As shown, based on the weight values ​​of the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature, the feature maps extracted from each residual feature and the feature maps extracted from the upsampled feature are weighted and concatenated. The weighted concatenated feature map is then input into a convolutional layer with a kernel of 1*1 to obtain a second feature map with the first scale after the fifth first operation.

[0187] In the field of image processing, feature extraction is performed on the original image to obtain its depth and pose information. Image reconstruction is then performed based on this information, and the feature extraction capability of the neural network used to extract depth and pose information is further evaluated by comparing the reconstructed image with the original image. Therefore, embodiments of this application also provide the following optional implementation methods:

[0188] Figure 6 A structural diagram of the neural network model provided in an embodiment of this application is shown. Figure 6 As shown, the neural network model includes an initial depth estimation model and an initial pose estimation model. During the training of this neural network model: a first image can be input into the initial depth estimation model, and the initial depth information corresponding to the first image is obtained through this model. The first image and the corresponding second image are input into the initial pose estimation model, and the pose information corresponding to the first image is obtained through this model. Based on the first depth information and pose information, image reconstruction is performed to obtain the reconstructed image of the first image. The initial depth estimation model has been specifically described above; the following will describe the initial pose estimation model with reference to specific embodiments:

[0189] Optionally, each training sample may further include a second image that is temporally consecutive to the first image in the training sample, and the neural network model may further include an initial pose estimation model. During the training of the neural network model, the method may further include:

[0190] For each first image, the first image and the corresponding second image are input into the initial pose estimation model to obtain the pose information of the first image;

[0191] For each first image, image reconstruction is performed based on the pose information and first depth information of the first image to obtain the reconstructed image corresponding to the first image;

[0192] The third training loss is determined based on the first difference between each first image and its corresponding reconstructed image;

[0193] Based on the first depth information and the true depth information corresponding to each first image, the total training loss can be determined, which may include:

[0194] Based on the first depth information and the true depth information corresponding to each first image, the first training loss is determined;

[0195] The total training loss is determined based on the first training loss and the third training loss.

[0196] The second image can be either the frame preceding or following the first image; this application does not impose any restrictions on this. That is, assuming the current time is t, the first image is image I acquired at the current time. t Then the corresponding second image can be the image I acquired at the previous time step. t-1 Or the image I acquired at the next moment after the current moment. t+1 Whether the second image is the frame preceding the first image or the frame following the first image, hereinafter referred to as I t′ This represents the second image, i.e., I. t′ ∈{I t-1 I t+1}

[0197] Of course, the first image and the second image can also be two adjacent frames in a video, wherein the first image I t Located in the t-th frame of the video, corresponding to the second image I t′ This refers to either the (t+1)th or (t-1)th frame of the video.

[0198] In this implementation, the initial pose estimation model can be a pre-trained pose estimation model or a pose estimation model to be trained; this embodiment of the application does not impose any restrictions on this. Optionally, the initial pose estimation model can be implemented based on a ResNet network.

[0199] By inputting the first image and the second image into the initial pose estimation model, the convolution channels of the pose estimation model can be changed from 3 to 6. Based on the temporal relationship between the first image and the second image, the relative pose between the first image and the second image can be determined, that is, the pose information of the first image. This pose information can be represented by 6-DOF vector parameters, which include the Euler angle between the first image and the second image, as well as the positional information between the first image and the second image in three dimensions: up / down, front / back, and left / right.

[0200] Figure 7 A schematic diagram of the initial attitude estimation model in an embodiment of this application is shown. Figure 7 As shown, by using the first image I t and the corresponding second image I t′ After being input into the initial pose estimation model, the first image I is then processed by this initial pose estimation model. t and the corresponding second image It′ By performing feature extraction, the pose parameters T corresponding to the first image can be obtained. t→t′ .

[0201] After obtaining the pose information T of the first image t→t′ and the first depth information D of the first image t Next, image reconstruction is performed to obtain the reconstructed image corresponding to the first image. Specifically, assuming the intrinsic parameter matrix of the image acquisition device used to acquire the first and second images is K, the image reconstruction process can be as follows:

[0202] The first depth information D t Convert to 3D point cloud, referencing the pose information T of the first image. t→t′ That is, the mapping relationship from the second image to the first image, which transforms the 3D point cloud into the second image I. t′ Using the corresponding 2D coordinate system, the intrinsic parameter matrix K of the image acquisition device is used to convert the transformed 3D point cloud into a 2D point cloud. The resulting 2D point cloud is used as a sampling grid for bilinear interpolation from the first image. By performing bilinear interpolation on this sampling grid, the reconstructed image corresponding to the first image is obtained. The reconstructed image corresponding to the first image can be represented as:

[0203] I t′→t =I t′ <proj(D t T t→t′ K)>

[0204] Where proj() represents the first depth information D t Second image I t′ The result in the corresponding 2D coordinate system. <> represents a sampling operation. I t′→t This represents the reconstructed image corresponding to the first image. To simplify annotation, the intrinsic parameter matrix K of the image acquisition devices used to acquire all first images and corresponding second images can be considered equal.

[0205] Optionally, the total training loss can be determined based on the L1 paradigm loss function, using the first depth information and the true depth information corresponding to each first image to determine the first training loss. The third training loss can be determined based on the similarity (i.e., the first difference) between the first image and the corresponding reconstructed image. Specifically, the third training loss can be determined in the following way:

[0206]

[0207] Here, pe represents the optical reconstruction loss, such as the L1 distance in pixel space. By determining the optical reconstruction loss and training the corresponding model using it, the difference between the first image and its corresponding reconstructed image can be reduced. p This represents the minimum optical reconstruction loss corresponding to each pixel in the first image.

[0208] Specifically, pe(I a ,I b This can be achieved using L1 and SSIM loss functions:

[0209]

[0210] Here, α can be an empirical or experimental value, or it can be determined based on the actual situation. For example, α can be set to 0.85. a I represents the image to be reconstructed. b This refers to the reconstructed image obtained after reconstructing the image to be reconstructed. In the embodiments of this application, I t That is I a I t′→t That is I b .

[0211] When determining the total training loss based on the first training loss and the third training loss, the sum of the first training loss and the third training loss can be directly used as the total training loss, or the first training loss and the third training loss can be weighted and summed based on the weight values ​​corresponding to the first training loss and the third training loss, respectively, to obtain the total training loss.

[0212] Currently, the total training loss can also be determined based on the first training loss, the second training loss, and the third training loss. Specifically, the total training loss can be directly determined by the sum of the first, second, and third training losses, or it can be obtained by weighted summation of the first, second, and third training losses based on their respective weight values.

[0213] When the first training loss, second training loss, and third training loss are weighted and summed based on their respective weight values ​​to obtain the total training loss, the determined total training loss can be:

[0214]

[0215] Among them, L p Indicates the third training loss. Indicates the first training loss. Let μ represent the weight of the third training loss, γ represent the weight of the first training loss, and β represent the weight of the second training loss.

[0216] By fusing feature maps of different resolutions, a higher-resolution feature map is obtained. Based on the obtained higher-resolution feature map, first depth information is obtained. This first depth information is further processed, and image reconstruction is performed based on the first depth information and the pose information obtained from the initial pose estimation model. This results in a higher-resolution reconstructed image corresponding to the first image. Based on the differences between each first image and its corresponding reconstructed image, a third training loss is determined. The total training loss is determined by combining the third training loss obtained from further processing with the first training loss. During the training of the neural network model based on this total training loss, a deeper level of training loss can be combined to train the neural network model, thereby making the obtained trained depth estimation model perform better, be more robust, and better meet practical needs.

[0217] Considering that the image reconstruction process described above assumes that the intrinsic parameter matrix K of the image acquisition device used to acquire all first images and corresponding second images is equal, i.e., the image acquisition device moves in a static scene, the determined third training loss may contain errors. Especially when the target object in the first image is moving at a speed similar to the image acquisition device, or when other non-target objects in the first image are moving while the image acquisition device has stopped moving (i.e., the other non-target objects are stationary in the coordinate system of the image acquisition device), this can lead to objects that are stationary relative to the coordinate system of the image acquisition device having infinite depth, affecting the training process of the initial depth estimation model. Therefore, embodiments of this application also provide the following implementation methods for filtering the depth information of pixels in the first image that are moving relatively stationary with respect to the image acquisition device:

[0218] Optionally, the method may further include:

[0219] For a first image, determine a second difference between the first image and the corresponding second image;

[0220] The weight values ​​of the third training loss are determined based on the magnitude relationship between the first difference and the second difference corresponding to each first image.

[0221] Determining the total training loss based on the first and third training losses can include:

[0222] Based on the weight values ​​of the third training loss, the first and third training losses are weighted and summed to obtain the total training loss.

[0223] In this implementation, the second difference between each first image and its corresponding second image can be determined by referring to the method described above, which is based on the first difference between each first image and its corresponding reconstructed image. The determined second difference L p ′ can be:

[0224]

[0225] Optionally, if the first difference corresponding to each first image is less than the corresponding second difference, the weight value of the third training loss is determined to be 1, i.e., a third training loss exists, or the third training loss cannot be ignored; if the first difference corresponding to each first image is greater than or equal to the corresponding second difference, the weight value of the third training loss is determined to be 0, i.e., no third training loss exists, or the third training loss can be ignored, or the determined third training loss has too low accuracy. Specifically, the weight value μ of the third training loss can be represented in the following way:

[0226]

[0227] In this context, [] represents Iverson brackets, where 1 is represented if the condition inside the brackets is met, and 0 is represented if the condition inside the brackets is not met.

[0228] The above method prevents the image acquisition device from moving at a similar speed to the objects in the acquired first image, filtering out the depth information of pixels in the first image that are relatively stationary compared to the image acquisition device. This allows all pixels in the first image to be masked when the image acquisition device is stationary (a situation rarely encountered in practice). When an object in the first image moves at the same speed as the image acquisition device, the pixels corresponding to that moving object can be masked. This further makes the determined third training loss and total training loss more reflective of the unstable model parameters of the neural network model.

[0229] Taking the acquisition of multiple training samples from the KITTI2015 dataset as an example, in this embodiment, the Eigen (a file library) partitioning method can also be used to divide the multiple training samples into a training set and a test set. Taking 100k training samples as an example, the ratio of the training set to the test set can be 3:7. By training the initial neural network model based on the training set and the test set, the trained deep estimation model can achieve better accuracy, stronger generalization ability, and higher coverage and precision.

[0230] For each first image in the training set, the same intrinsic parameter matrix of the image acquisition device can be used. For all first images in the training set, the principal point of the image acquisition device is set to the average of the center and focal points of the first image. All first images are then scaled to a fixed scale; for example, the scale of each first image can be 196*640.

[0231] When training a neural network model, an initial depth estimation network can be trained first. For example, the training epochs for the initial depth estimation model can be 20 (i.e., training 20 times based on all the first images), the batch size can be 12 (i.e., the first image in a single training iteration is 12), and the learning rate of the initial depth estimation model can be 10. -3 Extensive data analysis shows that after 15 training epochs, the learning rate of the initial depth estimation model can be decayed to 10. -4 .

[0232] After training the initial depth estimation model, train the initial pose estimation model. For example, the training epochs for the initial pose estimation model can be 20, the batch size can be 12, and the learning rate of the initial pose estimation model can be 10. -3 Extensive data analysis shows that after 15 training epochs, the learning rate of the initial pose estimation model can be decayed to 10. -4 .

[0233] After training the initial depth estimation network and the initial pose estimation model, the current depth estimation network and the current pose estimation model can be jointly trained (i.e., trained as a neural network model) by combining the first depth information output by the current depth estimation model and the pose information output by the current pose estimation model. The training cycle for the current neural network model can be 10, with a batch size of 12, until the learning rate of the current neural network model decays to 10. -4 .

[0234] During the training of the initial pose estimation model, the model parameters of the initial depth estimation model can be frozen. Optionally, the model parameters of the initial depth estimation model can be frozen by setting the learning rate of each network module in the initial depth estimation model to zero, thereby setting the weights of the initial depth estimation model to zero or keeping them fixed.

[0235] It is important to note that the training period for the initial depth estimation network should be greater than or equal to the training period for the current neural network model, and the training period for the initial pose estimation model should also be greater than or equal to the training period for the current neural network model.

[0236] To further verify the effectiveness of the training method of the depth estimation model provided in the real-time example of this application, the embodiments of this application also perform depth prediction on each first image in the test set based on the depth estimation model trained on the training set, and obtain the predicted depth information corresponding to each first image. Figure 8 This illustration shows a schematic diagram of depth information obtained from images in a dataset, applicable to the scenarios described in this application. For example... Figure 8 As shown, based on the trained depth estimation model provided in the embodiments of this application, by fusing features of different resolutions of each image, the contours of each object in the depth information corresponding to each image can be made clearer, thereby achieving more effective extraction of depth information from each image.

[0237] In this embodiment, the similarity between the predicted depth information and the actual depth information corresponding to each first image is further determined to verify the performance of the depth estimation model trained on the training set. It should be understood that the better the performance of the depth estimation model, the more accurate the depth information of the image to be processed determined by the training method based on the depth estimation model, and the better the effectiveness of the training method of the depth estimation model.

[0238] Considering that the similarity between the predicted depth information and the true depth information corresponding to each first image can be evaluated based on multiple indicators, including but not limited to absolute relative error (abs_rel), squared relative error (sq_rel), root mean square error (rmse), logarithmic root mean square error (rmse_log), and accuracy (a1, a2, a3), this embodiment of the application uses these indicators as examples to determine the similarity between the predicted depth information and the true depth information corresponding to each first image. Among them, abs_rel, sq_rel, rmse, and rmse_log can be determined according to the following formula:

[0239]

[0240]

[0241]

[0242]

[0243] The precisions a1, a2, and a3 are determined in the following ways:

[0244]

[0245] Where N is the total number of pixels, D i It is the estimated depth value of the i-th pixel. This is the true depth value corresponding to the i-th pixel. T is the threshold, and the three thresholds corresponding to a1, a2, and a3 are 1.25(a1), 1.25, and 1.25, respectively. 2 (a2), 1.25 3 (a3)

[0246] After determining the various indicators corresponding to the depth estimation model in this application embodiment, the indicators corresponding to this depth estimation model were also compared with those of depth estimation models in related technologies. The comparison results are as follows:

[0247] Model abs_rel sq_rel rmse rmse_log a1 a2 a3 Related technologies 0.115 0.903 4.863 0.193 0.877 0.959 0.981 This application 0.105 0.770 4.625 0.185 0.889 0.962 0.982

[0248] Considering that the smallest test values ​​for abs_rel, sq_rel, rmse, and rmse_log indicate a better model performance, and the largest test values ​​for a1, a2, and a3 indicate a better model performance, the table above shows that the depth estimation model provided in this application embodiment has a significant performance improvement compared to depth estimation models in related technologies. Correspondingly, the training method of the depth estimation model provided in this application embodiment is also more effective, and the depth information of the image to be processed determined by the training method based on this depth estimation model is more accurate, which can better meet practical needs.

[0249] This application also provides a method for determining depth information. Figure 9 A flowchart of a depth information determination method provided in an embodiment of this application is shown. Figure 9 As shown, the method includes steps S110 to S120.

[0250] S110: Obtain the image to be processed.

[0251] S120: Input the image to be processed into the trained depth estimation model, and obtain the depth information of the image to be processed through the model.

[0252] The trained depth estimation model is obtained by training the neural network model in the above manner.

[0253] In this implementation, the image to be processed can be any image, and this application does not impose any restrictions on it. For example, the image to be processed can be any image acquired by an image acquisition device, or any image obtained from a certain image data.

[0254] It should be understood that after obtaining the trained depth estimation model, the second feature map obtained from the last first operation based on the image to be processed, which corresponds to the first feature map with the largest scale, can be directly used as the target feature map to obtain the depth information corresponding to the image to be processed, without having to obtain the depth information of the target feature maps at other scales.

[0255] As mentioned earlier, a well-trained depth estimation model can effectively extract depth information from the input image (i.e., the image to be processed). Furthermore, when determining the depth information of the image to be processed using this well-trained depth estimation model, more accurate depth information can be obtained by fusing the multi-resolution features obtained from each feature extraction of the image to be processed, thus avoiding the loss of some image information during the depth determination process.

[0256] To facilitate understanding of the application value of the depth information determination method provided in the embodiments of this application, the following description is based on a specific application scenario.

[0257] Figure 10 A schematic diagram of a depth information determination system provided in an embodiment of this application is shown, such as... Figure 10 As shown, the system includes a terminal device 81, a network 82, an application server 83, and a model training server 84. The terminal device 81 communicates with the application server 83 through the network 82. The application server 83 and the model training server 84 can interact with each other; for example, the application server 83 can receive a trained image reconstruction model sent by the model training server 84. The terminal device 81 can be a mobile phone, iPad, computer, vehicle terminal, etc. Optionally, an image acquisition device can be installed in the terminal device 81 to acquire images to be processed. The terminal device 81 can also connect to other external image acquisition devices to acquire images corresponding to it.

[0258] The model training server 84 can be used to train a neural network model based on multiple training samples to obtain a trained depth estimation model. The trained depth estimation model is then sent to the application server 83, which deploys it. The application server 83 can then execute the depth information determination method provided in this embodiment based on the deployed trained depth estimation model.

[0259] The depth information determination system provided in this application can be applied to autonomous driving processes. The following is a combination of... Figure 10 The depth information determination system shown illustrates the process of the depth information determination method in this application scenario. In this depth information determination method, the application server 83 in the depth information determination system can be an in-vehicle server, and the terminal device 81 in the depth information determination system can be an in-vehicle terminal of an autonomous vehicle. Figure 11 The diagram illustrates the signaling interaction of the depth information determination method in the application scenario to which the embodiments of this application are applicable. Figure 12A schematic diagram illustrates a depth information determination method applicable to the scenario described in this application. For example... Figure 11 As shown, the method may include the following steps S11 to S13.

[0260] Step S11: Obtain multiple training samples, each training sample including a first image, a second image corresponding to the first image, and the true depth information corresponding to the first image. Input the obtained multiple training samples into the model training server 84.

[0261] Step S12: Based on multiple training samples, train the neural network model through the model training server 84 to obtain the trained depth estimation model, and send the trained depth estimation model to the vehicle server 83.

[0262] Step S13: The image acquisition device based on the vehicle terminal 81 acquires images of vehicles that are close to the current autonomous vehicle and sends the vehicle images to the vehicle server 83.

[0263] Step S14: Based on the trained depth estimation model described above, the vehicle server 83 determines the depth information in the vehicle image (i.e., Figure 12 (The process of obtaining depth information shown) obtains the actual distance between the autonomous vehicle and surrounding vehicles, and sends the relationship between the actual distance and the preset safe distance to the vehicle terminal 81.

[0264] Step S15: If the actual distance is greater than or equal to the preset safe distance, the vehicle terminal 81 controls the autonomous vehicle to continue driving along the original driving route;

[0265] If the actual distance is less than the preset safe distance, the on-board terminal 81 controls the autonomous vehicle to move away from the vehicle in the vehicle image.

[0266] Based on the same principle as the training method of the depth estimation model provided in the embodiments of this application, the embodiments of this application also provide a training device for the depth estimation model. Figure 13 This illustration shows a schematic diagram of a training device for a depth estimation model according to an embodiment of this application. The device 10 may include: a training sample acquisition module 101 and a depth estimation model training module 102, wherein...

[0267] The training sample acquisition module 101 is used to acquire multiple training samples, each training sample including a first image and the true depth information of the first image;

[0268] The depth estimation model training module 102 is used to iteratively train the neural network model based on multiple training samples until the total training loss of the neural network model meets a preset condition. The neural network model includes an initial depth estimation model, and the depth estimation model that meets the preset condition is used as the trained depth estimation model. The total training loss is determined in the following way:

[0269] For each first image, features are extracted from the first image using an initial depth estimation model to obtain first feature maps of at least two scales corresponding to the first image; the target feature map corresponding to the first image is obtained by fusing the first feature maps; and first depth information is obtained based on the target feature map.

[0270] The total training loss is determined based on the first depth information and the true depth information corresponding to each first image.

[0271] Optionally, when the depth estimation model training module 102 obtains the target feature map corresponding to the first image by fusing the first feature maps corresponding to the first image, it is specifically used for:

[0272] The first feature map corresponding to the first image is used as the initial feature map set. The first operation is performed a set number of times. The second feature map obtained from the last first operation is used as the target feature map.

[0273] When performing the first operation, the depth estimation model training module 102 is specifically used for:

[0274] Each feature map in the feature map set, except for the feature map with the smallest scale, is taken as a feature map to be processed. For each feature map to be processed, the feature map to be processed is fused with a feature map with a smaller scale to obtain a second feature map with the same scale as the feature map to be processed. The second feature map corresponding to each feature map to be processed is taken as the feature map set corresponding to the next first operation.

[0275] Optionally, for each first image, the depth estimation model training module 102 is further configured to:

[0276] For at least one feature map in each of the second feature maps obtained in the first operation, the second depth information of the first image corresponding to the feature map is obtained based on each feature map in the first operation.

[0277] The second training loss is determined based on the real depth information and the second depth information corresponding to each first image;

[0278] When determining the total training loss based on the first depth information and the true depth information corresponding to each first image, the depth estimation model training module 102 is specifically used for:

[0279] Based on the first depth information and the true depth information corresponding to each first image, the first training loss is determined;

[0280] The total training loss is obtained based on the first training loss and the second training loss.

[0281] Optionally, each training sample may further include a second image that is temporally consecutive to the first image in the training sample, and the neural network model may further include an initial pose estimation model. During the training of the neural network model, the depth estimation model training module 102 is also used for:

[0282] For each first image, the first image and the corresponding second image are input into the initial pose estimation model to obtain the pose information of the first image;

[0283] For each first image, image reconstruction is performed based on the pose information and first depth information of the first image to obtain the reconstructed image corresponding to the first image;

[0284] The third training loss is determined based on the first difference between each first image and its corresponding reconstructed image;

[0285] When determining the total training loss based on the first depth information and the true depth information corresponding to each first image, the depth estimation model training module 102 is specifically used for:

[0286] Based on the first depth information and the true depth information corresponding to each first image, the first training loss is determined;

[0287] The total training loss is determined based on the first training loss and the third training loss.

[0288] Optionally, the depth estimation model training module 102 is also used for:

[0289] For a first image, determine a second difference between the first image and the corresponding second image;

[0290] The weight values ​​of the third training loss are determined based on the magnitude relationship between the first difference and the second difference corresponding to each first image.

[0291] When determining the total training loss based on the first training loss and the third training loss, the depth estimation model training module 102 is specifically used for:

[0292] Based on the weight values ​​of the third training loss, the first and third training losses are weighted and summed to obtain the total training loss.

[0293] Optionally, for each first image, the depth estimation model training module 102, when performing the first operation, is also used to:

[0294] For the feature map with the smallest scale in the feature map set, residual feature extraction is performed on the feature map to obtain a second feature map with the same scale as the feature map.

[0295] For each feature map to be processed, the depth estimation model training module 102, when fusing the feature map to be processed with a feature map of a smaller scale to obtain a second feature map corresponding to the scale of the feature map to be processed, specifically performs the following:

[0296] Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed. The third feature map is then concatenated with the feature map to be processed with a smaller scale to obtain the fourth feature map corresponding to the feature map to be processed. Feature extraction is performed on the fourth feature map to obtain the second feature map with the same scale as the feature map to be processed.

[0297] Optionally, for each first image, the number of operations is set to at least three. For the third first operation and each subsequent first operation, when the depth estimation model training module 102 fuses the feature map to be processed with a feature map of a smaller scale to obtain a second feature map with a scale corresponding to the feature map to be processed, it is specifically used for:

[0298] Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed;

[0299] Determine the second feature map to be processed in the feature map set corresponding to at least one first operation prior to this first operation, whose scale corresponds to the feature map to be processed;

[0300] Residual features are extracted from each second feature map to be processed to obtain the fourth feature map corresponding to each second feature map to be processed.

[0301] Based on the correlation between the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed, the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed are determined respectively.

[0302] Based on the weights corresponding to the third feature map, each of the fourth feature maps, and the feature maps with a scale smaller than the feature map to be processed, the third feature map, each of the fourth feature maps, and the feature maps with a scale smaller than the feature map to be processed are fused to obtain a second feature map with a scale corresponding to the feature map to be processed.

[0303] Optionally, for each first image, the depth estimation model training module 102, when using the second feature map obtained from the last first operation as the target feature map, specifically performs the following:

[0304] For each of at least two scales, the second feature map obtained after the last first operation corresponding to that scale is determined as the target feature map for that scale;

[0305] For each first image, when the depth estimation model training module 102 obtains the first depth information of the first image based on the target feature map, it is specifically used for:

[0306] First depth information is obtained based on each feature map in at least one feature map of the target feature map at each scale.

[0307] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0308] Based on the same principle as the training method and apparatus for the depth estimation model provided in the embodiments of this application, the embodiments of this application also provide an electronic device (such as a server), which may include a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method provided in any optional embodiment of this application.

[0309] Optionally, Figure 14 A schematic diagram of the structure of an electronic device to which this application embodiment applies is shown, such as... Figure 14 As shown, Figure 14 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0310] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0311] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0312] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0313] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0314] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0315] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0316] The terms "first," "second," "third," "fourth," "1," "2," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0317] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0318] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A training method for a depth estimation model, characterized in that, include: Multiple training samples are acquired, each training sample including a first image and the true depth information of the first image; The neural network model is iteratively trained based on the multiple training samples until the total training loss of the neural network model meets a preset condition. The neural network model includes an initial depth estimation model, and the initial depth estimation model that meets the preset condition is used as the trained depth estimation model. The total training loss is determined in the following way: For each of the first images, features are extracted from the first image using the initial depth estimation model to obtain first feature maps of at least two scales corresponding to the first image; the first feature maps corresponding to the first image are used as an initial feature map set, and a first operation is performed a set number of times, with the second feature map obtained from the last first operation being used as the target feature map corresponding to the first image; based on the target feature map, the first depth information of the first image is obtained. The first operation includes: taking each feature map in the feature map set except for the feature map with the smallest scale as a feature map to be processed; for each feature map to be processed, fusing the feature map to be processed with a feature map with a smaller scale to obtain a second feature map with a scale corresponding to the feature map to be processed; and taking the second feature map corresponding to each feature map to be processed as the feature map set corresponding to the next first operation. For each of the first images, the set number of times is at least two. For at least one feature map in each of the second feature maps obtained in the first operation, the second depth information of the first image corresponding to the feature map is obtained based on each feature map in the at least one feature map. The second training loss is determined based on the real depth information and the second depth information corresponding to each of the first images; Based on the first depth information and the true depth information corresponding to each of the first images, the first training loss is determined; The total training loss is obtained based on the first training loss and the second training loss.

2. The method according to claim 1, characterized in that, Each training sample further includes a second image that is temporally consecutive to the first image in the training sample. The neural network model further includes an initial pose estimation model. During the training of the neural network model, the method further includes: For each of the first images, the first image and the corresponding second image are input into the initial pose estimation model to obtain the pose information of the first image; For each of the first images, image reconstruction is performed based on the pose information and the first depth information of the first image to obtain the reconstructed image corresponding to the first image; Based on the first difference between each of the first images and the corresponding reconstructed images, a third training loss is determined; The process of obtaining the total training loss based on the first training loss and the second training loss includes: The total training loss is determined based on the first training loss, the second training loss, and the third training loss.

3. The method according to claim 2, characterized in that, The method further includes: For a given first image, determine a second difference between the first image and the corresponding second image; The weight value of the third training loss is determined based on the magnitude relationship between the first difference and the second difference corresponding to each of the first images; Determining the total training loss based on the first training loss, the second training loss, and the third training loss includes: Based on the weight values ​​corresponding to the first training loss, the second training loss, and the third training loss respectively, the first training loss, the second training loss, and the third training loss are weighted and summed to obtain the total training loss.

4. The method according to claim 1, characterized in that, For each of the first images, the first operation further includes: For the feature map with the smallest scale in the feature map set, residual feature extraction is performed on the feature map to obtain a second feature map with the same scale as the feature map. For each of the aforementioned feature maps to be processed, the step of fusing the feature map to be processed with feature maps of a smaller scale to obtain a second feature map with a scale corresponding to the feature map to be processed includes: Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed. The third feature map is concatenated with a feature map with a smaller scale than the feature map to be processed to obtain the fourth feature map corresponding to the feature map to be processed. Feature extraction is performed on the fourth feature map to obtain the second feature map with the same scale as the feature map to be processed.

5. The method according to claim 1, characterized in that, For each of the first images, the set number of operations is at least three. For the third first operation and each subsequent first operation, the step of fusing the feature map to be processed with a feature map of a smaller scale to obtain a second feature map with a scale corresponding to the feature map to be processed includes: Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed; Determine the second feature map to be processed in the feature map set corresponding to at least one first operation prior to this first operation, whose scale corresponds to the feature map to be processed; Residual features are extracted from each of the second feature maps to be processed to obtain the fourth feature map corresponding to each of the second feature maps to be processed. Based on the correlation between the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed, the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed are determined respectively. Based on the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed, the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed are fused to obtain a second feature map with a scale corresponding to the feature map to be processed.

6. The method according to claim 1, characterized in that, For each of the first images, the step of using the second feature map obtained from the last first operation as the target feature map includes: For each of the at least two scales, the second feature map obtained from the last first operation corresponding to that scale is determined as the target feature map for that scale; For each of the first images, obtaining the first depth information of the first image based on the target feature map includes: The first depth information is obtained based on each feature map in at least one feature map of the target feature maps at each scale.

7. A training device for a depth estimation model, characterized in that, include: The training sample acquisition module is used to acquire multiple training samples, each training sample including a first image and the true depth information of the first image; A depth estimation model training module is used to iteratively train a neural network model based on the multiple training samples until the total training loss of the neural network model meets a preset condition. The neural network model includes an initial depth estimation model, and the depth estimation model that meets the preset condition is used as the trained depth estimation model. The total training loss is determined in the following way: For each of the first images, features are extracted from the first image using the initial depth estimation model to obtain first feature maps of at least two scales corresponding to the first image; the first feature maps corresponding to the first image are used as an initial feature map set, and a first operation is performed a set number of times, with the second feature map obtained from the last first operation being used as the target feature map corresponding to the first image; based on the target feature map, the first depth information of the first image is obtained. The first operation includes: taking each feature map in the feature map set except for the feature map with the smallest scale as a feature map to be processed; for each feature map to be processed, fusing the feature map to be processed with a feature map with a smaller scale to obtain a second feature map with a scale corresponding to the feature map to be processed; and taking the second feature map corresponding to each feature map to be processed as the feature map set corresponding to the next first operation. For each of the first images, the set number of times is at least two. For at least one feature map in each of the second feature maps obtained in the first operation, the second depth information of the first image corresponding to the feature map is obtained based on each feature map in the at least one feature map. The second training loss is determined based on the real depth information and the second depth information corresponding to each of the first images; Based on the first depth information and the true depth information corresponding to each of the first images, the first training loss is determined; The total training loss is obtained based on the first training loss and the second training loss.

8. The apparatus according to claim 7, characterized in that, Each training sample also includes a second image that is temporally consecutive to the first image in the training sample. The neural network model also includes an initial pose estimation model. During the training of the neural network model, the depth estimation model training module is further configured to: For each of the first images, the first image and the corresponding second image are input into the initial pose estimation model to obtain the pose information of the first image; For each of the first images, image reconstruction is performed based on the pose information and the first depth information of the first image to obtain the reconstructed image corresponding to the first image; Based on the first difference between each of the first images and the corresponding reconstructed images, a third training loss is determined; When the depth estimation model training module obtains the total training loss based on the first training loss and the second training loss, it is specifically used for: The total training loss is determined based on the first training loss, the second training loss, and the third training loss.

9. The apparatus according to claim 8, characterized in that, The depth estimation model training module is also used for: For a given first image, determine a second difference between the first image and the corresponding second image; The weight value of the third training loss is determined based on the magnitude relationship between the first difference and the second difference corresponding to each of the first images; When determining the total training loss based on the first training loss, the second training loss, and the third training loss, the depth estimation model training module is specifically used for: Based on the weight values ​​corresponding to the first training loss, the second training loss, and the third training loss respectively, the first training loss, the second training loss, and the third training loss are weighted and summed to obtain the total training loss.

10. The apparatus according to claim 7, characterized in that, For each first image, the depth estimation model training module is further configured to: For the feature map with the smallest scale in the feature map set, residual feature extraction is performed on the feature map to obtain a second feature map with the same scale as the feature map. For each of the aforementioned feature maps to be processed, when the depth estimation model training module fuses the feature map to be processed with a feature map of a smaller scale to obtain a second feature map corresponding to the scale of the feature map to be processed, it is specifically used for: Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed. The third feature map is concatenated with a feature map with a smaller scale than the feature map to be processed to obtain the fourth feature map corresponding to the feature map to be processed. Feature extraction is performed on the fourth feature map to obtain the second feature map with the same scale as the feature map to be processed.

11. The apparatus according to claim 7, characterized in that, For each of the first images, the set number of operations is at least three. For the third first operation and each subsequent first operation, when the depth estimation model training module fuses the feature map to be processed with a feature map of a smaller scale to obtain a second feature map with a scale corresponding to the feature map to be processed, it is specifically used for: Residual feature extraction is performed on the feature map to be processed to obtain the third feature map corresponding to the feature map to be processed; Determine the second feature map to be processed in the feature map set corresponding to at least one first operation prior to this first operation, whose scale corresponds to the feature map to be processed; Residual features are extracted from each of the second feature maps to be processed to obtain the fourth feature map corresponding to each of the second feature maps to be processed. Based on the correlation between the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed, the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed are determined respectively. Based on the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed, the weights corresponding to the third feature map, each of the fourth feature maps, and feature maps with a scale smaller than the feature map to be processed are fused to obtain a second feature map with a scale corresponding to the feature map to be processed.

12. The apparatus according to claim 7, characterized in that, For each of the first images, when the depth estimation model training module uses the second feature map obtained from the last first operation as the target feature map, it specifically performs the following: For each of the at least two scales, the second feature map obtained from the last first operation corresponding to that scale is determined as the target feature map for that scale; For each of the first images, when the depth estimation model training module obtains the first depth information of the first image based on the target feature map, it specifically performs the following: The first depth information is obtained based on each feature map in at least one feature map of the target feature maps at each scale.

13. A method for determining depth information, characterized in that, include: Obtain the image to be processed; The image to be processed is input into a trained depth estimation model to obtain the depth information of the image to be processed; The trained depth estimation model is obtained by training the neural network model using the method described in any one of claims 1 to 6.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-6 or claim 13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6 or claim 13.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6 or claim 13.

Images