Model training method and device, depth estimation method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a model training method and device, a depth estimation method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a plurality of image samples and a depth image of each image sample; for each depth image, constructing Gaussian distribution corresponding to each pixel point according to the depth value of each pixel point in the depth image; performing depth estimation on the image sample by using a depth estimation model to obtain depth probability distribution of each pixel point; determining a loss value based on the depth probability distribution and the Gaussian distribution; and obtaining a trained depth estimation model based on the loss value. Through the method and the device, the unbiased transformation process of the loss function can be constructed through Gaussian distribution, the loss function of the model is improved, the convergence speed of the model can be improved, and thus the model training efficiency is improved.

Description

Technical Field

[0001] The present application relates to deep learning technology, and in particular to a model training method, a depth estimation method, an apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] Image depth estimation is a research direction in the field of computer vision. Its core task is to estimate the depth information of each point in the scene by analyzing images or image sequences. However, early image depth estimation was mainly based on geometric and physical principles, such as triangulation and structured light. With the development of deep learning technology, deep learning-based depth estimation models have begun to emerge. These models can directly predict depth information from images through training with large-scale data sets. However, related technologies have poor performance in model training. During the training process, gradient explosion is very likely to occur, resulting in the failure of the model to converge. Summary of the invention

[0003] The embodiments of the present application provide a model training method, a depth estimation method, an apparatus, an electronic device, and a computer-readable storage medium, which can construct an unbiased transformation process of the loss function through Gaussian distribution, improve the loss function of the model, increase the convergence speed of the model, and thus improve the model training efficiency.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a model training method, the method comprising:

[0006] Acquire a plurality of image samples and a depth image of each of the image samples;

[0007] For each depth image, constructing a Gaussian distribution corresponding to each pixel in the depth image according to the depth value of each pixel;

[0008] Using a depth estimation model to perform depth estimation on the image sample to obtain a depth probability distribution of each pixel;

[0009] Determining a loss value based on the depth probability distribution and the Gaussian distribution;

[0010] Based on the loss value, the depth estimation model is updated to obtain a trained depth estimation model.

[0011] The present invention provides a depth estimation method, including:

[0012] Obtain an image to be estimated and a trained depth estimation model, wherein the depth estimation model is trained according to the model training method provided in an embodiment of the present application;

[0013] The trained depth estimation model is used to perform depth estimation on the image to be estimated to obtain a depth estimation result.

[0014] The present application provides a model training device, including:

[0015] A first acquisition module, used to acquire a plurality of image samples and a depth image of each of the image samples;

[0016] A construction module, configured to construct, for each depth image, a Gaussian distribution corresponding to each pixel point according to a depth value of each pixel point in the depth image;

[0017] A training module is used to use a depth estimation model to perform depth estimation on the image sample to obtain a depth probability distribution for each pixel; based on the depth probability distribution and the Gaussian distribution, a loss value is determined; based on the loss value, the depth estimation model is updated to obtain a trained depth estimation model.

[0018] The present invention provides a depth estimation device, comprising:

[0019] A second acquisition module is used to acquire an image to be estimated and a trained depth estimation model, wherein the depth estimation model is trained according to the model training method provided in an embodiment of the present application;

[0020] The estimation module is used to perform depth estimation on the image to be estimated by using the trained depth estimation model to obtain a depth estimation result.

[0021] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0022] A memory for storing computer executable instructions or computer programs;

[0023] The processor is used to implement the model training method or depth estimation method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0024] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which are used to implement the model training method or depth estimation method provided in the embodiment of the present application when executed by a processor.

[0025] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the model training method or depth estimation method provided in the embodiment of the present application is implemented.

[0026] The embodiments of the present application have the following beneficial effects:

[0027] First, multiple image samples and depth images of each image sample are obtained. Then, for each depth image, a Gaussian distribution corresponding to each pixel can be constructed according to the depth value of each pixel in the depth image. The Gaussian distribution can be used as a supervisory signal in the model training process. Then, the depth estimation model is used to estimate the depth of the image sample to obtain the depth probability distribution of each pixel. Then, based on the depth probability distribution and the Gaussian distribution, the loss value is determined. Finally, based on the loss value, the depth estimation model is updated to obtain the trained depth estimation model. The Gaussian distribution corresponding to each pixel can be used as a supervisory signal, and the loss value can be determined together with the depth probability distribution of each pixel. Thus, in the process of updating the depth estimation model according to the loss value, the depth probability distribution will continue to approach the Gaussian distribution. Compared with constructing the loss function based on the comparison between the estimated depth image and the actual depth image, this is an unbiased transformation, which can make the loss value and gradient smaller, reduce the probability of gradient explosion, improve the stability and convergence speed of the model training process, and thus improve the model training effect and training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of the architecture of a depth estimation system provided in an embodiment of the present application;

[0029] Figure 2A is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0030] Figure 2B is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0031] Figure 3 It is a first flow chart of the model training method provided in the embodiment of the present application;

[0032] Figure 4 It is a second flow chart of the model training method provided in the embodiment of the present application;

[0033] Figure 5 It is a third flow chart of the model training method provided in the embodiment of the present application;

[0034] Figure 6 It is a fourth flow chart of the model training method provided in the embodiment of the present application;

[0035] Figure 7 It is a fifth flow chart of the model training method provided in the embodiment of the present application;

[0036] Figure 8 is a first flow chart of a depth estimation method provided in an embodiment of the present application;

[0037] Fig. 9 is a second flow chart of the depth estimation method provided in an embodiment of the present application;

[0038] Fig.10 This is a third flow chart of the depth estimation method provided in an embodiment of the present application.

[0039] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0041] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0044] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0045] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0046] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0047] 1) Depth image: It is a grayscale image in which the pixel value of each pixel is the depth value, which represents the distance of the object at that pixel from the camera.

[0048] 2) Depth value (Ground Truth): It is the distance between the object at the pixel point in the depth image and the camera.

[0049] 3) Gaussian distribution: Also known as normal distribution, it is a very important continuous probability distribution in statistics. The probability density function of Gaussian distribution can describe the probability density of a random variable near a certain value point. For Gaussian distribution, its probability density function is determined by two parameters: mean (also called expected value) and standard deviation. The mean, which is at the center of the Gaussian distribution, represents the average value of the Gaussian distribution. In the probability density function, the mean also determines the center position of the Gaussian distribution curve. The standard deviation, a statistic that measures the degree of dispersion of Gaussian distribution data, determines the width of the probability density function. The larger the standard deviation, the more dispersed the data distribution and the wider the curve; the smaller the standard deviation, the more concentrated the data distribution and the narrower the curve. For example, the depth value of a pixel can be expected, and the standard deviation can be set based on experience to construct the Gaussian distribution corresponding to the pixel.

[0050] 4) Binocular depth estimation model: A model that takes two images as input and estimates the depth value of each pixel in the image through a convolutional neural network.

[0051] 5) Kullback-Leibler divergence (KL divergence): also known as relative entropy, is a measure used in information theory to measure the difference between two probability distributions. It measures the difference between one probability distribution P and another probability distribution Q. The larger the value of KL divergence, the more significant the difference between the two distributions.

[0052] In the related art, when training a depth estimation model, a loss function is usually constructed based on the comparison between the estimated depth image and the actual depth image. When updating the parameters of the model, it is necessary to obtain the derivative of the loss function with respect to the parameters. The derivative expression is as follows:

[0053]

[0054] Among them, L is the loss function, e i is the model parameter, and D is the depth value of depth estimation.

[0055] Mathematically transforming formula (1) we get:

[0056]

[0057] Among them, L is the loss function, e i is the model parameter, D is the depth value of depth estimation, d i is the index on the D axis.

[0058] According to formula (2), when d i The larger the gap between -D, the greater the loss value of the corresponding model, which will cause the obtained gradient to become larger. When the gradient becomes larger, it is easy to cause gradient explosion, which will lead to unstable training process and may even cause the model to fail to converge.

[0059] The embodiments of the present application provide a model training method, a depth estimation method, a device, an electronic device, and a computer-readable storage medium, which can construct an unbiased transformation process of the loss function through Gaussian distribution, improve the loss function of the model, and improve the convergence speed of the model, thereby improving the model training efficiency. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, vehicle-mounted terminals, intelligent robots, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a server will be described.

[0060] For example, see Figure 1 , Figure 1 It is an architectural diagram of the depth estimation system provided in an embodiment of the present application. In order to support an application that improves the efficiency of model training, the server 400 is connected to the terminal 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0061] In the depth estimation system 100, the server 400 is used to obtain multiple image samples and depth images of each image sample, and then, for each depth image, according to the depth value of each pixel in the depth image, construct a Gaussian distribution corresponding to each pixel, and then use the depth estimation model to perform depth estimation on the image sample to obtain the depth probability distribution of each pixel, and then determine the loss value based on the depth probability distribution and the Gaussian distribution, and finally update the depth estimation model based on the loss value to obtain the trained depth estimation model. In the process of updating the depth estimation model according to the loss value, the depth probability distribution will continue to approach the Gaussian distribution. Compared with constructing a loss function based on the comparison between the estimated depth image and the actual depth image, this is an unbiased transformation, which can make the loss value and gradient smaller, reduce the probability of gradient explosion, and improve the stability of the model training process. Therefore, the unbiased transformation process of constructing the loss function through the Gaussian distribution can improve the loss function of the model, improve the convergence speed of the model, and thus improve the efficiency of model training.

[0062] The server 400 is also used to receive the image to be estimated sent by the terminal 200, and then use the trained depth estimation model to perform depth estimation on the image to be estimated to obtain a depth estimation result, and determine the direction and speed of movement based on the depth estimation result. The server 400 sends the determined direction and speed of movement to the terminal 200.

[0063] In some embodiments, the server 400 can also send the trained depth estimation model to the terminal 200. After acquiring the image to be estimated, the terminal 200 uses the trained depth estimation model to perform depth estimation on the image to be estimated, obtains a depth estimation result, and determines the moving direction and moving speed based on the depth estimation result, and then controls the movement of the terminal 200 based on the moving direction and moving speed.

[0064] In some embodiments, the server 400 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0065] The structure of the electronic device provided in the embodiment of the present application is further described below. Figure 2A , Figure 2A is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2AThe electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2A Various buses are labeled as bus system 540 .

[0066] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0068] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.

[0069] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0070] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0071] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0072] A network communication module 552, used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0073] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0074] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0075] In some embodiments, the model training device of the electronic device provided in the embodiments of the present application can be implemented in a software manner. Figure 2A The model training device 555 of the electronic device stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 5551, a construction module 5552 and a training module 5553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0076] Next, an electronic device for implementing depth estimation provided by an embodiment of the present application is described. Figure 2B , Figure 2B is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application, Figure 2B The electronic device 600 shown includes: at least one processor 610, a memory 650, at least one network interface 620, and a user interface 630. The various components in the electronic device 600 are coupled together via a bus system 640. The user interface 630 includes: an output device 631, an input device 632. The memory 650 includes: an operating system 651, a network communication module 652, a presentation module 653, and an input processing module 654. It should be noted that Figure 2B The function of the structure in Figure 2A The functions of the structures in are similar.

[0077] In some embodiments, the depth estimation device provided in the embodiments of the present application can be implemented in software. Figure 2BA depth estimation device 655 stored in the memory 650 is shown, which may be software in the form of a program and a plug-in, etc., and includes the following software modules: a second acquisition module 6551 and an estimation module 6552. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0078] In other embodiments, the model training device provided in the embodiments of the present application can be implemented in hardware. As an example, the model training device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0079] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, a vehicle terminal, and an intelligent robot, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a server.

[0080] The model training method provided by the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided in the embodiment of the present application. As mentioned above, the electronic device that implements the model training method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated below.

[0081] See also Figure 3 , Figure 3 This is a first flow chart of the model training method provided in the embodiment of the present application, which will be combined with Figure 3 Steps 101 to 105 are shown for explanation.

[0082] In step 101, a plurality of image samples and a depth image of each image sample are acquired.

[0083] In some embodiments, the depth estimation model can be a monocular depth estimation model or a binocular depth estimation model. If the depth estimation model is a monocular depth estimation model, the image sample can be an image sample taken by a monocular device; if the depth estimation model is a binocular depth estimation model, the image sample can be composed of two images taken by the same monocular device at different perspectives, or can be composed of two images taken by a binocular device. There are many ways to obtain the depth image of the image sample, including: obtaining depth information through sensors such as laser radar or structured light, and then converting it into a depth image; calculating the depth information using the parallax information of a binocular or multi-camera, and then converting it into a depth image; using a depth camera to shoot to obtain a depth image.

[0084] In step 102, for each depth image, a Gaussian distribution corresponding to each pixel is constructed according to the depth value of each pixel in the depth image.

[0085] In some embodiments, see Figure 4 , Figure 4 is a second flow chart of the model training method provided in an embodiment of the present application, Figure 3 The step 102 of "constructing a Gaussian distribution corresponding to each pixel according to the depth value of each pixel in the depth image" can be implemented by following the steps 1021 to 1022, which are described in detail below.

[0086] For each depth image, the following steps 1021 to 1022 may be performed:

[0087] In step 1021, for each pixel, the depth value of the pixel is determined as an expected value, and a preset standard deviation is obtained.

[0088] In some embodiments, for each pixel, the actual depth value of the pixel can be obtained from the depth image, the depth value of the pixel can be used as the expected value of the Gaussian distribution to be constructed, and a preset standard deviation can be set based on experience. The preset standard deviation value range can be set to 1.0~1.4. The actual value of the preset standard deviation can be selected based on the specific actual situation, and the embodiments of the present application are not limited to this.

[0089] In step 1022, a Gaussian distribution corresponding to the pixel point is constructed based on the expected value and the preset standard deviation.

[0090] In some embodiments, the depth value d of the pixel to be estimated is used as an independent variable, and the actual depth value of the pixel is used as an expected value. The Gaussian distribution corresponding to the pixel is constructed according to the independent variable, the expected value and the preset standard deviation. The expression is as follows:

[0091]

[0092] Among them, G is the Gaussian distribution corresponding to each pixel, σ is the standard deviation, μ is the expected value, that is, the true depth value of the pixel (i.e., Ground Truth), d is the independent variable, that is, the depth value of the pixel predicted by the depth estimation model, and the value range of d can be set to [0, 192).

[0093] In addition, according to the characteristics of Gaussian distribution, the following formula can be obtained:

[0094]

[0095] Among them, σ is the standard deviation, μ is the expected value, and d is the independent variable. According to formula (4), the sum of the probability density of the Gaussian distribution is 1.

[0096] Through steps 1021 to 1022, a Gaussian distribution can be constructed according to the true value of the pixel point, and used as a supervisory signal for the training of the depth estimation model, laying the foundation for the subsequent improvement of the loss function of the depth estimation model. By constructing an unbiased transformation process of the loss function through the Gaussian distribution, the loss function of the model can be improved, thereby improving the training efficiency of the depth estimation model.

[0097] Continue to see Figure 3 , and the description continues with step 102 above.

[0098] In step 103, the depth of the image sample is estimated using a depth estimation model to obtain a depth probability distribution of each pixel.

[0099] In some embodiments, see Figure 5 , Figure 5 is a third flow chart of the model training method provided in the embodiment of the present application, Figure 3 The illustrated step 103 may be implemented by following steps 1031A to 1032A, which are described in detail below.

[0100] In step 1031A, the depth estimation model is used to extract features of the image samples to obtain image features corresponding to the image samples.

[0101] In some embodiments, a convolutional neural network (CNN) in a depth estimation model may be used to extract features from image samples, thereby obtaining image features corresponding to the image samples.

[0102] As an example, assume that the CNN in the depth estimation model is a convolutional neural network model, such as a residual network (ResNet) to extract features of image samples. The specific steps are as follows:

[0103] First, load the ResNet model. It should be noted that the ResNet model has been trained on a large-scale image dataset.

[0104] Secondly, the image samples can be preprocessed, such as adjusting the image sample size to the size required by the ResNet model and converting it into a format acceptable to the ResNet model (usually a 3-channel Red, Green, Blue (RGB) image).

[0105] Then, the preprocessed image samples are passed as input to the ResNet model, which automatically learns and extracts the features of the image through multiple layers of convolution and pooling operations. These image features can be found in certain specific layers, such as the output of the pooling layer or the input of the fully connected layer. Image features can be used to handle further image processing tasks.

[0106] In step 1032A, the image features are predicted to obtain a depth probability distribution of each pixel.

[0107] In some embodiments, after obtaining the image features, the image features are fed into one or more fully connected layers, which convert the image features into a one-dimensional vector. After the fully connected layer, a SOFTMAX layer is added, which converts the output of the depth value of each pixel into a probability density through a SOFTMAX function, ensuring that the sum of all probability density values ​​is 1. The expression is as follows:

[0108]

[0109] Among them, p(d i ) is the predicted depth value d i The probability of z i Yes i The unnormalized log probability, z j Yes j The unnormalized log probability of .

[0110] Through step 1031A to step 1032A, the depth estimation model can be used to extract features of image samples and obtain the depth probability distribution of each pixel. The depth probability distribution can be used to improve the subsequent loss function. By improving the loss function, the training efficiency of the model is improved.

[0111] In some embodiments, when the depth estimation model is a binocular depth estimation model, the image samples include a first image sample and a second image sample from different perspectives, see Figure 6 , Figure 6 is a fourth flow chart of the model training method provided in an embodiment of the present application, Figure 3 The illustrated step 103 may also be implemented by the following steps 1031B to 1033B, which are described in detail below.

[0112] In step 1031B, the binocular depth estimation model is used to perform feature extraction on the first image sample and the second image sample respectively to obtain first image features and second image features corresponding to the first image sample and the second image sample respectively.

[0113] In some embodiments, the binocular depth estimation model performs feature extraction on the first image sample and the second image sample under different viewing angles respectively. The feature extraction on the first image sample and the second image sample can be performed with reference to the implementation method of step 1031A, which will not be repeated here. Afterwards, the first image features and the second image features corresponding to the first image sample and the second image sample respectively can be obtained.

[0114] In step 1032B, the first image feature and the second image feature are fused to obtain a third image feature.

[0115] In some embodiments, the first image feature and the second image feature can be spliced, and the spliced ​​image feature can be input into the feature fusion network layer to obtain the third image feature; or a stereo matching algorithm can be used to find matching key points in the first image sample and the second image sample (i.e., specific areas in the image that are unique and stable, and have significant features, such as corners, edges, or obvious textures). The features corresponding to these key points are the associated features, or for each pixel point, check whether its feature vectors in the first image sample and the second image sample are spatially consistent. If the two feature vectors are very close in spatial position, they may represent the same physical features, and the features corresponding to these pixels are the associated features. Then, the associated features in the first image feature and the second image feature are spliced, and the spliced ​​image features are input into the feature fusion network layer to obtain the third image feature.

[0116] In step 1033B, prediction processing is performed on the third image feature to obtain a depth probability distribution of each pixel.

[0117] In some embodiments, the depth probability distribution of each pixel point can be obtained by referring to the implementation method of step 1032A, which will not be repeated here.

[0118] Through steps 1031B to 1033B, the binocular depth estimation model can be used to perform feature extraction on the first image sample and the second image sample. By extracting image features of image samples from two different perspectives, more comprehensive information can be obtained, so that the depth probability distribution of each pixel point is more accurate. The depth probability distribution can be used to improve the subsequent loss function, which can improve the performance of binocular depth estimation and make the depth estimation result more accurate and reliable.

[0119] Continue to see Figure 3 , and the description continues with step 103 above.

[0120] In step 104 , a loss value is determined based on the depth probability distribution and the Gaussian distribution.

[0121] In some embodiments, see Figure 7 , Figure 7 is a fifth flow chart of the model training method provided in the embodiment of the present application, Figure 3 The illustrated step 104 may be implemented by following steps 1041 to 1043, which are described in detail below.

[0122] In step 1041, a preset loss function is obtained, where the loss function is used to determine the distance between two probability distributions.

[0123] In some embodiments, the loss function is used to determine the distance between two probability distributions. There are many ways to determine the distance between two probability distributions, including: cross entropy is a metric used to measure the difference between two probability distributions. Cross entropy measures the degree of match between one probability distribution (usually the predicted distribution of the model) and another probability distribution (usually the true distribution of the actual data). If the two distributions are exactly the same, then the cross entropy is zero, indicating that there is no difference. When the difference between the two distributions is greater, the value of the cross entropy is also greater; JS divergence is a metric used to measure the difference between two probability distributions. It measures the degree of difference between two probability distributions relative to their respective average distributions. JS divergence determines dissimilarity by calculating the average of the KL divergences of the two distributions relative to their respective average distributions. If the two distributions are exactly the same, then the JS divergence is zero, indicating that there is no difference. When the difference between the two distributions is greater, the value of the JS divergence will also increase, reflecting the degree of mismatch. The value of JS divergence is between 0 and 1, which not only considers the overlapping area of ​​the two distributions, but also the unique part, so it can more comprehensively describe the similarity of the two distributions; KL divergence can be called relative entropy, which can measure the degree of difference between two probability distributions. When the KL divergence is larger, the greater the difference between the two; and when the KL divergence is smaller, the difference between the two is smaller. Therefore, the embodiment of the present application can use the KL divergence function as a loss function. It should be noted that the embodiment of the present application does not limit the loss function and can be selected according to actual conditions.

[0124] In step 1042 , based on the loss function, the distance between the depth probability distribution and the Gaussian distribution is determined.

[0125] In some embodiments, the KL divergence between the depth probability distribution and the Gaussian distribution may be calculated as the distance between the depth probability distribution and the Gaussian distribution. The specific calculation formula is as follows:

[0126]

[0127] Among them, Loss is the loss function, D KL (P||Q) represents the KL divergence of P with respect to Q, P is a Gaussian distribution, Q is a deep probability distribution, p(x) is the probability density function of P, and q(x) is the probability density function of Q.

[0128] In step 1043, the distance is determined as a loss value.

[0129] In some embodiments, the calculated KL divergence result may be used as the loss value of the loss function.

[0130] Through steps 1041 to 1043, the constructed Gaussian distribution can be used as a supervisory signal in the depth estimation model training process, and the loss function of the depth estimation model can be improved. The improved loss function helps to reduce the instability in the depth estimation model training process, so that the depth estimation model converges to the optimal solution more smoothly during the training process. The improved loss function also helps the depth estimation model to learn more accurate feature representation, thereby improving the accuracy of the depth estimation model in practical applications.

[0131] Continue to see Figure 3 , and the description continues with step 104 above.

[0132] In step 105, the depth estimation model is updated based on the loss value to obtain a trained depth estimation model.

[0133] In some embodiments, according to the obtained loss value, the gradient of the loss function with respect to the depth estimation model parameters is calculated, and the depth estimation model parameters are updated according to an optimization algorithm (such as Stochastic Gradient Descent (SGD), Adaptive Moment Estimation (Adam), etc.), and the optimization algorithm can determine how to adjust the parameters to minimize the loss function. After the parameters of the depth estimation model are updated, the process of forward propagation, loss calculation, back propagation and parameter update is repeated until a predetermined number of training rounds is reached or the verification loss value no longer decreases, and the depth estimation model after preliminary training is obtained. After that, the performance of the depth estimation model after preliminary training is evaluated on the validation set. First, the depth estimation model after preliminary training is loaded into the evaluation environment, and each image sample in the validation set is forward propagated. The depth value of each pixel is estimated using the model, and then the depth value estimated by the model is compared with the true depth value corresponding to the image sample in the validation set to obtain an evaluation result. When it is determined based on the evaluation result that the depth estimation model after preliminary training passes the evaluation, the depth estimation model after preliminary training is determined as the trained depth estimation model. When it is determined based on the evaluation result that the initially trained depth estimation model has not passed the evaluation, image samples are obtained again, and the initially trained depth estimation model is continuously trained until a trained depth estimation model that passes the evaluation is obtained and saved.

[0134] In this process, the depth estimation model will gradually learn how to more accurately predict the depth value of each pixel in the image.

[0135] Through the embodiments of the present application, first, multiple image samples and depth images of each image sample are obtained. Then, for each depth image, a Gaussian distribution corresponding to each pixel can be constructed according to the depth value of each pixel in the depth image, which can be used as a supervisory signal in the model training process. The depth estimation model is then used to perform depth estimation on the image samples to obtain the depth probability distribution of each pixel. The loss value is then determined based on the depth probability distribution and the Gaussian distribution. Finally, the depth estimation model is updated based on the loss value to obtain a trained depth estimation model. The Gaussian distribution corresponding to each pixel is used as a supervisory signal, and the loss value is determined together with the depth probability distribution of each pixel. Therefore, in the process of updating the depth estimation model according to the loss value, the depth probability distribution will continue to approach the Gaussian distribution. Compared with constructing a loss function based on the estimated depth image and the actual depth image, this is an unbiased transformation, which can make the loss value and gradient smaller, reduce the probability of gradient explosion, and improve the stability of the model training process. Therefore, through the unbiased transformation process of constructing the loss function through Gaussian distribution, the loss function of the model can be improved, which can increase the convergence speed of the model and thus improve the efficiency of model training; the depth estimation model is used to extract features from image samples and obtain the depth probability distribution of each pixel. The depth probability distribution can be used to improve the subsequent loss function. By improving the loss function, the model is improved. The training efficiency of the model can be improved; the binocular depth estimation model can be used to extract features from the first image sample and the second image sample. By extracting image features of image samples from two different perspectives, more comprehensive information can be obtained, so that the depth probability distribution of each pixel point is more accurate. The depth probability distribution can be used to improve the subsequent loss function, which can improve the performance of binocular depth estimation and make the depth estimation result more accurate and reliable; the constructed Gaussian distribution can be used as a supervisory signal in the training process of the depth estimation model, and the loss function of the depth estimation model can be improved. The improved loss function helps to reduce the instability in the training process of the depth estimation model, so that the depth estimation model converges to the optimal solution more smoothly during the training process. The improved loss function also helps the depth estimation model to learn more accurate feature representation, thereby improving the accuracy of the depth estimation model in practical applications.

[0136] In the following, the depth estimation method provided by the embodiment of the present application will be described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. Taking the execution subject as a terminal as an example, the depth estimation method provided by the embodiment of the present application will be described.

[0137] See also Figure 8 , Figure 8 is a first flow chart of the depth estimation method provided in the embodiment of the present application, which will be combined with Figure 8Steps 201 to 202 are shown for explanation.

[0138] In step 201, an image to be estimated and a trained depth estimation model are obtained.

[0139] The depth estimation model is obtained by training according to the model training method provided in the embodiment of the present application.

[0140] In some embodiments, if the trained depth estimation model is a monocular depth estimation model, the image to be estimated can be a directly input image or an image taken by a monocular device; if the trained depth estimation model is a binocular depth estimation model, the image to be estimated can be two directly input images taken from different perspectives by the same monocular device or two images taken by a binocular device.

[0141] In step 202, the trained depth estimation model is used to perform depth estimation on the image to be estimated to obtain a depth estimation result.

[0142] In some embodiments, when the depth estimation model is a binocular depth estimation model, the image to be estimated includes a first image and a second image from different perspectives. Fig. 9 , Fig. 9 It is a second flow chart of the depth estimation method provided in an embodiment of the present application. Figure 8 The illustrated step 202 may be implemented by following steps 2021 to 2023, which are described in detail below.

[0143] In step 2021, a binocular depth estimation model is used to perform depth estimation on the first image and the second image to obtain a depth probability distribution of each pixel.

[0144] In some embodiments, the depth probability distribution of each pixel point may be obtained by referring to the implementation method of steps 1031B to 1033B, which will not be described in detail here.

[0145] In step 2022, according to the depth probability distribution of each pixel, a depth value sequence of each pixel and a probability value corresponding to each depth value in the depth value sequence are determined.

[0146] In some embodiments, referring to the above formula (5), the depth value sequence of each pixel can be determined according to the value of d in formula (5). For example, when the value range of d is set to [0, 192), the depth value sequence of each pixel is arranged in order from 0 to 192 (excluding 192). After that, each value of d is substituted into formula (5) to obtain the corresponding probability value p(d).

[0147] In step 2023, each depth value and probability value is summed to obtain a depth estimation result for each pixel.

[0148] In some embodiments, the probability value may be used as a weight coefficient to perform weighted summation processing on the depth values ​​to obtain the sum of the depth values, which is used as the depth estimation result for each pixel.

[0149] Through steps 2021 to 2023, by summing the depth value and its probability value through the depth probability distribution of each pixel point, a comprehensive depth estimation result can be obtained, so that the depth estimation result is closer to the actual depth value, thereby improving the accuracy of the model depth estimation.

[0150] Through step 201 to step 202, the trained depth estimation model can perform depth estimation, indicating that when the depth estimation model is not trained using a loss function constructed by comparing the predicted depth image with the actual depth image, the depth estimation model can still perform reasoning normally.

[0151] In some embodiments, see Fig.10 , Fig.10 is a third flow chart of the depth estimation method provided in the embodiment of the present application, Figure 8 After step 202, the electronic device may be controlled to move, which will be described in detail below in conjunction with steps 301 to 303.

[0152] In step 301, relative position information between the electronic device and a target object in an image to be estimated is determined based on a depth estimation result.

[0153] In some embodiments, after obtaining the depth estimation result of each pixel point, the coordinates of the pixel points occupied by the target object in the image to be estimated can be determined, and the pixel points can be matched one by one according to the coordinates. Then, the depth estimation results corresponding to the pixel points occupied by the target object can be determined, and the average of these depth estimation results can be calculated as the relative position information between the electronic device and the target object in the image to be estimated; these depth estimation results can also be compared, and the smallest depth estimation result can be used as the relative position information between the electronic device and the target object in the image to be estimated; these depth estimation results can also be compared, and the smallest depth estimation result can be used as the relative position information between the electronic device and the target object in the image to be estimated.

[0154] For example, the electronic device can be an autonomous walking robot. The robot can capture images of the surrounding environment through a configured camera. For example, if there is a cat in front, the camera can capture the cat in the image. The trained depth estimation model is then used to perform depth estimation on the image to obtain a depth estimation result. The depth estimation result corresponding to each pixel point corresponding to the cat in the image can be determined based on the depth estimation result. The depth estimation results can be compared and the smallest depth estimation result can be used as the relative position information between the robot and the cat. For example, if the smallest depth estimation result is 5 meters, the robot and the cat are 5 meters apart.

[0155] In step 302, the moving speed and moving direction of the electronic device are determined based on the relative position information.

[0156] In some embodiments, the next moving speed and moving direction of the electronic device can be determined based on the obtained relative position information and the current moving speed and moving direction of the electronic device. For example, if the electronic device is in a stationary state and is ready to move forward, assuming that the relative position information between the electronic device and the target object in the image to be estimated is less than the collision distance threshold, the moving direction of the electronic device can be determined to be a direction away from the target object to avoid colliding with the target object. Assuming that the relative position information between the electronic device and the target object in the image to be estimated is greater than the collision distance threshold, the moving direction of the electronic device can be determined to be moving forward; if the electronic device is in a moving state and moving forward, assuming that the relative position information between the electronic device and the target object in the image to be estimated is less than the collision distance threshold, the moving direction of the electronic device can be determined to be a direction away from the target object, and the moving speed can be reduced, or the moving direction is not changed and the moving speed is reduced to 0 to avoid colliding with the target object. Assuming that the relative position information between the electronic device and the target object in the image to be estimated is greater than the collision distance threshold, the moving direction of the electronic device can be determined to be moving forward; it should be noted that the collision distance threshold can be determined according to the moving speed of the electronic device or can be pre-set, and the embodiments of the present application are not limited to this.

[0157] As an example, the electronic device may be an autonomous walking robot. For example, it is determined through depth estimation results that there is a cat 0.1 meters in front of the robot. At this time, the robot is moving forward at a speed of 2m / s. The collision distance threshold of the robot can be pre-set to 2m. If the relative position information between the robot and the cat is less than the collision distance threshold, it can be determined that the robot should immediately reduce the moving speed and stop moving.

[0158] In step 303, the electronic device is controlled to move based on the moving speed and the moving direction.

[0159] In some embodiments, after the moving speed and moving direction of the electronic device are determined, the movement of the electronic device is controlled according to the moving speed and moving direction.

[0160] As an example, for example, the electronic device may be an autonomous walking robot, and if it has been determined that the robot should immediately reduce its moving speed and stop moving, the robot is controlled to reduce its moving speed and reduce the speed to 0, and if there is no information about the moving direction, the robot continues to maintain the current moving direction.

[0161] Through step 301 to step 303, the electronic device can be controlled by using the depth estimation model to achieve autonomous movement of the electronic device.

[0162] Through the embodiments of the present application, the trained depth estimation model can perform depth estimation, indicating that when the depth estimation model is not trained by using a loss function constructed by comparing the predicted depth image and the actual depth image, the depth estimation model can still perform reasoning normally; the depth estimation model can be used to control electronic devices to achieve autonomous movement of the electronic devices.

[0163] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0164] Depth estimation model is an important technology in the field of computer vision. It can recover the depth information of the scene from images or videos, which is crucial for many practical applications. The following are some practical application scenarios of depth estimation model: 1. Autonomous driving: In autonomous driving systems, depth estimation models can help vehicles understand the three-dimensional structure of the surrounding environment. For example, by estimating the distance of obstacles, pedestrians or other vehicles on the road, the system can make safer driving decisions. For example, an autonomous vehicle uses a depth estimation model to determine the distance of pedestrians in front of it so that it can accurately avoid them. 2. Robot navigation: When robots perform navigation tasks, depth estimation models can help robots locate and avoid obstacles in complex environments. For example, a warehouse robot uses a depth estimation model to identify the distance between shelves to ensure safe movement between shelves. 3. Drone perception: When drones perform flight missions, depth estimation models can help them avoid collisions and hover accurately at a specific height. For example, when drones perform mapping or search and rescue missions, they use depth estimation models to identify terrain and obstacles to ensure safe flight. 4. Intelligent monitoring: In monitoring systems, depth estimation models can help analyze dynamic changes in scenes, such as the movement and size changes of objects. For example, an intelligent monitoring system uses a depth estimation model to track individuals or vehicles within the monitoring area, thereby improving the efficiency and accuracy of monitoring.

[0165] Below, a specific example is used to illustrate that the embodiments of the present application can be applied to a humanoid robot. The humanoid robot sets up a binocular camera by simulating the human eye to acquire images of the surrounding environment. After that, the trained depth estimation model can be used to perform depth estimation on the acquired image to obtain a depth estimation result. The humanoid robot can then decide the direction and speed of movement based on the depth estimation result.

[0166] The following specifically describes that the depth estimation model in the humanoid robot can be a binocular depth estimation model, and the process of training the binocular depth estimation model is as follows:

[0167] First, multiple groups of image samples and depth images of each group of image samples are obtained, each group of image samples consists of a left-eye image and a right-eye image, and the depth image of each group of image samples can be obtained by a depth camera.

[0168] For each pixel in each depth image, the depth value of the pixel is taken as the expected value, and the standard deviation is set to a range of 1.0 to 1.4 (determined according to empirical values) to construct a Gaussian distribution. The expression of the sampling signal corresponding to the Gaussian distribution of each pixel can be found in formula (3).

[0169] After that, the depth estimation model is used to estimate the depth of the image samples. The final output layer of the stereo depth estimation network is:

[0170] Step 1: Get the convolution result

[0171] a=CNN(*) (7)

[0172] Among them, a is the convolution result, and * represents the image features obtained by the depth estimation model by extracting features from the image samples.

[0173] Step 2: Get the depth probability distribution

[0174] z=softmax(a) (8)

[0175] Among them, z is the depth probability distribution corresponding to the pixel point, which can be represented by a vector with a dimension of [B, C, D, H, W]. D is a manually set hyperparameter, usually 192 (referring to the pixel depth value of 0-192), and a is the convolution result.

[0176] Step 3: Get the depth estimation result

[0177] D = sum(z*d, dim = 2) (9)

[0178] Wherein, D is the depth estimation result of each pixel, and dim=2 represents the index on the axis of the second dimension D in the d vector z (the range of d is 0-192).

[0179] Finally, the second and third steps can be equivalent to the soft argmax function, expressed as follows:

[0180]

[0181] Where D is the depth estimation result of each pixel, d i Indicates the depth value of each pixel. Indicates the probability of each depth value.

[0182] The above formula (10) can be regarded as finding the expectation of a distribution: that is, expectation is equal to probability multiplied by exponent (Expectation = possibility * index).

[0183] Finally, the KL divergence is used to calculate the loss expression of G and softmax results as follows:

[0184] Loss=KLdiv(G,softmax) (11)

[0185] Among them, Loss is the loss function, G is the Gaussian distribution, and softmax is the deep probability distribution

[0186] The model trained by the loss function in the above formula (11) can continue to use formula (10) for calculation during inference (that is, the above loss function does not affect the inference, and formula (10) can still be used to calculate the depth). Because the process designed here to construct the loss function by sampling Gaussian signals is an unbiased transformation, and formula (10) can be directly equivalent to finding the expected value in the Gaussian distribution, formula (10) can continue to be used for calculation during inference.

[0187] Through the embodiments of the present application, it is possible to construct an unbiased transformation process of the loss function through Gaussian distribution, reduce the probability of gradient explosion during model training, improve the stability and convergence speed of the model training process, thereby improving the model training effect and training efficiency, making the depth estimation result more accurate, and then providing more accurate depth information in actual application scenarios, helping electronic devices equipped with depth estimation models to make more reasonable decisions, for example, the effect is better when applied to large multimodal models on humanoid robots.

[0188] The following is a description of an exemplary structure of a model training device 555 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the model training device 555 of the memory 550 may include:

[0189] The first acquisition module 5551 is used to obtain multiple image samples and depth images of each image sample; the construction module 5552 is used to construct a Gaussian distribution corresponding to each pixel in each depth image according to the depth value of each pixel in the depth image; the training module 5553 is used to use the depth estimation model to perform depth estimation on the image samples to obtain the depth probability distribution of each pixel; based on the depth probability distribution and the Gaussian distribution, determine the loss value; based on the loss value, update the depth estimation model to obtain the trained depth estimation model.

[0190] In some embodiments, the construction module 5552 is further used to determine the depth value of each pixel as an expected value and obtain a preset standard deviation; based on the expected value and the preset standard deviation, construct a Gaussian distribution corresponding to the pixel.

[0191] In some embodiments, the training module 5553 is also used to obtain a preset loss function, which is used to determine the distance between two probability distributions; based on the loss function, determine the distance between the depth probability distribution and the Gaussian distribution; and determine the distance as the loss value.

[0192] In some embodiments, the training module 5553 is also used to use the depth estimation model to extract features of image samples to obtain image features corresponding to the image samples; and perform prediction processing on the image features to obtain the depth probability distribution of each pixel.

[0193] In some embodiments, the depth estimation model includes a binocular depth estimation model, the image samples include a first image sample and a second image sample from different perspectives, and the training module 5553 is also used to use the binocular depth estimation model to perform feature extraction on the first image sample and the second image sample respectively, to obtain first image features and second image features corresponding to the first image sample and the second image sample respectively; to fuse the first image features and the second image features to obtain a third image feature; and to perform prediction processing on the third image feature to obtain a depth probability distribution of each pixel point.

[0194] The following is a description of an exemplary structure of the depth estimation device 655 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2B As shown, the software modules stored in the depth estimation device 655 of the memory 650 may include:

[0195] The second acquisition module 6551 is used to obtain the image to be estimated and the trained depth estimation model, where the depth estimation model is trained according to the model training method provided in the embodiment of the present application; the estimation module 6552 is used to use the trained depth estimation model to perform depth estimation on the image to be estimated to obtain a depth estimation result.

[0196] In some embodiments, when the depth estimation model is a binocular depth estimation model, the images to be estimated include a first image and a second image from different perspectives, and the estimation module 6552 is further used to perform depth estimation on the first image and the second image using the binocular depth estimation model to obtain a depth probability distribution for each pixel; according to the depth probability distribution of each pixel, determine a depth value sequence for each pixel and a probability value corresponding to each depth value in the depth value sequence; and sum each depth value and probability value to obtain a depth estimation result for each pixel.

[0197] In some embodiments, the estimation module 6552 is also used to determine the relative position information between the electronic device and the target object in the image to be estimated based on the depth estimation result; determine the moving speed and moving direction of the electronic device based on the relative position information; and control the movement of the electronic device based on the moving speed and moving direction.

[0198] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs the model training method described above in the embodiment of the present application.

[0199] The present application embodiment provides a computer-readable storage medium, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the model training method or depth estimation method provided in the present application embodiment, for example, Figures 3 to 7 The model training method shown, or Figures 8 to 10 The depth estimation method shown.

[0200] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0201] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0202] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0203] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0204] To summarize, through the embodiments of the present application, first, multiple image samples and depth images of each image sample are obtained, and then for each depth image, a Gaussian distribution corresponding to each pixel can be constructed according to the depth value of each pixel in the depth image, which can be used as a supervisory signal in the model training process. The depth estimation model is then used to perform depth estimation on the image samples to obtain the depth probability distribution of each pixel, and then the loss value is determined based on the depth probability distribution and the Gaussian distribution. Finally, based on the loss value, the depth estimation model is updated to obtain the trained depth estimation model.The Gaussian distribution corresponding to each pixel is used as a supervision signal, and the loss value is determined together with the depth probability distribution of each pixel. Therefore, in the process of updating the depth estimation model according to the loss value, the depth probability distribution will continue to approach the Gaussian distribution. Compared with constructing a loss function based on the estimated depth image and the actual depth image, this is an unbiased transformation, which can make the loss value and gradient smaller, reduce the probability of gradient explosion, and improve the stability of the model training process. Therefore, the unbiased transformation process of constructing the loss function through the Gaussian distribution can improve the loss function of the model, increase the convergence speed of the model, and thus improve the efficiency of model training; using the depth estimation model to The image samples are feature extracted and the depth probability distribution of each pixel is obtained. The depth probability distribution can be used to improve the subsequent loss function. By improving the loss function, the training efficiency of the model is improved; the binocular depth estimation model can be used to extract features from the first image sample and the second image sample. By extracting the image features of the image samples from two different perspectives, more comprehensive information can be obtained, so that the depth probability distribution of each pixel is more accurate. The depth probability distribution can be used to improve the subsequent loss function, which can improve the performance of the binocular depth estimation and make the depth estimation result more accurate and reliable; the constructed Gaussian distribution can be used as a supervisory signal in the depth estimation model training process, and The loss function of the depth estimation model is improved. The improved loss function helps to reduce the instability in the depth estimation model training process, so that the depth estimation model converges to the optimal solution more smoothly during the training process. The improved loss function also helps the depth estimation model to learn more accurate feature representations, thereby improving the accuracy of the depth estimation model in practical applications. By summing the depth value and its probability value through the depth probability distribution of each pixel point, a comprehensive depth estimation result can be obtained, which makes the depth estimation result closer to the true depth value and improves the accuracy of the model's depth estimation. The trained depth estimation model can perform depth estimation, indicating that when the predicted depth image and the actual depth are not used, the depth estimation result can be closer to the true depth value, thereby improving the accuracy of the model's depth estimation. When the loss function constructed by image comparison is used to train the depth estimation model, the depth estimation model can still perform reasoning normally; the depth estimation model can be used to control electronic devices to achieve autonomous movement of electronic devices; through the implementation of the unbiased transformation process that can construct the loss function through Gaussian distribution in this application, the probability of gradient explosion during model training is reduced, the stability and convergence speed of the model training process are improved, thereby improving the model training effect and training efficiency, making the depth estimation results more accurate, and then providing more accurate depth information in actual application scenarios, helping electronic devices equipped with depth estimation models to make more reasonable decisions, for example, the effect is better when applied to large multimodal models on humanoid robots.

[0205] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a plurality of image samples and a depth image of each of the image samples; For each depth image, constructing a Gaussian distribution corresponding to each pixel in the depth image according to the depth value of each pixel; Using a depth estimation model to perform depth estimation on the image sample to obtain a depth probability distribution of each pixel; Determining a loss value based on the depth probability distribution and the Gaussian distribution; Based on the loss value, the depth estimation model is updated to obtain a trained depth estimation model.

2. The method according to claim 1, characterized in that The step of constructing a Gaussian distribution corresponding to each pixel in the depth image according to the depth value of each pixel includes: For each pixel, determining the depth value of the pixel as an expected value, and obtaining a preset standard deviation; Based on the expected value and the preset standard deviation, a Gaussian distribution corresponding to the pixel point is constructed.

3. The method according to claim 1, characterized in that The determining of the loss value based on the depth probability distribution and the Gaussian distribution includes: Obtaining a preset loss function, wherein the loss function is used to determine the distance between two probability distributions; Based on the loss function, determining a distance between the depth probability distribution and the Gaussian distribution; The distance is determined as the loss value.

4. The method according to claim 1, characterized in that The using the depth estimation model to perform depth estimation on the image sample to obtain a depth probability distribution of each pixel point includes: Extracting features from the image samples using a depth estimation model to obtain image features corresponding to the image samples; The image features are predicted to obtain a depth probability distribution of each pixel.

5. The method according to any one of claims 1 to 4, characterized in that: The depth estimation model includes a binocular depth estimation model, the image samples include a first image sample and a second image sample from different perspectives, and the depth estimation model is used to perform depth estimation on the image samples to obtain a depth probability distribution of each pixel point, including: Using a binocular depth estimation model, feature extraction is performed on the first image sample and the second image sample to obtain first image features and second image features corresponding to the first image sample and the second image sample, respectively; fusing the first image feature and the second image feature to obtain a third image feature; Prediction processing is performed on the third image feature to obtain a depth probability distribution of each pixel.

6. A depth estimation method, characterized in that: The method comprises: Obtaining an image to be estimated and a trained depth estimation model, wherein the depth estimation model is trained by the method according to any one of claims 1 to 5; The trained depth estimation model is used to perform depth estimation on the image to be estimated to obtain a depth estimation result.

7. The method according to claim 6, characterized in that When the depth estimation model is a binocular depth estimation model, the image to be estimated includes a first image and a second image from different perspectives, and the depth estimation of the image to be estimated using the trained depth estimation model to obtain a depth estimation result includes: Performing depth estimation on the first image and the second image using the binocular depth estimation model to obtain a depth probability distribution of each pixel; Determine, according to the depth probability distribution of each pixel, a depth value sequence of each pixel and a probability value corresponding to each depth value in the depth value sequence; Each depth value and the probability value are summed to obtain a depth estimation result for each pixel.

8. The method according to claim 6, characterized in that After performing depth estimation on the image to be estimated by using the trained depth estimation model to obtain a depth estimation result, the method further includes: Determine relative position information between the electronic device and the target object in the image to be estimated based on the depth estimation result; Determining the moving speed and moving direction of the electronic device based on the relative position information; The electronic device is controlled to move based on the moving speed and the moving direction.

9. A model training device, characterized in that: The device comprises: A first acquisition module, used to acquire a plurality of image samples and a depth image of each of the image samples; A construction module, configured to construct, for each depth image, a Gaussian distribution corresponding to each pixel point according to a depth value of each pixel point in the depth image; A training module is used to use a depth estimation model to perform depth estimation on the image sample to obtain a depth probability distribution for each pixel; based on the depth probability distribution and the Gaussian distribution, a loss value is determined; based on the loss value, the depth estimation model is updated to obtain a trained depth estimation model.

10. A depth estimation device, characterized in that: The device comprises: A second acquisition module, used to acquire an image to be estimated and a trained depth estimation model, wherein the depth estimation model is trained by the method according to any one of claims 1 to 5; The estimation module is used to perform depth estimation on the image to be estimated by using the trained depth estimation model to obtain a depth estimation result.

11. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the model training method described in any one of claims 1 to 5, or the depth estimation method described in any one of claims 6 to 8 when executing computer executable instructions or computer programs stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the model training method described in any one of claims 1 to 5 or the depth estimation method described in any one of claims 6 to 8 is implemented.

Images