Depth estimation model training method, depth estimation method and electronic device

Abstract

The present application discloses a depth estimation model training method, a depth estimation method and an electronic device, and relates to the field of machine vision technology. The depth estimation model training method of one embodiment of the present application includes: obtaining a first image pair from a training data set, the first image pair including a first left image and a first right image. Input the first left image into the depth estimation model to be trained to obtain a disparity map. Add the first left image to the disparity map to obtain a second right image. Calculate the mean square error and cosine similarity of the pixel values ​​at all corresponding pixel points in the first right image and the second right image. Calculate the average value of the mean square error and the cosine similarity of the pixel values ​​at all corresponding pixel points, respectively, to obtain a first average value of the mean square error and a second average value of the cosine similarity. Add the first average value and the second average value to obtain a loss value of the depth estimation model. Iteratively train the depth estimation model according to the loss value.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a depth estimation model training method, a depth estimation method, and an electronic device. Background Art

[0002] Image depth estimation is a fundamental problem in machine vision, with applications in autonomous driving, scene understanding, robotics, 3D reconstruction, photography and videography, intelligent medicine, intelligent human-computer interaction, spatial mapping, augmented reality, and other fields. For example, in autonomous driving, image depth information can be used to identify obstacles in front of the vehicle, such as pedestrians or other vehicles.

[0003] Depth estimation requires reconstructing an image to obtain depth information. However, using current depth estimation methods, the color differences between individual pixels in low-texture areas of an image are small, which can lead to pixel misjudgment in these low-texture areas, thus affecting the accuracy of depth information. Summary of the Invention

[0004] In view of this, the present application provides a depth estimation model training method, a depth estimation method and an electronic device, which can improve the sensitivity of the depth estimation model to color, thereby improving the accuracy of depth information.

[0005] In a first aspect, the present application provides a method for training a depth estimation model, comprising: obtaining a first image pair from a training data set, the first image pair comprising a first left image and a first right image. Inputting the first left image into the depth estimation model to be trained to obtain a disparity map. Adding the first left image to the disparity map to obtain a second right image. Calculating the mean square error and cosine similarity of the pixel values ​​at all corresponding pixel points in the first right image and the second right image. Calculating the average values ​​of the mean square error and cosine similarity of the pixel values ​​at all corresponding pixel points, respectively, to obtain a first average value of the mean square error and a second average value of the cosine similarity. Adding the first average value and the second average value to obtain a loss value of the depth estimation model. Iteratively training the depth estimation model based on the loss value.

[0006] By adopting the depth estimation model training method of this embodiment, the loss value of the depth estimation model is combined with the mean square error and cosine similarity, which can not only improve the prediction accuracy of the depth estimation model, but also improve the depth estimation model's sensitivity to color, and can distinguish the color differences between each pixel even in low-texture areas.

[0007] A second aspect of the present application provides a depth estimation method, comprising: acquiring a first image, inputting the first image into a pre-trained depth estimation model, and acquiring a first depth image.

[0008] Among them, the depth estimation model is a model trained using the depth estimation model training method provided in the first aspect of this application.

[0009] By adopting the depth estimation method of this embodiment and acquiring the first depth image through the depth estimation model, the accuracy of the depth information in the first depth image can be improved.

[0010] The third aspect of the present application provides an electronic device, including a processor and a memory, wherein the processor can run a computer program or code stored in the memory to implement the depth estimation model training method provided in the first aspect of the present application, or implement the depth estimation method provided in the second aspect of the present application.

[0011] It can be understood that the specific implementation methods and beneficial effects of the third aspect of the present application are the same as the specific implementation methods and beneficial effects of the first and second aspects of the present application, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Schematic diagram of an application scenario of the depth estimation method provided in an embodiment of the present application.

[0013] Figure 2 This is a flowchart of a depth estimation method provided in an embodiment of the present application.

[0014] Figure 3 This is a flowchart of the depth estimation model training method provided in an embodiment of the present application.

[0015] Figure 4 It is a schematic structural diagram of an electronic device according to one embodiment of the present application.

[0016] Description of main component symbols

[0017] Vehicles 100, 120, 130

[0018] Windshield 10

[0019] Depth Estimation System 20

[0020] Camera equipment 201

[0021] Distance acquisition device 202

[0022] Processor 203, 41

[0023] Horizontal coverage area 110, 140

[0024] Electronic equipment 40

[0025] Memory 42 DETAILED DESCRIPTION

[0026] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0027] It should also be noted that the method disclosed in the embodiments of the present application or the method shown in the flowchart includes one or more steps for implementing the method. Without departing from the scope of the claims, the execution order of multiple steps can be interchanged with each other, and some steps can also be deleted.

[0028] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0029] 1. Depth Estimation

[0030] Depth estimation is used to obtain the distance information from each pixel in the image to the camera. The image containing distance information is called a depth image.

[0031] 2. Parallax

[0032] The pixel coordinates of the same object in two images are different. Objects closer to the camera have larger differences, while objects farther away have smaller differences. The difference in pixel coordinates between points in the same world coordinate system and in different images is called parallax. The parallax between different images can be used to calculate the distance of the object from the camera, or depth, based on camera parameters.

[0033] A disparity map is a stereo image whose size matches the size of the reference image and whose element values ​​are disparity values. Disparity estimation is the process of finding the disparity between corresponding pixels in the left and right images, also known as stereo matching.

[0034] 3. Autoencoder (AE)

[0035] Autoencoders are a type of artificial neural network (ANN) used in semi-supervised and unsupervised learning. They represent input information by using it as a learning target. An autoencoder consists of two parts: an encoder and a decoder. Based on the learning paradigm, autoencoders can be categorized as contractive, regularized, and variational. Depending on the architecture, autoencoders can be either feedforward or recursive neural networks.

[0036] The following describes the depth estimation method provided in the embodiment of the present application as applied to an autonomous driving scenario. It is understood that the depth estimation method provided in the embodiment of the present application is not limited to application in autonomous driving scenarios.

[0037] See Figure 1 , Figure 1 Schematic diagram of an application scenario of the depth estimation method provided in an embodiment of the present application.

[0038] like Figure 1 As shown, the vehicle 100 includes a depth estimation system 20 disposed in an interior compartment behind the windshield 10 of the vehicle 100. The depth estimation system 20 includes a camera 201, a distance acquisition device 202, and a processor 203. The processor 203 is electrically connected to the camera 201 and the distance acquisition device 202.

[0039] It is understood that the camera device 201, the distance acquisition device 202, and the processor 203 can be installed at other locations on the vehicle 100, so that the camera device 201 can acquire images in front of the vehicle 100, and the distance acquisition device 202 can detect the distance of objects in front of the vehicle 100. For example, the camera device 201 and the distance acquisition device 202 can be located in the metal grille or front bumper of the vehicle 100. Figure 1 Although only one distance acquisition device 202 is shown, the vehicle 100 may have multiple distance acquisition devices 202 pointing in different directions (such as the side, front, rear, etc.). Each distance acquisition device 202 can be set in a position such as a windshield, a door panel, a bumper, or a metal grille.

[0040] In this embodiment, the camera device 201 on the vehicle 100 can capture images of the scene in front of and on both sides of the vehicle 100. Figure 1As shown, there are two objects, vehicle 120 and vehicle 130, within the horizontal coverage area 110 (shown by a dotted line) that can be detected by the camera 201. The camera 201 can capture images of the vehicle 120 and the vehicle 130 in front of the vehicle 100.

[0041] In some embodiments, the camera device 201 may be a binocular camera or a monocular camera.

[0042] In some embodiments, the camera device 201 can be implemented as a driving recorder. The driving recorder is used to record information such as images and sounds of the vehicle 100 while it is driving. After the driving recorder is installed on the vehicle 100, the driving recorder can record images and sounds of the entire driving process of the vehicle 100, thereby providing effective evidence for traffic accidents. As an example, in addition to the functions described above, the functions provided by the driving recorder may also include global positioning system (GPS) positioning, driving trajectory capture, remote monitoring, electronic dog, navigation, etc., which are not specifically limited in this embodiment.

[0043] The distance acquisition device 202 can be used to detect objects in front of and on both sides of the vehicle 100 to obtain the distance between the object and the distance acquisition device 202. Figure 1 As shown, the distance acquisition device 202 on the vehicle 100 can acquire the distance between the vehicle 120 and the distance acquisition device 202, and the distance between the vehicle 130 and the distance acquisition device 202. The distance acquisition device 202 can be an infrared sensor, a lidar, or a radar.

[0044] Taking the distance acquisition device 202 as a radar, for example, the radar uses radio frequency (RF) waves to determine the distance, direction, speed, and / or height of objects in front of the vehicle. Specifically, the radar includes a transmitter and a receiver. The transmitter transmits RF waves (radar signals), which are reflected when encountering objects along their path. The RF waves reflected by the object return a small portion of their energy to the receiver. Figure 1 As shown, the radar is configured to transmit radar signals through the windshield in the horizontal coverage area 140 and receive radar signals reflected by any object within the horizontal coverage area 140 , so as to obtain a three-dimensional point cloud image of any object within the horizontal coverage area 140 .

[0045] In this embodiment, the horizontal coverage area 110 and the horizontal coverage area 140 may completely overlap or partially overlap.

[0046] In some embodiments, the camera 201 can capture images of a scene within the horizontal coverage area 110 at a certain periodic rate. Similarly, the radar can capture three-dimensional point cloud images of the scene within the horizontal coverage area 140 at a certain periodic rate. The periodic rates at which the camera 201 and the radar capture their respective image frames can be the same or different. The images and three-dimensional point cloud images captured by each camera 201 can be annotated with a timestamp. When the camera 201 and the radar capture their respective image frames at different periodic rates, the timestamp can be used to simultaneously or nearly simultaneously select the captured images and three-dimensional point cloud images for further processing (e.g., image fusion).

[0047] A 3D point cloud, also known as a laser point cloud (PCD) or simply a point cloud, can be a collection of numerous points that express the spatial distribution and surface characteristics of an object, obtained by using a laser to acquire the 3D coordinates of each sampling point on the object's surface within a single spatial reference frame. Compared to an image, a 3D point cloud contains richer 3D spatial information, including the distance between the object and the distance acquisition device 202.

[0048] For example, Figure 1 As shown, at time T0, the camera device 201 can capture images of the vehicle 120 and the vehicle 130. At the same time (time T0), the distance acquisition device 202 can also capture a 3D point cloud image within the horizontal coverage area 140, that is, at time T0, the distance information between the vehicle 120 and the distance acquisition device 202, as well as the distance information between the vehicle 130 and the distance acquisition device 202, is acquired.

[0049] In this embodiment, the processor 203 may include one or more processing units. For example, the processor 203 may include, but is not limited to, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0050] In an embodiment, the processor 203 may identify depth information of objects within the captured scene based on an image of the scene captured by the camera device 201 and distance information of the same scene collected by the distance acquisition device 202 at the same time. The objects may be other vehicles, pedestrians, road signs, or obstacles.

[0051] It is understood that the structure shown in this embodiment does not constitute a specific limitation on the depth estimation system. In other embodiments, the depth estimation system may include more or fewer components than shown, or combine or split some components, or arrange the components differently.

[0052] See Figure 2 , Figure 2 A flowchart of a depth estimation method provided in an embodiment of the present application.

[0053] Depth estimation methods can be applied to Figure 1 The depth estimation system 20 is shown. Figure 2 As shown, the depth estimation method may include the following steps:

[0054] S11, acquiring a first image.

[0055] In this embodiment, the depth estimation system can obtain a first image captured by a camera. For example, the camera may be a monocular camera that captures a video. The depth estimation system can then extract a frame from the video as the first image. Alternatively, the monocular camera captures an image and uses the captured image as the first image.

[0056] S12: Input the first image into a pre-trained depth estimation model to obtain a first depth image.

[0057] In some embodiments, the depth estimation model may include an autoencoder (AE) and an image conversion module. After the depth estimation system inputs a first image into the depth estimation model, the autoencoder processes the first image and outputs a disparity map corresponding to the first image. The image conversion module then converts the disparity map and outputs a first depth image.

[0058] In other embodiments, the depth estimation model may not include an image conversion module. The depth estimation model processes the first image and outputs a disparity map corresponding to the first image. The depth estimation system then converts the disparity map and outputs a first depth image.

[0059] The following describes the training method of the depth estimation model in detail.

[0060] Please also refer to Figure 3 , Figure 3 Flowchart of the depth estimation model training method provided in an embodiment of the present application.

[0061] S31, obtaining a first image pair from a training dataset.

[0062] The first image pair includes a first left image and a first right image.

[0063] It can be understood that an image pair refers to two images of the same scene captured by a camera at the same time, including a left image and a right image. The left image and the right image have the same size and the same number of pixels.

[0064] In this embodiment, the training data set may be a data set of images captured by a binocular camera while the vehicle is traveling. The images captured by the binocular camera include an image pair of the same scene captured by two cameras at the same time.

[0065] S32: Input the first left image into a depth estimation model to be trained to obtain a disparity map.

[0066] It is understood that the depth estimation model to be trained is an initialization model, and the parameters of the initialization model can be set as needed.

[0067] S33: Add the first left image and the disparity map to obtain a second right image.

[0068] The second right image is the right image predicted by the depth estimation model. The second right image has the same size and the same number of pixels as the first right image.

[0069] S34, calculating the mean square error and cosine similarity of the pixel values ​​at all corresponding pixel points in the first right image and the second right image.

[0070] Corresponding pixels refer to two pixels in two images that have a corresponding positional relationship. For example, the first right image contains a first pixel, and the second right image contains a second pixel corresponding to the first pixel. The position of the first pixel in the first right image is the same as the position of the second pixel in the second right image.

[0071] It can be understood that the pixels in the image include pixel values ​​of three color channels: red (R), green (G), and blue (B).

[0072] In this embodiment, the formula for calculating the mean square error (MSE) of pixel values ​​at two corresponding pixel points in the first right image and the second right image is shown in formula (1):

[0073]

[0074] Where n is the number of all pixels in the first right image or the second right image, y ik is the pixel value of the kth color channel at the i-th pixel in the first right image, is the pixel value of the kth color channel at the i-th pixel in the second right image.

[0075] In this embodiment, the mean square error (MSE) can be used to measure the difference in pixel values ​​between two corresponding pixels. Minimizing the MSE can minimize the difference in pixel values ​​between the two corresponding pixels. The smaller the MSE value, the higher the prediction accuracy of the depth estimation model. When the MSE is 0, it means that the pixel values ​​of the two corresponding pixels are the same, that is, the predicted value of the depth estimation model is the same as the true value.

[0076] The formula for calculating the cosine similarity cos(θ) of the pixel values ​​at two corresponding pixel points in the first right image and the second right image is shown in formula (2):

[0077]

[0078] In this embodiment, cosine similarity can be used to measure the color difference between two corresponding pixels in a color space. The smaller the angle θ between the RGB values ​​at the two corresponding pixels, the smaller the cosine similarity value, indicating that the colors of the two pixels are more similar. When the angle θ between the RGB values ​​at the two corresponding pixels is 0, the cosine similarity value cos(θ) is 0, indicating that the colors of the two pixels are the same.

[0079] S35 , respectively calculating average values ​​of the mean square errors and cosine similarities of the pixel values ​​at all corresponding pixel points to obtain a first average value of the mean square errors and a second average value of the cosine similarities.

[0080] In this embodiment, each pixel point in the first right image or the second right image is polled in sequence, and the mean square difference of the pixel value at each corresponding pixel point in the first right image and the second right image is calculated in sequence using the above formula (1) until multiple mean square differences of the pixel values ​​at all corresponding pixels are obtained. The average of the multiple mean square differences is then calculated to obtain the first average value.

[0081] Similarly, each pixel point in the first right image or the second right image is polled in sequence, and the cosine similarity of the pixel value at each corresponding pixel point in the first right image and the second right image can be calculated in sequence using the above formula (2) until multiple cosine similarities of the pixel values ​​at all corresponding pixel points are obtained. The average of the multiple cosine similarities is then calculated to obtain the second average value.

[0082] S36: Add the first average value and the second average value to obtain a loss value of the depth estimation model.

[0083] In this embodiment, when the loss value of the depth estimation model is 0, the depth estimation model converges.

[0084] S37, updating various parameters of the depth estimation model through a backpropagation algorithm according to the loss value.

[0085] In this embodiment, the parameters of the depth estimation model are updated through the back propagation algorithm, which can reduce the loss between the true value and the predicted value.

[0086] S38 , looping through steps S31 to S37 , iteratively training the depth estimation model until all first image pairs in the training dataset are trained or the depth estimation model converges.

[0087] In some embodiments, when all first image pairs in the training dataset are trained, the depth estimation model training ends. At this point, the parameters of the depth estimation model with the smallest loss value are selected as the final model parameters.

[0088] In some other embodiments, during the model training process, when the depth estimation model converges, the training is terminated, and the parameters of the converged depth estimation model are used as the final model parameters.

[0089] It can be understood that in this embodiment, the loss value of the depth estimation model combines the mean square error and cosine similarity, which can not only improve the prediction accuracy of the depth estimation model, but also improve the depth estimation model's sensitivity to color, and can distinguish the color differences between individual pixels even in low-texture areas. Using the depth estimation model of this embodiment to obtain a depth image can improve the accuracy of depth information.

[0090] Figure 4 It is a schematic structural diagram of an electronic device 40 according to an embodiment of the present application.

[0091] See Figure 4 , the electronic device 40 may include a processor 41 and a memory 42. The processor 41 may run a computer program or code stored in the memory 42 to implement the depth estimation model training method and the depth estimation method of the embodiment of the present application.

[0092] It can be understood that the specific implementation of the processor 41 is the same as that of the processor 203 and will not be repeated here.

[0093] The memory 42 may include an external memory interface and an internal memory. Among them, the external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 40. The external memory card communicates with the processor 41 via the external memory interface to implement a data storage function. The internal memory can be used to store computer executable program code, and the executable program code includes instructions. The internal memory may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 40 (such as audio data, a phone book, etc.). In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or a universal flash memory (Universal Flash Storage, UFS), etc. The processor 41 executes various functional applications and data processing of the electronic device 40 by running instructions stored in the internal memory and / or instructions stored in a memory provided in the processor 41, such as implementing the depth estimation model training method and depth estimation method of the embodiment of the present application.

[0094] In some embodiments, the electronic device 40 may further include a camera device and a distance acquisition device.

[0095] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 40. In other embodiments of the present application, the electronic device 40 may include more or fewer components than shown, or combine or separate certain components, or arrange the components differently.

[0096] An embodiment of the present application also provides a storage medium for storing a computer program or code. When the computer program or code is executed by a processor, the depth estimation model training method and depth estimation method of the embodiment of the present application are implemented.

[0097] Storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0098] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A depth estimation model training method, characterized in that: The method comprises: Obtaining a first image pair from a training dataset, the first image pair comprising a first left image and a first right image; Inputting the first left image into a depth estimation model to be trained to obtain a disparity map; Adding the first left image to the disparity map to obtain a second right image; Calculating the mean square error and cosine similarity of pixel values ​​at all corresponding pixel points in the first right image and the second right image; Calculating average values ​​of the mean square errors and cosine similarities of the pixel values ​​at all corresponding pixel points, respectively, to obtain a first average value of the mean square errors and a second average value of the cosine similarities; Adding the first average value and the second average value to obtain a loss value of the depth estimation model; The depth estimation model is iteratively trained according to the loss value.

2. The depth estimation model training method according to claim 1, wherein: The iteratively training the depth estimation model according to the loss value comprises: Updating various parameters of the depth estimation model through a back propagation algorithm according to the loss value; The depth estimation model is iteratively trained until all the first image pairs in the training dataset are trained, or until the depth estimation model converges.

3. The depth estimation model training method according to claim 2, wherein: After the first image pairs in the training data set are trained, the method further includes: The parameters of the depth estimation model with the smallest loss value are selected as the final model parameters.

4. The depth estimation model training method according to claim 2, wherein: After the depth estimation model converges, the method further includes: The converged parameters of the depth estimation model are used as final model parameters.

5. The depth estimation model training method according to claim 2, wherein: When the loss value is 0, the depth estimation model converges.

6. The depth estimation model training method according to claim 1, wherein: The mean square error of the pixel values ​​at two corresponding pixel points in the first right image and the second right image is: Wherein, MSE is the mean square error, n is the number of all pixels in the first right image or the second right image, and y ik is the pixel value of the kth color channel at the i-th pixel in the first right image, is the pixel value of the kth color channel at the i-th pixel in the second right image.

7. The depth estimation model training method according to claim 1, wherein: The cosine similarity of the pixel values ​​at two corresponding pixel points in the first right image and the second right image is: Wherein, cos(θ) is the cosine similarity, n is the number of all pixels in the first right image or the second right image, and y ik is the pixel value of the kth color channel at the i-th pixel in the first right image, is the pixel value of the kth color channel at the i-th pixel in the second right image.

8. A depth estimation method, characterized in that: The method comprises: acquiring a first image; Inputting the first image into a pre-trained depth estimation model to obtain a first depth image; The depth estimation model is a model trained using the depth estimation model training method according to any one of claims 1 to 7.

9. The depth estimation method according to claim 8, wherein: Inputting the first image into a pre-trained depth estimation model to obtain a first depth image includes: Inputting the first image into a pre-trained depth estimation model to obtain a disparity map; The disparity map is converted to obtain the first depth image.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor can run a computer program or code stored in the memory to implement the depth estimation model training method as described in any one of claims 1 to 7, or implement the depth estimation method as described in claim 8 or 9.

Images