Depth estimation method, device, electronic device and computer-readable storage medium

Abstract

The present application relates to a depth estimation method and device, an electronic device, and a computer-readable storage medium. The method includes: after acquiring an image to be processed, inputting the image to be processed into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed. Because the preset depth estimation network is a network generated by training based on the semantic segmentation results and depth estimation results of each image frame in the training set, in the process of inputting the image to be processed into the preset depth estimation network for depth estimation, auxiliary adjustments can be made to the depth estimation results based on the semantic segmentation results. Ultimately, the accuracy of the obtained depth information is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a depth estimation method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the rapid development of computer technology, computer vision related technologies have also developed rapidly. Computer vision refers to the use of cameras and computers to replace human eyes to perform machine vision operations such as identification, tracking and measurement of targets, and further image processing operations to generate images that are more suitable for human eye observation or transmission to instrument detection.

[0003] Depth estimation is a very important subfield in the field of computer vision, and its purpose is to obtain the distance between the object and the shooting point, that is, the depth information. Therefore, depth estimation can provide depth information for a series of tasks such as 3D reconstruction, distance perception, SLAM, visual odometer, liveness detection, video interpolation, image reconstruction, etc.

[0004] However, for scenes with very complex depth information, using a traditional depth estimation network to perform depth estimation on images of the scene will result in low accuracy of the depth information obtained. Summary of the invention

[0005] Embodiments of the present application provide a depth estimation method, device, electronic device, and computer-readable storage medium, which can improve the accuracy of the obtained depth information.

[0006] In one aspect, a depth estimation method is provided, comprising:

[0007] Get the image to be processed;

[0008] The image to be processed is input into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed; the preset depth estimation network is a network generated by training based on the semantic segmentation results and depth estimation results of each image frame in the training set.

[0009] On the other hand, a training method for a depth estimation network is provided, comprising:

[0010] Obtaining the depth estimation result according to each image frame in the training set and the intermediate depth estimation network, and obtaining the semantic segmentation result according to each image frame and the intermediate semantic segmentation network;

[0011] The intermediate depth estimation network is trained according to the depth estimation result and the semantic segmentation result to generate the preset depth estimation network.

[0012] In another aspect, a depth estimation device is provided, comprising:

[0013] The module for acquiring the image to be processed is used to acquire the image to be processed;

[0014] The depth estimation module is used to input the image to be processed into a preset depth estimation network for depth estimation to obtain the depth information of the image to be processed; the preset depth estimation network is a network generated by training based on the semantic segmentation results and depth estimation results of each image frame in the training set.

[0015] On the other hand, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the focus control method as described above.

[0016] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0017] After acquiring the image to be processed, the above-mentioned depth estimation method and device, electronic device, and computer-readable storage medium input the image to be processed into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed. Because the preset depth estimation network is a network generated by training based on the semantic segmentation results and depth estimation results of each image frame in the training set, in the process of inputting the image to be processed into the preset depth estimation network for depth estimation, auxiliary adjustments can be made to the depth estimation results based on the semantic segmentation results. Ultimately, the accuracy of the obtained depth information is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A diagram showing an application environment of a depth estimation method in an embodiment;

[0020] Figure 2 is a flow chart of a depth estimation method in one embodiment;

[0021] Figure 3 is a flowchart of a depth estimation method in another embodiment;

[0022] Figure 4 for Figure 3 A flowchart of a method for generating a preset depth estimation network;

[0023] Figure 5 is a schematic diagram of an edge area of ​​a foreground segmentation result in a semantic segmentation result in one embodiment;

[0024] Figure 6 for Figure 4 A flowchart of a method for obtaining a first depth estimation result of an edge area and a second depth estimation result of an internal area from the depth estimation result;

[0025] Figure 7 is a schematic diagram of an internal area of ​​a foreground segmentation result in a semantic segmentation result in one embodiment;

[0026] Figure 8 for Figure 4 A flowchart of a method for training an intermediate depth estimation network based on a first depth estimation result and a second depth estimation result to generate a preset depth estimation network;

[0027] Fig.9A A schematic diagram of a preset depth estimation network training process in one embodiment;

[0028] Fig. 9B A schematic diagram of the network structure of ResNet18 in one embodiment;

[0029] Fig.10 A schematic diagram of a preset depth estimation network training process in yet another embodiment;

[0030] Fig.11 is a schematic diagram of a depth estimation method in yet another embodiment;

[0031] Fig.12 is a schematic diagram of a depth estimation method in a specific embodiment;

[0032] Fig.13 is a structural block diagram of a depth estimation device in one embodiment;

[0033] Fig.14 for Fig.13 The structural block diagram of the preset depth estimation network training module;

[0034] Fig.15 FIG. 1 is a schematic diagram of the internal structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first depth estimation result may be referred to as a second depth estimation result, and similarly, a second depth estimation result may be referred to as a first depth estimation result. Both the first depth estimation result and the second depth estimation result are depth estimation results, but they are not the same depth estimation result.

[0037] Figure 1 FIG. 1 is a schematic diagram of an application environment of a depth estimation method in an embodiment. Figure 1 As shown, the application environment includes an electronic device 120. The electronic device 120 can obtain an image to be processed 110; input the image to be processed 110 into a preset depth estimation network for depth estimation to obtain depth information (depth map 130) of the image to be processed; the preset depth estimation network is a network generated by training based on the semantic segmentation results and depth estimation results of each image frame in the training set. The electronic device 120 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0038] Figure 2 FIG. 1 is a flowchart of a depth estimation method in an embodiment. The depth estimation method in this embodiment is executed at Figure 1 The electronic device in the example is used to describe. Figure 2 As shown, the depth estimation method includes steps 220 to 240, wherein:

[0039] Step 220: Obtain an image to be processed.

[0040] The image to be processed may refer to an image captured by the electronic device through a camera. In other cases, the image to be processed may also be an image stored locally in the electronic device, or an image downloaded from the network by the electronic device, etc., which is not limited in this application. The image to be processed may also be an image acquired by a monocular camera or an image acquired by a binocular camera, which is not limited in this application.

[0041] After the electronic device obtains the image to be processed, it can pre-process the image to be processed. Specifically, the image can be scaled based on bilinear interpolation. For example, the resolution of the image to be processed is sampled from 1241*376 to 1024*320. Of course, this application is not limited to this. Data augmentation processing can also be performed on the image to be processed to increase the data set of the image to be processed. For example, data augmentation can be performed on the image to be processed by changing the image parameters of the image to be processed, or by horizontally flipping the image to be processed. The image parameters here include but are not limited to brightness, contrast, saturation, hue, etc. Of course, in order to increase the speed of depth estimation, data augmentation processing may not be performed.

[0042] After preprocessing the image to be processed to generate a preprocessed image, depth estimation can be performed on the preprocessed image.

[0043] Step 240, input the image to be processed into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed; the preset depth estimation network is a network generated by training based on the semantic segmentation results and depth estimation results of each image frame in the training set.

[0044] Specifically, the image to be processed can be directly input into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed. The preprocessed image can also be input into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed, which is not limited in this application. Here, the architecture of the depth estimation network includes a convolutional neural network, etc., which is not limited in this application.

[0045] The preset depth estimation network is a network generated by training the intermediate depth estimation network based on the semantic segmentation results and depth estimation results of each image frame in the training set. The intermediate depth estimation network here is a network generated by training the initial depth estimation network based on each image frame in the training set. The depth estimation result here is the result generated by inputting each image frame in the training set into the intermediate depth estimation network, for example, the depth estimation result is the depth information of each image frame.

[0046] The semantic segmentation result here is the result generated by inputting each image frame in the training set into the intermediate semantic segmentation network. For example, for each image frame in the training set, a semantic category is assigned to each pixel in the image frame to obtain pixelated dense categories. These categories are the semantic segmentation results of the image frame. Among them, these categories can include various animals, people, sky, beach, green grass, trees, food, vehicles, houses, street lights, etc., which are not limited in this application. Among them, the intermediate semantic segmentation network is a network generated by training based on each image frame in the training set and the initial semantic segmentation network.

[0047] Then, the image to be processed can be input into the preset depth estimation network for depth estimation to obtain the depth information of the image to be processed, and thus obtain the depth map of the image to be processed. The depth map can also be called a distance image, which means that the distance from the shooting device to each point in the scene is recorded in the depth map, reflecting the geometric shape of the objects in the scene. Each pixel value in the depth map represents the distance from the object to the camera's xy plane, generally in mm.

[0048] In the embodiment of the present application, after obtaining the image to be processed, the image to be processed is input into a preset depth estimation network for depth estimation to obtain the depth information of the image to be processed. Because the preset depth estimation network is a network generated by training based on the semantic segmentation results and depth estimation results of each image frame in the training set, in the process of inputting the image to be processed into the preset depth estimation network for depth estimation, auxiliary adjustments can be made to the depth estimation results based on the semantic segmentation results. Ultimately, the accuracy of the depth information obtained is improved.

[0049] In the previous embodiment, a depth estimation method is provided. After obtaining an image to be processed, the image to be processed is input into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed. The preset depth estimation network here is pre-trained. Then, in one embodiment, a specific training method for the preset depth estimation network is further explained. Specifically, Figure 3 As shown, a depth estimation method is provided, which also includes:

[0050] Step 260, obtaining a depth estimation result according to each image frame in the training set and the intermediate depth estimation network, and obtaining a semantic segmentation result according to each image frame and the intermediate semantic segmentation network.

[0051] The intermediate depth estimation network here is a network generated by training based on each image frame in the training set and the initial depth estimation network. Specifically, each image frame in the training set is input into the intermediate depth estimation network for training, and a depth estimation result can be generated, for example, the depth information of each image frame.

[0052] The intermediate semantic segmentation network here is a network generated by training the initial semantic segmentation network based on each image frame in the training set. The architecture of the semantic segmentation network includes an encoder-decoder network. The encoder usually includes a pre-trained category network, such as VGG (Visual Geometry Group), ResNet network, etc. After the encoder, a decoder network is connected. The task of the decoder network is to semantically project the discriminable features (lower resolution) learned by the encoder to the pixel space (higher resolution) to obtain dense categories.

[0053] Specifically, each image frame in the training set is input into the intermediate semantic segmentation network for training, and the semantic segmentation result can be generated. For each image frame in the training set, a semantic category is assigned to each pixel in the image frame to obtain pixelated dense categories, which are the semantic segmentation results of the image frame.

[0054] Step 280: training the intermediate depth estimation network according to the depth estimation result and the semantic segmentation result to generate a preset depth estimation network.

[0055] After inputting each image frame in the training set into the intermediate depth estimation network for training to generate a depth estimation result, and inputting each image frame in the training set into the intermediate semantic segmentation network for training to generate a semantic segmentation result, the intermediate depth estimation network can be trained based on the depth estimation result and semantic segmentation result of each image frame to generate a preset depth estimation network. Compared with the traditional method, the depth estimation network is trained only based on the depth estimation result, that is, each pixel is treated equally, and the semantic segmentation information of each pixel cannot be considered outside the depth information. Among them, pixels with different semantic segmentation information can have different depth features. For example, for pixels whose semantic segmentation information is sky, their depth value is greater than that of pixels whose semantic segmentation information is human. Therefore, the intermediate depth estimation network can be trained based on the different semantic segmentation information and depth estimation information of each pixel in the image frame to generate a preset depth estimation network.

[0056] In this application, because the semantic segmentation results can more accurately distinguish the edge and the interior of an object, the depth estimation results of the edge and the interior of an object in each image frame can be adjusted based on the semantic segmentation results of the image frame. Finally, based on the adjusted depth estimation results, the intermediate depth estimation network can be trained to generate a preset depth estimation network.

[0057] By presetting the depth estimation network, the depth information at the edge and inside the object can be better estimated to generate a more accurate depth map.

[0058] In an embodiment of the present application, each image frame in the training set is input into the intermediate depth estimation network for training, and a depth estimation result can be generated. And each image frame in the training set is input into the intermediate semantic segmentation network for training, and a semantic segmentation result can be generated. Since pixels with different semantic segmentation information can have different depth features, the intermediate depth estimation network can be trained based on the depth estimation result and semantic segmentation result of each image frame to generate a preset depth estimation network.

[0059] Therefore, by presetting the depth estimation network, the depth information at the edge and inside the object can be better estimated to generate a more accurate depth map.

[0060] In the previous embodiment, the training process of the preset depth estimation network is described. In this embodiment, step 280 is described in detail, and the intermediate depth estimation network is trained according to the depth estimation result and the semantic segmentation result to generate the preset depth estimation network, including:

[0061] Calculate the value of the loss function according to the depth estimation result and the semantic segmentation result;

[0062] The intermediate depth estimation network is trained based on the value of the loss function to generate the preset depth estimation network.

[0063] In an embodiment of the present application, after each image frame in the training set is input into the intermediate depth estimation network for training to generate a depth estimation result, and each image frame in the training set is input into the intermediate semantic segmentation network for training to generate a semantic segmentation result, the intermediate depth estimation network can be trained based on the depth estimation result and the semantic segmentation result of each image frame to generate a preset depth estimation network. Specifically, the value of the loss function is calculated based on the depth estimation result and the semantic segmentation result, and the intermediate depth estimation network is trained based on the value of the loss function to generate the preset depth estimation network. The loss function is used to improve the accuracy of the generated preset depth estimation network.

[0064] In the previous embodiment, the training process of the preset depth estimation network is described. Figure 4 As shown, in this embodiment, step 280 is described in detail, and the specific implementation steps of training the intermediate depth estimation network according to the depth estimation result and semantic segmentation result of the current frame to generate the preset depth estimation network include:

[0065] Step 282: Determine the edge region and the internal region of the foreground in the semantic segmentation result from the semantic segmentation result.

[0066] Each image frame in the training set is input into the intermediate semantic segmentation network for training. After the semantic segmentation result is generated, the edge area and internal area of ​​the foreground in the semantic segmentation result are further determined from the semantic segmentation result.

[0067] Since a semantic category is assigned to each pixel in each image frame in the training set to obtain pixelated dense categories, these categories are the semantic segmentation results of the image frames. Therefore, the foreground segmentation mask can be determined from the semantic segmentation results, wherein the foreground area can be extracted from the image to be processed based on the foreground segmentation mask. Specifically, the pixel index of the foreground can be extracted from the semantic segmentation results, and the foreground segmentation mask can be obtained based on the pixel index of the foreground. Among them, the pixel index can be understood as the coordinates of the pixel points belonging to the same category on each image frame are divided into the category, thus forming a pixel index.

[0068] Assume that the categories in the semantic segmentation result of the image frame include 19 categories, that is, each pixel can be divided into 19 categories, which are represented by numbers 0-18. Among them, [5,6,7,11,12,13,14,15,16,17,18] are the labels of the foreground categories, and these labels correspond to the specific categories [pole, traffic light, traffic sign, person, rider, car, truck, bus, train, motorcycle, bicycle]. For each foreground category, the coordinates (y, x) of the pixels corresponding to these foreground categories are obtained from the image frame, and the coordinates of the pixels belonging to the same foreground category are divided into the same category. For each background category, the operation is performed in the same way, so that the pixel index of the entire image frame is obtained. Among them, the coordinates (y, x) of the pixel indicate that the pixel is located in the yth column and the xth row in the image frame.

[0069] Based on the pixel index of each foreground category, the foreground segmentation mask can be obtained. Specifically, an all-0 matrix consistent with the image frame size is obtained, and a target point consistent with the coordinates of the pixel in the pixel index corresponding to the foreground category is determined in the all-0 matrix, and the target point is set to 1, so that the foreground segmentation mask is obtained. Then, the foreground area can be extracted from the image frame based on the foreground segmentation mask.

[0070] After the foreground area can be extracted from the image frame based on the foreground segmentation mask, the edge area and the internal area of ​​the foreground can be further extracted from the foreground area. Specifically, the foreground area can be subjected to a preset process to extract the edge area and the internal area of ​​the foreground. The preset process here includes various processing methods in morphological processing, which is not limited in this application.

[0071] Step 284 : Obtain a first depth estimation result of the edge area and a second depth estimation result of the internal area from the depth estimation results according to the edge area and the internal area.

[0072] After determining the edge region and the internal region of the foreground in the semantic segmentation result from the semantic segmentation result, a first depth estimation result of the edge region and a second depth estimation result of the internal region are obtained from the depth estimation result based on the edge region and the internal region. Specifically, the first depth estimation results of multiple pairs of pixels in the edge region and the second depth estimation results of multiple pairs of pixels in the internal region can be obtained from the depth estimation result of the image frame.

[0073] Step 286 : Based on the first depth estimation result and the second depth estimation result, train the intermediate depth estimation network to generate a preset depth estimation network.

[0074] For each image frame, based on the first depth estimation results of multiple pairs of pixel points in the edge area of ​​the image frame and the second depth estimation results of multiple pairs of pixel points in the internal area of ​​the image frame, the intermediate depth estimation network is trained to generate a preset depth estimation network.

[0075] Specifically, for each image frame in the training set, each image frame is annotated in advance to obtain the annotated depth result of the image frame. The annotated depth result here includes the annotated depth result of the edge area and the annotated depth result of the internal area. Then, the value of the loss function can be calculated based on the first depth estimation result of multiple pairs of pixels in the edge area of ​​the image frame and the annotated depth result of the edge area, the second depth estimation result of multiple pairs of pixels in the internal area, and the annotated depth result of the internal area. The parameters of the intermediate depth estimation network are adjusted based on the value of the loss function to generate a preset depth estimation network.

[0076] Due to the traditional depth estimation method, the accuracy of the depth information obtained by estimating the depth of the edge of the object on the image frame is low. Especially for crowded scenes or occluded scenes, due to the large number of objects in the scene and the mutual occlusion between the objects, the depth information of the image captured in the scene is very complex. Therefore, when performing depth estimation on images captured in crowded scenes and occluded scenes, the accuracy of the depth information obtained is even lower.

[0077] In an embodiment of the present application, the edge area and internal area of ​​the foreground in the semantic segmentation result are first determined from the semantic segmentation result, that is, the edge of the foreground or the edge of the object in the image frame is clarified by the semantic segmentation result. Among them, the edge area and the internal area of ​​the foreground have different depth features. Then, the first depth estimation result of the edge area and the second depth estimation result of the internal area can be obtained from the depth estimation result. Based on the first depth estimation result and the second depth estimation result and the annotated depth estimation result, the intermediate depth estimation network is trained to generate a preset depth estimation network. In this way, for crowded scenes or occluded scenes, the trained preset depth estimation network can more accurately calculate the depth information of the edge area of ​​the foreground.

[0078] In the previous embodiment, the training process of the preset depth estimation network is described. In the embodiment of the present application, the specific implementation steps of determining the edge area and the internal area of ​​the foreground in the semantic segmentation result from the semantic segmentation result are further described in detail, including:

[0079] Performing an expansion operation on the foreground segmentation result in the semantic segmentation result to obtain an outer edge, and performing an erosion operation on the foreground segmentation result to obtain an inner edge;

[0080] Determine the edge area of ​​the foreground according to the outer edge and the inner edge;

[0081] The area within the edge area in the foreground segmentation result is regarded as the internal area of ​​the foreground.

[0082] Combination Figure 5 FIG. 1 is a schematic diagram of a foreground segmentation result in a semantic segmentation result in an embodiment. Figure 5 As shown, the foreground segmentation result (i.e., the segmentation boundary) in the image frame is 520, and it is assumed that the real contour boundary of the foreground in the image frame is 540. Then, the foreground segmentation result in the semantic segmentation result is expanded to obtain the outer edge, specifically, the segmentation boundary 520 is expanded to obtain the outer edge 560. Then, the foreground segmentation result is eroded to obtain the inner edge, specifically, the segmentation boundary 520 is eroded to obtain the outer edge 580. From a mathematical point of view, the expansion operation or the erosion operation is the process of convolving an image (or a part of an image, such as region A) with a convolution kernel (such as matrix B). Here, the expansion operation can be understood as an operation to find the local maximum value, and conversely, the erosion is an operation to find the local minimum value. Since the expansion process or the erosion process is to process the binary image, the highlight area in the binary image can be enlarged by the expansion operation, and the highlight area in the binary image can be reduced by the erosion process.

[0083] After obtaining the outer edge and the inner edge based on the foreground segmentation result, the edge region of the foreground can be determined according to the outer edge and the inner edge. Specifically, the region between the outer edge and the inner edge in the same foreground segmentation result can be used as the edge region of the foreground.

[0084] Furthermore, the area within the edge area in the same foreground segmentation result is regarded as the internal area of ​​the foreground.

[0085] In an embodiment of the present application, for each semantic segmentation result in the image frame, a dilation operation is performed on the foreground segmentation result in the semantic segmentation result to obtain an outer edge, and an erosion operation is performed on the foreground segmentation result to obtain an inner edge. The edge area of ​​the foreground is then determined based on the outer edge and the inner edge, and the area within the edge area in the foreground segmentation result is used as the inner area of ​​the foreground. In this way, the edge area and the inner area of ​​the foreground are determined based on the semantic segmentation result. Then, a first depth estimation result of the edge area and a second depth estimation result of the inner area can be obtained from the depth estimation result. Finally, based on the first depth estimation result and the second depth estimation result, the intermediate depth estimation network is trained to generate a preset depth estimation network. In this way, for crowded scenes or occluded scenes, the trained preset depth estimation network can more accurately calculate the depth information of the edge area of ​​the foreground.

[0086] Continuing with the previous embodiment, Figure 6 As shown, step 284, obtaining a first depth estimation result of the edge area and a second depth estimation result of the internal area from the depth estimation results of the edge area and the internal area, includes:

[0087] Step 284a, for the m first pixel points on the outer edge, determine the m second pixel points on the inner edge that are closest to the m first pixel points, and obtain a first point pair set based on the m first pixel points and the corresponding m second pixel points, where m is a positive integer.

[0088] Combination Figure 5 As shown, for each first pixel point on the outer edge 560, a second pixel point closest to the first pixel point is determined on the inner edge. out , determine the first pixel point P on the inner edge 580 out The second pixel point P that is closest to in Based on multiple groups of first pixel points P out and the second pixel P in , and get the first set of point pairs.

[0089] Step 284b, in the internal area, determine n third pixel points and n fourth pixel points corresponding to the n third pixel points, and obtain a second point pair set based on the n third pixel points and the n fourth pixel points, where n is a positive integer; the positional relationship between the third pixel point and the fourth pixel point meets the preset conditions.

[0090] Combination Figure 7 As shown in FIG. 1 , the area within the edge area in the same foreground segmentation result is taken as the internal area of ​​the foreground. Then, multiple groups of third pixel points P can be randomly determined in the internal area. in0 and a fourth pixel point P corresponding to the third pixel point in1 , that is, randomly determine pixel pairs in the internal area. Then, based on multiple groups of third pixel points P in0 and the fourth pixel P in1 , and obtain a second set of point pairs. Here, the positional relationship between each set of third pixel points and fourth pixel points satisfies a unified preset condition. For example, the preset condition here can be limited to the third pixel point P in0 and the fourth pixel P in1 The distance between them is a preset distance, and of course, this application does not limit this.

[0091] Step 284c: Obtain, from the depth estimation results, a first depth estimation result for each point pair in the first point pair set and a second depth estimation result for each point pair in the second point pair set.

[0092] Based on multiple groups of first pixel points P out and the second pixel P in Get the first point pair set, and based on multiple sets of third pixel points P in0 and the fourth pixel P in1 After obtaining the second point pair set, the first depth estimation result of each point pair in the first point pair set and the second depth estimation result of each point pair in the second point pair set can be obtained from the depth estimation result.

[0093] In the embodiment of the present application, for each foreground segmentation result in the image frame, for each first pixel on the outer edge, the second pixel closest to the first pixel is determined on the inner edge, and a first point pair set is obtained based on the first pixel and the corresponding second pixel. In the inner area, a third pixel and a fourth pixel corresponding to the third pixel are determined, and a second point pair set is obtained based on the third pixel and the fourth pixel, wherein the positional relationship between the third pixel and the fourth pixel satisfies a preset condition. From the depth estimation result, the first depth estimation result of each point pair in the first point pair set and the second depth estimation result of each point pair in the second point pair set are obtained. Because the first point pair set is located in the edge area of ​​the foreground and the second point pair set is located in the inner area of ​​the foreground, based on the first depth estimation result of the point pair in the first point pair set and the second depth estimation result of each point pair in the second point pair set, the intermediate depth estimation network can be trained based on the first depth estimation result, the second depth estimation result, and the annotated depth estimation result to generate a preset depth estimation network. In this way, for crowded scenes or occluded scenes, the trained preset depth estimation network can more accurately calculate the depth information of the edge area of ​​the foreground.

[0094] Continuing from the previous embodiment, how to determine the first point pair set and the second point pair set is described. In this embodiment, it is further described that the positional relationship between the third pixel point and the fourth pixel point satisfies the preset condition, including:

[0095] The distance between the third pixel point and the fourth pixel point in the first direction is less than the first preset distance threshold, and the distance between the third pixel point and the fourth pixel point in the second direction is less than the second preset distance threshold; wherein the first direction is perpendicular to the second direction.

[0096] Since the first direction and the second direction are perpendicular to each other, Figure 7 As shown, a coordinate system is established on the image frame, with the first direction as the X-axis of the coordinate system and the second direction as the Y-axis of the coordinate system. in0 With the fourth pixel P in1 The distance S1 in the first direction is less than the first preset distance threshold r1, and the third pixel point P in0 With the fourth pixel P in1 The distance S2 between the distances in the second direction is less than the second preset distance threshold r2. The first preset distance threshold r1 and the second preset distance threshold r2 may be the average radius of the inner area of ​​the foreground or the shortest length of the inner area of ​​the foreground. Of course, this application does not limit this. The first preset distance threshold r1 and the second preset distance threshold r2 may be equal or unequal.

[0097] From the first direction and the second direction, the third pixel point Pin0 With the fourth pixel P in1 The positional relationship between them can be limited to ensure that the fourth pixel point P in1 At the third pixel P in0 The predetermined range of the second point pair is increased to improve the probability that the second point pair set is in the inner area of ​​the foreground. Thus, the accuracy of the determined second point pair set is improved. Figure 7 The first direction and the second direction shown in the figure may also be other directions perpendicular to each other, which is not limited in the present application.

[0098] In the embodiment of the present application, in the internal area, a third pixel point and a fourth pixel point corresponding to the third pixel point are determined, and a second point pair set is obtained based on the third pixel point and the fourth pixel point, wherein the distance between the third pixel point and the fourth pixel point in the first direction is less than the first preset distance threshold, the distance between the third pixel point and the fourth pixel point in the second direction is less than the second preset distance threshold, and the first direction is perpendicular to the second direction.

[0099] Because the first point pair set is located at the edge area of ​​the foreground, and the second point pair set is located at the inner area of ​​the foreground, based on the first depth estimation result of the point pairs in the first point pair set and the second depth estimation result of each point pair in the second point pair set, the intermediate depth estimation network can be trained based on the first depth estimation result, the second depth estimation result, and the annotated depth estimation result to generate a preset depth estimation network. In this way, for crowded scenes or occluded scenes, the trained preset depth estimation network can more accurately calculate the depth information of the edge area of ​​the foreground.

[0100] In the foregoing embodiment, the training process of the preset depth estimation network is described. In this embodiment, it is further described in detail. Figure 8 As shown, step 286, based on the first depth estimation result and the second depth estimation result, the intermediate depth estimation network is trained to generate the specific implementation steps of the preset depth estimation network, including:

[0101] Step 286a, calculate the value of the first semantic loss function according to the first depth estimation results and the first labeled depth results of each first point pair in the first point pair set; the first semantic loss function is used to penalize the first point pair whose difference between the first depth estimation results is less than the first preset depth difference threshold.

[0102] Specifically, for each first point pair, a first semantic loss function is constructed based on the difference between the first depth estimation result of the first pixel in the first point pair and the first depth estimation result of the second pixel in the first point pair. Because the first point pair is a point pair at the inner edge and outer edge of the foreground, the greater the difference in depth values ​​between the first point pairs, the more certain it is that the first point pair is in the edge area of ​​the foreground. Therefore, in the first semantic loss function, the absolute value of the difference between the first depth estimation result of the first pixel and the first depth estimation result of the second pixel is negatively valued as a power exponent. This implements the use of the first semantic loss function to penalize the first point pair whose difference between the first depth estimation results is less than the first preset depth difference threshold. Among them, the first semantic loss function L edge The specific formula is as follows:

[0103]

[0104] in, Refers to the first pixel P out The depth value of Refers to the second pixel P in The depth value of . N represents the total number of first point pairs in the first point pair set.

[0105] Step 286b, calculate the value of the second semantic loss function according to the second depth estimation results and the second annotated depth results of each second point pair in the second point pair set; the second semantic loss function is used to penalize the second point pairs whose difference between the second depth estimation results is greater than the second preset depth difference threshold.

[0106] For each second point pair, a second semantic loss function is constructed based on the difference between the second depth estimation result of the third pixel in the second point pair and the second depth estimation result of the fourth pixel in the second point pair. Because the second point pair is a point pair in the internal area of ​​the foreground, the smaller the difference in depth value between the second point pairs, the more certain it is that the second point pair is in the internal area of ​​the foreground. Therefore, in the second semantic loss function, the square sum of the difference between the second depth estimation result of the third pixel and the second depth estimation result of the fourth pixel is taken. This implements the use of the second semantic loss function to penalize the second point pair whose difference between the second depth estimation results is greater than the second preset depth difference threshold. Among them, the second semantic loss function L inside The specific formula is as follows:

[0107]

[0108] in, Refers to the third pixel P in0 The depth value of Refers to the fourth pixel P in1N represents the total number of second point pairs in the second point pair set.

[0109] Step 286c: adjust the parameters of the intermediate depth estimation network according to the value of the first semantic loss function and the value of the second semantic loss function to generate a preset depth estimation network.

[0110] For each image frame in the training set, determine the edge area and internal area of ​​the foreground in the semantic segmentation results from the semantic segmentation results of the image frames. For each first pixel on the outer edge, determine the second pixel closest to the first pixel on the inner edge, and obtain a first point pair set based on the first pixel and the corresponding second pixel. In the internal area, determine a third pixel and a fourth pixel corresponding to the third pixel, and obtain a second point pair set based on the third pixel and the fourth pixel. From the depth estimation results, obtain the first depth estimation results for each point pair in the first point pair set and the second depth estimation results for each point pair in the second point pair set.

[0111] The value of the first semantic loss function is calculated based on the first depth estimation result and the first labeled depth result of each first point pair in the first point pair set. Then, the value of the second semantic loss function is calculated based on the second depth estimation result and the second labeled depth result of each second point pair in the second point pair set. Among them, the first labeled depth result and the second labeled depth result are both the results of pre-labeling the training set, which can be manual labeling or generated by other methods, and this application does not limit this. Finally, the parameters of the intermediate depth estimation network are adjusted according to the values ​​of the first semantic loss function and the second semantic loss function to generate a preset depth estimation network.

[0112] In an embodiment of the present application, for the first point pair at the inner edge and outer edge of the foreground, the constructed first semantic loss function can be used to penalize the first point pair whose difference between the first depth estimation results is less than the first preset depth difference threshold. Thus, it is achieved that the first point pair with a higher probability of being at the inner edge and the outer edge is screened out from the first point pair. For the second point pair in the internal area of ​​the foreground, the constructed second semantic loss function can be used to penalize the second point pair whose difference between the second depth estimation results is greater than the second preset depth difference threshold. Thus, it is achieved that the second point pair with a higher probability of being in the internal area is screened out from the second point pair. Finally, the parameters of the intermediate depth estimation network can be adjusted according to the values ​​of the first semantic loss function and the second semantic loss function to generate a preset depth estimation network. It is achieved that through the preset depth estimation network, the depth information of the edge area and the internal area of ​​the foreground can be calculated more accurately.

[0113] In the above embodiment, a training method for a preset depth estimation network is described, including: obtaining a depth estimation result based on an image frame in a training set and an intermediate depth estimation network, and obtaining a semantic segmentation result based on an image frame and an intermediate semantic segmentation network; training the intermediate depth estimation network based on the depth estimation result and the semantic segmentation result to generate a preset depth estimation network. Wherein, obtaining a depth estimation result based on an image frame in a training set and an intermediate depth estimation network includes:

[0114] Input each current frame in the training set into the initial depth estimation network for processing to generate a depth prediction result;

[0115] Input the adjacent frames of each current frame in the training set into the pose estimation network to generate pose transformation information;

[0116] According to the depth prediction results and pose transformation information, the initial depth estimation network is trained to generate an intermediate depth estimation network;

[0117] The depth estimation result of the current frame is obtained through the intermediate depth estimation network.

[0118] Combination Fig.9A FIG. 1 is a schematic diagram of a preset depth estimation network training process in one embodiment. Fig.9A As shown, a set of continuous frames {I t-1 ,I t ,I t+1}, where I t Indicates the current frame, I t′ ∈{I t-1 ,I t+1}I t First, the current frame I in the training set t Input into the initial depth estimation network for processing to generate the depth prediction result D t . Here, the initial depth estimation network mainly uses continuous frame images for self-supervised training, or self-supervised learning. Self-supervised learning can be regarded as an "ideal state" of machine learning. The model learns directly from unlabeled data without the need for labeled data. Among them, the initial depth estimation network includes a first feature extraction network and a first upsampling network. The first feature extraction network can be used to extract the depth feature of the current frame, and then the first depth feature of the current frame is input into the first upsampling network for upsampling processing to generate a depth prediction result D t Among them, the first feature extraction network here can use ResNet18, ResNet50, ResNet101, etc. as the shared backbone network. The specific network structure of ResNet18 and the output size of each layer of the network can be referred to Fig. 9B As shown. The first feature extraction network here can also use MobileNet, etc. This application does not limit the specific network structure of ResNet18 and the output size of each layer of the network.

[0119] Secondly, the adjacent frames I of the current frame in the training set t′ Input to the pose estimation network to generate pose transformation information T t′→t ;

[0120] Again, according to the depth prediction result D t And posture transformation information T t′→t , train the initial depth estimation network and generate an intermediate depth estimation network.

[0121] Furthermore, the depth prediction result D t And posture transformation information T t′→t , generate a synthetic image I t′→t ; According to the synthetic image I t′→t and the current frame I t , calculate the value of the projection loss function; train the initial depth estimation network through the value of the projection loss function to generate an intermediate depth estimation network.

[0122] Finally, the current frame is input into the intermediate depth estimation network for depth estimation to obtain the depth estimation result of the current frame.

[0123] In the embodiment of the present application, when obtaining the depth estimation result based on the image frames in the training set and the intermediate depth estimation network, the depth estimation network is trained twice. Among them, the first time is to input the current frame in the training set into the initial depth estimation network for processing to generate a depth prediction result. Then, the second time is to input the adjacent frames of the current frame in the training set into the pose estimation network to generate pose transformation information; then, based on the depth prediction result and the pose transformation information, the initial depth estimation network is trained to generate an intermediate depth estimation network; and the depth estimation result of the current frame is obtained through the intermediate depth estimation network.

[0124] The depth prediction result obtained by the first input to the initial depth estimation network is used as the input to the initial depth estimation network for the second input. The initial depth estimation network is trained again in combination with the pose transformation information between the current frame and the adjacent frame to generate an intermediate depth estimation network. Finally, the current frame is input to the intermediate depth estimation network for processing to obtain the depth estimation result of the current frame. Obviously, after two depth estimation network trainings, the intermediate depth estimation network has a higher accuracy.

[0125] Continuing from the previous embodiment, the initial depth estimation network is trained according to the depth prediction result and the pose transformation information to generate an intermediate depth estimation network, including:

[0126] Generate a synthetic image based on the depth prediction results and pose transformation information;

[0127] Calculate the value of the projection loss function based on the synthesized image and the current frame;

[0128] The initial depth estimation network is trained by projecting the value of the loss function to generate an intermediate depth estimation network.

[0129] Specifically, according to the depth prediction result D t And posture transformation information T t′→t , generate a synthetic image I t′→t ; According to the synthetic image I t′→t and the current frame I t , calculate the value of the projection loss function; train the initial depth estimation network through the value of the projection loss function to generate an intermediate depth estimation network.

[0130] Among them, the specific formula of the projection loss function is as follows:

[0131]

[0132] Among them, the full name of SSIM is structural similarity index, which is a measure of the similarity between two images, and α is the weight coefficient.

[0133] The projection loss function is obtained by minimizing I t′→t with I t The apparent error is used to guide network training. The apparent error refers to the difference between the two images, which consists of two parts: one is the L1-based error of the two images, and the other is the structural similarity error of the two images obtained by the SSIM algorithm.

[0134] In the embodiment of the present application, after the current frame in the training set is input into the initial depth estimation network for processing and the depth prediction result is generated, since the number of depth prediction results generated based on the current frame is limited, the amount of data in the training set is increased by using adjacent frames. Specifically, according to the depth prediction result D t And the pose transformation information T between the current frame and the adjacent frame t′→t , the initial depth estimation network is trained to generate an intermediate depth estimation network. Thus, the amount of data in the training set is increased by using adjacent frames, and the accuracy of the generated intermediate depth estimation network is improved based on the current frame and adjacent frames.

[0135] In the above embodiment, a training method for a preset depth estimation network is described, including: obtaining a depth estimation result based on an image frame and an intermediate depth estimation network in a training set, and obtaining a semantic segmentation result based on an image frame and an intermediate semantic segmentation network; training the intermediate depth estimation network based on the depth estimation result and the semantic segmentation result to generate a preset depth estimation network. Wherein, obtaining a semantic segmentation result based on an image frame and an intermediate semantic segmentation network also includes:

[0136] First, each current frame in the training set is input into the initial semantic segmentation network to obtain the semantic segmentation prediction result.

[0137] Specifically, the architecture of the semantic segmentation network includes an encoder-decoder network, and the encoder usually includes a pre-trained category network, such as VGG, ResNet network, etc. The current frame in the training set is input into the initial semantic segmentation network to obtain a semantic segmentation prediction result. That is, the semantic category corresponding to each pixel in the current frame is obtained. These semantic categories may include various animals, people, sky, beach, green grass, trees, food, vehicles, houses, street lights, etc., which are not limited in this application. Among them, the initial semantic segmentation network includes a second feature extraction network and a second upsampling network. The second feature extraction network here can be consistent with the first feature extraction network in the initial depth estimation network, and of course, it may not be consistent. The semantic segmentation features of the current frame can be extracted through the second feature extraction network, and then the semantic segmentation features of the current frame are input into the second upsampling network for upsampling processing to generate a semantic segmentation prediction result S. t Among them, the second feature extraction network here can use ResNet18 as the shared backbone network. The specific network structure of ResNet18 and the output size of each layer of the network can be referred to Figure 2 This application does not limit the specific network structure of ResNet18 and the output size of each layer of the network.

[0138] Secondly, based on the semantic segmentation prediction results and the semantic annotation dataset, the initial semantic segmentation network is trained to generate an intermediate semantic segmentation network.

[0139] Among them, the semantic annotation dataset L tis a pre-trained semantic dataset. Specifically, the KITTI semantic instance segmentation dataset can be obtained first, where the KITTI semantic instance segmentation dataset contains 200 semantic segmentation training images and 200 unlabeled test images. Then, through the PSPNet network, continue to train the 200 semantic segmentation training images of KITTI to generate a more accurate target semantic segmentation dataset. Finally, according to the semantic segmentation prediction results and the semantic segmentation dataset (target semantic segmentation dataset), a cross entropy loss function is constructed; the initial semantic segmentation network is trained by calculating the value of the cross entropy loss function to generate an intermediate semantic segmentation network.

[0140] Among them, the cross entropy loss function can be used in classification problems, especially when neural networks are used for classification problems, cross entropy can be used as a loss function. In addition, since cross entropy involves calculating the probability of each category, cross entropy almost always appears with the sigmoid (or softmax) function.

[0141] Finally, the semantic segmentation result of the current frame is obtained through the intermediate semantic segmentation network.

[0142] By inputting the current frame into the intermediate semantic segmentation network for semantic segmentation processing, the semantic segmentation result of the current frame can be generated, that is, the semantic category corresponding to each pixel in the current frame can be obtained.

[0143] In an embodiment of the present application, first, the current frame in the training set is input into the initial semantic segmentation network to obtain a semantic segmentation prediction result. Secondly, according to the semantic segmentation prediction result and the semantic annotation data set, the initial semantic segmentation network is trained to generate an intermediate semantic segmentation network. Specifically, according to the semantic segmentation prediction result and the semantic segmentation data set, a cross entropy loss function is constructed; the initial semantic segmentation network is trained by the cross entropy loss function to generate an intermediate semantic segmentation network. Finally, the semantic segmentation result of the current frame is obtained through the intermediate semantic segmentation network. The cross entropy loss function is used to improve the accuracy of classification, thereby improving the accuracy of the semantic segmentation result of the current frame.

[0144] In one embodiment, a feature extraction network is included before the initial depth estimation network and the initial semantic segmentation network; the initial depth estimation network includes a first upsampling network, and the initial semantic segmentation network includes a second upsampling network.

[0145] like Fig.10 FIG. 2 is a schematic diagram of a preset depth estimation network training process in another embodiment. Fig.10 As shown, a set of continuous frames {I t-1 ,I t ,I t+1}, where It Indicates the current frame, I t′ ∈{I t-1 ,I t+1}I t First, the current frame I in the training set t Input to the feature extraction network for feature extraction, which can extract the depth feature of the current frame or the semantic feature of the current frame. After the feature extraction network, an initial depth estimation network and the initial semantic segmentation network are included, wherein the initial depth estimation network includes a first upsampling network, and the initial semantic segmentation network includes a second upsampling network.

[0146] The extracted features are respectively input into the initial depth estimation network and the initial semantic segmentation network for processing. Specifically, the extracted depth features are respectively input into the initial depth estimation network for processing to generate the depth prediction result D t The initial depth estimation network includes a first upsampling network. The first depth feature of the current frame is input into the first upsampling network for upsampling processing to generate a depth prediction result D t Among them, the first feature extraction network here can use ResNet18, ResNet50, ResNet101, etc. as the shared backbone network. The specific network structure of ResNet18 and the output size of each layer of the network can be referred to Fig. 9B As shown. The first feature extraction network here can also use MobileNet, etc. This application does not limit the specific network structure of ResNet18 and the output size of each layer of the network.

[0147] On the one hand, the adjacent frames I of the current frame in the training set t′ Input to the pose estimation network to generate pose transformation information T t′→t ; Again, according to the depth prediction result D t And posture transformation information T t′→t , train the initial depth estimation network to generate an intermediate depth estimation network. Further, the depth prediction result D t And posture transformation information T t′→t , generate a synthetic image I t′→t ; According to the synthetic image I t′→t and the current frame I t , calculate the value of the projection loss function; train the initial depth estimation network through the value of the projection loss function to generate an intermediate depth estimation network. Finally, input the current frame into the intermediate depth estimation network for depth estimation to obtain the depth estimation result of the current frame.

[0148] On the other hand, each current frame I in the training set tInput into the initial semantic segmentation network to obtain the semantic segmentation prediction result S t . Secondly, according to the semantic segmentation prediction result S t And semantic annotation dataset L t , train the initial semantic segmentation network and generate an intermediate semantic segmentation network. Finally, obtain the semantic segmentation result of the current frame through the intermediate semantic segmentation network.

[0149] In an embodiment of the present application, the depth prediction result obtained by the first input to the initial depth estimation network is used as the input to the initial depth estimation network for the second input, and then combined with the pose transformation information between the current frame and the adjacent frame, the initial depth estimation network is trained again to generate an intermediate depth estimation network. In this way, the current frame is finally input to the intermediate depth estimation network for processing to obtain the depth estimation result of the current frame. Obviously, after two depth estimation network trainings, the intermediate depth estimation network has a higher accuracy.

[0150] The current frame in the training set is input into the initial semantic segmentation network to obtain the semantic segmentation prediction result. Secondly, according to the semantic segmentation prediction result and the semantic annotation data set, the initial semantic segmentation network is trained to generate an intermediate semantic segmentation network. Specifically, according to the semantic segmentation prediction result and the semantic segmentation data set, a cross entropy loss function is constructed; the initial semantic segmentation network is trained through the cross entropy loss function to generate an intermediate semantic segmentation network. Finally, the semantic segmentation result of the current frame is obtained through the intermediate semantic segmentation network. The cross entropy loss function is used to improve the accuracy of classification, thereby improving the accuracy of the semantic segmentation result of the current frame.

[0151] In one embodiment, Fig.11 As shown, a training method for a depth estimation network is provided, including:

[0152] Step 1120, obtaining a depth estimation result according to each image frame in the training set and the intermediate depth estimation network, and obtaining a semantic segmentation result according to each image frame and the intermediate semantic segmentation network.

[0153] The intermediate depth estimation network here is a network generated by training based on each image frame in the training set and the initial depth estimation network. Specifically, each image frame in the training set is input into the intermediate depth estimation network for training, and a depth estimation result can be generated, for example, the depth information of each image frame.

[0154] The intermediate semantic segmentation network here is a network generated by training the initial semantic segmentation network based on each image frame in the training set. The architecture of the semantic segmentation network includes an encoder-decoder network, and the encoder usually includes a pre-trained category network, such as VGG, ResNet network, etc. After the encoder, a decoder network is connected. The task of the decoder network is to semantically project the discriminable features (lower resolution) learned by the encoder to the pixel space (higher resolution) to obtain dense categories.

[0155] Specifically, each image frame in the training set is input into the intermediate semantic segmentation network for training, and the semantic segmentation result can be generated. For each image frame in the training set, a semantic category is assigned to each pixel in the image frame to obtain pixelated dense categories, which are the semantic segmentation results of the image frame.

[0156] Step 1140: training the intermediate depth estimation network according to the depth estimation result and the semantic segmentation result to generate a preset depth estimation network.

[0157] After inputting each image frame in the training set into the intermediate depth estimation network for training to generate a depth estimation result, and inputting each image frame in the training set into the intermediate semantic segmentation network for training to generate a semantic segmentation result, the intermediate depth estimation network can be trained based on the depth estimation result and semantic segmentation result of each image frame to generate a preset depth estimation network. Compared with the traditional method, the depth estimation network is trained only based on the depth estimation result, that is, each pixel is treated equally, and the semantic segmentation information of each pixel cannot be considered outside the depth information. Among them, pixels with different semantic segmentation information can have different depth features. For example, for pixels whose semantic segmentation information is sky, their depth value is greater than that of pixels whose semantic segmentation information is human. Therefore, the intermediate depth estimation network can be trained based on the different semantic segmentation information and depth estimation information of each pixel in the image frame to generate a preset depth estimation network.

[0158] In this application, because the semantic segmentation results can more accurately distinguish the edge and the interior of an object, the depth estimation results of the edge and the interior of an object in each image frame can be adjusted based on the semantic segmentation results of the image frame. Finally, based on the adjusted depth estimation results, the intermediate depth estimation network can be trained to generate a preset depth estimation network.

[0159] By presetting the depth estimation network, the depth information at the edge and inside the object can be better estimated to generate a more accurate depth map.

[0160] In an embodiment of the present application, each image frame in the training set is input into the intermediate depth estimation network for training, and a depth estimation result can be generated. And each image frame in the training set is input into the intermediate semantic segmentation network for training, and a semantic segmentation result can be generated. Since pixels with different semantic segmentation information can have different depth features, the intermediate depth estimation network can be trained based on the depth estimation result and semantic segmentation result of each image frame to generate a preset depth estimation network.

[0161] Therefore, by presetting the depth estimation network, the depth information at the edge and inside the object can be better estimated to generate a more accurate depth map.

[0162] In a specific embodiment, Fig.12 As shown, a depth estimation method is provided, comprising:

[0163] Step 1202, inputting the current frame in the training set into the initial depth estimation network for processing to generate a depth prediction result;

[0164] Step 1204, inputting adjacent frames of the current frame in the training set into a pose estimation network to generate pose transformation information;

[0165] Step 1206, training the initial depth estimation network according to the depth prediction result and the pose transformation information to generate the intermediate depth estimation network;

[0166] Step 1208: Obtain a depth estimation result of the current frame through the intermediate depth estimation network.

[0167] Step 1210, inputting the current frame in the training set into the initial semantic segmentation network to obtain a semantic segmentation prediction result;

[0168] Step 1212, training the initial semantic segmentation network according to the semantic segmentation prediction result and the semantic annotation data set to generate the intermediate semantic segmentation network;

[0169] Step 1214: Obtain the semantic segmentation result of the current frame through the intermediate semantic segmentation network.

[0170] Step 1216, performing an expansion operation on the foreground segmentation result in the semantic segmentation result to obtain an outer edge, and performing an erosion operation on the foreground segmentation result to obtain an inner edge; determining an edge region of the foreground according to the outer edge and the inner edge;

[0171] Step 1218, taking the area within the edge area in the foreground segmentation result as the internal area of ​​the foreground;

[0172] Step 1220: for each first pixel point on the outer edge, determine a second pixel point on the inner edge that is closest to the first pixel point, and obtain a first point pair set based on the first pixel point and the corresponding second pixel point;

[0173] Step 1222, in the internal area, determine a third pixel point and a fourth pixel point corresponding to the third pixel point, and obtain a second point pair set based on the third pixel point and the fourth pixel point; the distance between the third pixel point and the fourth pixel point in the first direction and the distance between the third pixel point and the fourth pixel point in the second direction are both less than a preset threshold; the first direction is perpendicular to the second direction;

[0174] Step 1224, obtaining, from the depth estimation results, a first depth estimation result of each point pair in the first point pair set and a second depth estimation result of each point pair in the second point pair set;

[0175] Step 1226, calculating a value of a first semantic loss function according to the first depth estimation result and the first annotated depth result of each first point pair in the first point pair set; the first semantic loss function is used to penalize the first point pairs whose difference between the first depth estimation results is less than a preset threshold;

[0176] Step 1228, calculating a value of a second semantic loss function according to the second depth estimation result and the second annotated depth result of each second point pair in the second point pair set; the second semantic loss function is used to penalize the second point pairs whose difference between the second depth estimation results is greater than a preset threshold;

[0177] Step 1230, adjusting the parameters of the intermediate depth estimation network according to the value of the first semantic loss function and the value of the second semantic loss function to generate the preset depth estimation network;

[0178] Step 1232, obtaining an image to be processed;

[0179] Step 1234, inputting the image to be processed into a preset depth estimation network to perform depth estimation to obtain depth information of the image to be processed;

[0180] Step 1236 , normalize the range of the depth information to a predetermined depth range by scaling.

[0181] In an embodiment of the present application, each image frame in the training set is input into the intermediate depth estimation network for training, and a depth estimation result can be generated. And each image frame in the training set is input into the intermediate semantic segmentation network for training, and a semantic segmentation result can be generated. Since pixels with different semantic segmentation information can have different depth features, the intermediate depth estimation network can be trained based on the depth estimation result and semantic segmentation result of each image frame to generate a preset depth estimation network.

[0182] Therefore, by presetting the depth estimation network, the depth information at the edge and inside the object can be better estimated to generate a more accurate depth map.

[0183] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0184] In one embodiment, Fig.13 As shown, a depth estimation device 1300 is provided, comprising:

[0185] The to-be-processed image acquisition module 1320 is used to acquire the to-be-processed image;

[0186] The depth estimation module 1340 is used to input the image to be processed into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed; the preset depth estimation network is a network generated by training based on the semantic segmentation results and depth estimation results of each image frame in the training set.

[0187] In one embodiment, Fig.13 As shown, a depth estimation device 1300 is provided, which also includes: a preset depth estimation network training module 1360; wherein, as Fig.14 As shown, the preset depth estimation network training module 1360 includes a depth estimation and semantic segmentation unit 1362 and a preset depth estimation network generation unit 1364; wherein,

[0188] A depth estimation and semantic segmentation unit 1362, configured to obtain a depth estimation result based on each image frame in a training set and an intermediate depth estimation network, and to obtain a semantic segmentation result based on each image frame and an intermediate semantic segmentation network;

[0189] The preset depth estimation network generating unit 1364 is used to train the intermediate depth estimation network according to the depth estimation result and the semantic segmentation result to generate a preset depth estimation network.

[0190] In one embodiment, the preset depth estimation network generation unit 1364 is further used to calculate the value of the loss function according to the depth estimation result and the semantic segmentation result; and train the intermediate depth estimation network based on the value of the loss function to generate the preset depth estimation network.

[0191] In one embodiment, the preset depth estimation network generation unit 1364 includes:

[0192] A region determination subunit, used to determine the edge region and the internal region of the foreground in the semantic segmentation result from the semantic segmentation result;

[0193] A first and second depth estimation result determination subunit, configured to obtain a first depth estimation result of the edge area and a second depth estimation result of the internal area from the depth estimation results according to the edge area and the internal area;

[0194] The preset depth estimation network generation subunit is used to train the intermediate depth estimation network based on the first depth estimation result and the second depth estimation result to generate a preset depth estimation network.

[0195] In one embodiment, the region determination subunit is also used to perform an expansion operation on the foreground segmentation result in the semantic segmentation result to obtain an outer edge, and to perform an erosion operation on the foreground segmentation result to obtain an inner edge; determine the edge region of the foreground based on the outer edge and the inner edge; and use the region in the foreground segmentation result that is within the edge region as the inner region of the foreground.

[0196] In one embodiment, the first and second depth estimation result determination subunits are also used to determine, for the m first pixel points on the outer edge, the m second pixel points that are closest to the m first pixel points on the inner edge, and obtain a first point pair set based on the m first pixel points and the corresponding m second pixel points, where m is a positive integer; in the internal area, n third pixel points and n fourth pixel points corresponding to the n third pixel points are determined, and a second point pair set is obtained based on the n third pixel points and the n fourth pixel points, where n is a positive integer; the positional relationship between the third pixel point and the fourth pixel point satisfies a preset condition; and from the depth estimation results, obtain the first depth estimation results for each point pair in the first point pair set and the second depth estimation results for each point pair in the second point pair set.

[0197] In one embodiment, the positional relationship between the third pixel point and the fourth pixel point satisfies preset conditions, including: the distance between the third pixel point and the fourth pixel point in the first direction is less than a first preset distance threshold, and the distance between the third pixel point and the fourth pixel point in the second direction is less than a second preset distance threshold; the first direction and the second direction are perpendicular to each other.

[0198] In one embodiment, the preset depth estimation network generation subunit is also used to calculate the value of a first semantic loss function based on the first depth estimation results of each first point pair in the first point pair set and the first labeled depth result; the first semantic loss function is used to penalize the first point pairs whose difference between the first depth estimation results is less than the first preset depth difference threshold; the value of the second semantic loss function is calculated based on the second depth estimation results of each second point pair in the second point pair set and the second labeled depth result; the second semantic loss function is used to penalize the second point pairs whose difference between the second depth estimation results is greater than the second preset depth difference threshold; the parameters of the intermediate depth estimation network are adjusted according to the value of the first semantic loss function and the value of the second semantic loss function to generate a preset depth estimation network.

[0199] In one embodiment, the preset depth estimation network generation subunit is also used to construct, for each first point pair, a first semantic loss function based on the difference between a first depth estimation result of a first pixel point in the first point pair and a first depth estimation result of a second pixel point in the first point pair; and for each second point pair, a second semantic loss function is constructed based on the difference between a second depth estimation result of a third pixel point in the second point pair and a second depth estimation result of a fourth pixel point in the second point pair.

[0200] In one embodiment, the depth estimation and semantic segmentation unit includes:

[0201] A depth prediction result generation subunit is used to input each current frame in the training set into the initial depth estimation network for processing to generate a depth prediction result;

[0202] A pose transformation information generation subunit, used for inputting adjacent frames of each current frame in the training set into the pose estimation network to generate pose transformation information;

[0203] An intermediate depth estimation network generation subunit is used to train the initial depth estimation network according to the depth prediction result and the pose transformation information to generate an intermediate depth estimation network;

[0204] The depth estimation result acquisition subunit of the current frame is used to obtain the depth estimation result of the current frame through the intermediate depth estimation network.

[0205] In one embodiment, the intermediate depth estimation network generation subunit is also used to generate a synthetic image based on the depth prediction results and posture transformation information; calculate the value of the projection loss function based on the synthetic image and the current frame; and train the initial depth estimation network through the value of the projection loss function to generate an intermediate depth estimation network.

[0206] In one embodiment, the depth estimation and semantic segmentation unit further includes:

[0207] The semantic segmentation prediction result generating subunit is used to input each current frame in the training set into the initial semantic segmentation network to obtain the semantic segmentation prediction result;

[0208] An intermediate semantic segmentation network generation subunit is used to train the initial semantic segmentation network according to the semantic segmentation prediction results and the semantic annotation data set to generate an intermediate semantic segmentation network;

[0209] The semantic segmentation result generation subunit of the current frame is used to obtain the semantic segmentation result of the current frame through the intermediate semantic segmentation network.

[0210] In one embodiment, the intermediate semantic segmentation network generation subunit is also used to construct a cross entropy loss function based on the semantic segmentation prediction result and the semantic segmentation data set; the initial semantic segmentation network is trained by the cross entropy loss function to generate an intermediate semantic segmentation network.

[0211] In one embodiment, a feature extraction network is included before the initial depth estimation network and the initial semantic segmentation network; the initial depth estimation network includes a first upsampling network, and the initial semantic segmentation network includes a second upsampling network.

[0212] In one embodiment, the initial depth estimation network includes a first feature extraction network and a first upsampling network, and the initial semantic segmentation network includes a second feature extraction network and a second upsampling network.

[0213] In one embodiment, a training device for a depth estimation network is provided, comprising:

[0214] A depth estimation and semantic segmentation module, used to obtain a depth estimation result based on each image frame in the training set and the intermediate depth estimation network, and to obtain a semantic segmentation result based on the image frame and the intermediate semantic segmentation network;

[0215] The preset depth estimation network generation module is used to train the intermediate depth estimation network according to the depth estimation results and semantic segmentation results to generate a preset depth estimation network.

[0216] The division of the various modules in the above-mentioned depth estimation device is only for illustration. In other embodiments, the depth estimation device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned depth estimation device.

[0217] For the specific definition of the depth estimation device, please refer to the definition of the depth estimation method above, which will not be repeated here. Each module in the above-mentioned depth estimation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0218] Fig.15 Schematic diagram of the internal structure of an electronic device in an embodiment. The electronic device may be any terminal device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a wearable device, etc. The electronic device includes a processor and a memory connected via a system bus. Among them, the processor may include one or more processing units. The processor may be a CPU (Central Processing Unit) or a DSP (Digital Signal Processing), etc. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by a processor to implement a depth estimation method provided in each of the following embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium.

[0219] The implementation of each module in the depth estimation device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on an electronic device. The program module constituted by the computer program can be stored in the memory of the electronic device. When the computer program is executed by the processor, the steps of the method described in the embodiment of the present application are implemented.

[0220] The embodiment of the present application further provides a computer-readable storage medium, which is one or more non-volatile computer-readable storage media containing computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the depth estimation method.

[0221] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a depth estimation method.

[0222] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include ROM (Read-Only Memory), PROM (Programmable Read-only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-only Memory) or flash memory. Volatile memory may include RAM (Random Access Memory), which is used as an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDRSDRAM (Double Data Rate Synchronous Dynamic Random Access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory), SLDRAM (Sync Link Dynamic Random Access Memory), RDRAM (Rambus Dynamic Random Access Memory), and DRDRAM (Direct Rambus Dynamic Random Access Memory).

[0223] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A depth estimation method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a preset depth estimation network for depth estimation to obtain depth information of the image to be processed; The preset depth estimation network obtains depth estimation results based on each image frame and the intermediate depth estimation network in the training set, and obtains semantic segmentation results based on each image frame and the intermediate semantic segmentation network. From the semantic segmentation results, the edge area and the internal area of ​​the foreground in the semantic segmentation results are determined, and a first depth estimation result of the edge area and a second depth estimation result of the internal area are obtained from the depth estimation results based on the edge area and the internal area. Based on the first depth estimation result and the second depth estimation result, the intermediate depth estimation network is trained to generate a network.

2. The method according to claim 1, characterized in that The step of determining the edge region and the internal region of the foreground in the semantic segmentation result from the semantic segmentation result includes: Performing an expansion operation on a foreground segmentation result in the semantic segmentation result to obtain an outer edge, and performing an erosion operation on the foreground segmentation result to obtain an inner edge; Determine an edge area of ​​the foreground according to the outer edge and the inner edge; The area within the edge area in the foreground segmentation result is used as the internal area of ​​the foreground.

3. The method according to claim 2, characterized in that The obtaining, from the depth estimation results according to the edge area and the internal area, a first depth estimation result of the edge area and a second depth estimation result of the internal area, comprises: For the m first pixel points on the outer edge, determine m second pixel points on the inner edge that are closest to the m first pixel points, and obtain a first point pair set based on the m first pixel points and the corresponding m second pixel points, where m is a positive integer; In the internal area, n third pixel points and n fourth pixel points corresponding to the n third pixel points are determined, and a second point pair set is obtained based on the n third pixel points and the n fourth pixel points, where n is a positive integer; and a positional relationship between the third pixel point and the fourth pixel point satisfies a preset condition; A first depth estimation result for each point pair in the first point pair set and a second depth estimation result for each point pair in the second point pair set are obtained from the depth estimation results.

4. The method according to claim 3, characterized in that The positional relationship between the third pixel point and the fourth pixel point satisfies a preset condition, including: The distance between the third pixel point and the fourth pixel point in the first direction is less than a first preset distance threshold, and the distance between the third pixel point and the fourth pixel point in the second direction is less than a second preset distance threshold; the first direction and the second direction are perpendicular to each other.

5. The method according to claim 3, characterized in that: The training of the intermediate depth estimation network based on the first depth estimation result and the second depth estimation result includes: Calculating a value of a first semantic loss function according to the first depth estimation results and the first annotated depth results of each first point pair in the first point pair set; the first semantic loss function is used to penalize the first point pairs whose difference between the first depth estimation results is less than a first preset depth difference threshold; Calculating a value of a second semantic loss function according to the second depth estimation results and the second annotated depth results of each second point pair in the second point pair set; the second semantic loss function is used to penalize the second point pairs whose difference between the second depth estimation results is greater than a second preset depth difference threshold; The parameters of the intermediate depth estimation network are adjusted according to the value of the first semantic loss function and the value of the second semantic loss function to generate the preset depth estimation network.

6. The method according to claim 5, characterized in that The method further comprises: For each of the first point pairs, constructing the first semantic loss function based on a difference between a first depth estimation result of the first pixel point in the first point pair and a first depth estimation result of the second pixel point in the first point pair; For each of the second point pairs, the second semantic loss function is constructed based on a difference between a second depth estimation result of the third pixel point in the second point pair and a second depth estimation result of the fourth pixel point in the second point pair.

7. The method according to claim 1, characterized in that The depth estimation result is obtained according to each image frame and the intermediate depth estimation network in the training set, including: Input each current frame in the training set into the initial depth estimation network for processing to generate a depth prediction result; Inputting adjacent frames of each current frame in the training set into a pose estimation network to generate pose transformation information; Training the initial depth estimation network according to the depth prediction result and the pose transformation information to generate the intermediate depth estimation network; The depth estimation result of the current frame is obtained through the intermediate depth estimation network.

8. The method according to claim 7, characterized in that The training of the initial depth estimation network according to the depth prediction result and the pose transformation information to generate the intermediate depth estimation network includes: Generate a synthetic image according to the depth prediction result and the posture transformation information; Calculating a value of a projection loss function according to the synthesized image and the current frame; The initial depth estimation network is trained using the value of the projection loss function to generate the intermediate depth estimation network.

9. The method according to claim 1, characterized in that: The obtaining the semantic segmentation result according to each of the image frames and the intermediate semantic segmentation network includes: Inputting each current frame in the training set into the initial semantic segmentation network to obtain a semantic segmentation prediction result; According to the semantic segmentation prediction result and the semantic annotation data set, the initial semantic segmentation network is trained to generate the intermediate semantic segmentation network; The semantic segmentation result of the current frame is obtained through the intermediate semantic segmentation network.

10. The method according to claim 9, characterized in that The step of training the initial semantic segmentation network according to the semantic segmentation prediction result and the semantic segmentation data set to generate an intermediate semantic segmentation network includes: Constructing a cross entropy loss function based on the semantic segmentation prediction result and the semantic segmentation dataset; The initial semantic segmentation network is trained by the cross entropy loss function to generate an intermediate semantic segmentation network.

11. The method according to claim 9, characterized in that A feature extraction network is included before the initial depth estimation network and the initial semantic segmentation network; the initial depth estimation network includes a first upsampling network, and the initial semantic segmentation network includes a second upsampling network.

12. The method according to claim 9, characterized in that The initial depth estimation network includes a first feature extraction network and a first upsampling network, and the initial semantic segmentation network includes a second feature extraction network and a second upsampling network.

13. A method for training a depth estimation network, characterized in that: include: Obtaining a depth estimation result according to each image frame in the training set and the intermediate depth estimation network, and obtaining a semantic segmentation result according to each of the image frames and the intermediate semantic segmentation network; From the semantic segmentation result, determine the edge area and the internal area of ​​the foreground in the semantic segmentation result, obtain a first depth estimation result of the edge area and a second depth estimation result of the internal area from the depth estimation result according to the edge area and the internal area, and train the intermediate depth estimation network based on the first depth estimation result and the second depth estimation result to generate a preset depth estimation network.

14. A depth estimation device, characterized in that: include: The module for acquiring the image to be processed is used to acquire the image to be processed; A depth estimation module is used to input the image to be processed into a preset depth estimation network to perform depth estimation and obtain depth information of the image to be processed; The preset depth estimation network obtains depth estimation results based on each image frame and the intermediate depth estimation network in the training set, and obtains semantic segmentation results based on each image frame and the intermediate semantic segmentation network. From the semantic segmentation results, the edge area and the internal area of ​​the foreground in the semantic segmentation results are determined, and a first depth estimation result of the edge area and a second depth estimation result of the internal area are obtained from the depth estimation results based on the edge area and the internal area. Based on the first depth estimation result and the second depth estimation result, the intermediate depth estimation network is trained to generate a network.

15. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor executes the steps of the depth estimation method according to any one of claims 1 to 12 and the training method of the depth estimation network according to claim 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the depth estimation method according to any one of claims 1 to 12 and the training method of the depth estimation network according to claim 13 are implemented.

Images