Depth map optimization method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a depth map optimization method and device, electronic equipment and storage medium. The method comprises: obtaining a target image and a to-be-optimized depth map corresponding to the target image; using a trained target neural network for depth map optimization to perform feature extraction on the to-be-optimized depth map and the target image to generate a noise map; wherein the noise map is used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map; using the target neural network and the noise map to perform optimization processing on the to-be-optimized depth map to obtain a target depth map.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of deep learning, in particular, to a depth map optimization method and device, electronic equipment and storage medium. BACKGROUND

[0002] A depth image can be referred to as a distance image, which refers to an image taking the distance (depth) from an image collector to each point in a scene as a pixel value, wherein the depth map plays a crucial role in virtual reality and scene reconstruction and other application scenarios.

[0003] Generally, a depth camera can be used to obtain a depth image, or a three-dimensional vision calculation method can also be used to obtain a depth image. However, due to the influence of factors such as texture, reflection conditions, and object material, the accuracy of the obtained depth image is low. SUMMARY

[0004] Therefore, the present disclosure provides at least a depth map optimization method and device, electronic equipment and storage medium.

[0005] In a first aspect, the present disclosure provides a depth map optimization method, comprising:

[0006] obtaining a target image and a to-be-optimized depth map corresponding to the target image;

[0007] performing feature extraction on the to-be-optimized depth map and the target image by using a target neural network trained for depth map optimization to generate a noise map; wherein the noise map is used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map;

[0008] performing optimization processing on the to-be-optimized depth map by using the target neural network and the noise map to obtain a target depth map.

[0009] In the above method, by using the target neural network, feature extraction is performed on the to-be-optimized depth map and the target image to generate a noise map, which can be used to represent noise information and / or depth missing information contained in the to-be-optimized depth map, and then the target neural network and the noise map can be used to perform optimization processing on the to-be-optimized depth map, such as eliminating noise information in the to-be-optimized depth map and / or supplementing missing depth information in the to-be-optimized depth map, to obtain a target depth map with higher accuracy.

[0010] In a possible implementation, the performing optimization processing on the to-be-optimized depth map by using the target neural network and the noise map to obtain a target depth map comprises:

[0011] extract features from the noise map and the to-be-optimized depth map by using the target neural network to generate a target feature map; wherein the target feature map is used to eliminate noise information in the to-be-optimized depth map and / or is used to supplement depth information in the to-be-optimized depth map;

[0012] perform optimization processing on the to-be-optimized depth map by using the target feature map to generate the target depth map.

[0013] By generating the target feature map, the noise information in the to-be-optimized depth map can be eliminated and / or the depth information in the to-be-optimized depth map can be supplemented, and then the target feature map is used to perform optimization processing on the to-be-optimized depth map to generate the target depth map with high precision.

[0014] In a possible implementation, the step of extracting features from the noise map and the to-be-optimized depth map by using the target neural network to generate a target feature map includes:

[0015] extract features from the noise map and the to-be-optimized depth map by using the target neural network to generate a plurality of first feature maps with different sizes;

[0016] sample features from the to-be-processed feature map to generate a second feature map, wherein the feature sampling includes convolution processing and / or feature up-sampling;

[0017] generate a third feature map based on the second feature map and a first feature map with a size consistent with that of the second feature map;

[0018] return to the step of sampling features from the to-be-processed feature map to generate a second feature map by taking the third feature map as an updated to-be-processed feature map until the size of the third feature map is consistent with a maximum size corresponding to the first feature map, and determine the third feature map with the maximum size as the target feature map.

[0019] Here, after generating a plurality of first feature maps with different sizes, the to-be-processed feature map can be sampled to generate a second feature map, and a third feature map can be generated based on the second feature map and a first feature map with a size consistent with that of the second feature map. Since the first feature map with the size consistent with that of the second feature map has rich feature information, the information loss caused by the down-sampling process is alleviated, so that the information in the third feature map is rich, and the information in the generated target feature map is also rich, so that a target depth map with high precision can be obtained based on the target feature map with rich information.

[0020] In a possible implementation, the target neural network is trained according to the following steps:

[0021] obtaining training samples, wherein the training samples comprise sample images, first depth maps corresponding to the sample images, and second depth maps corresponding to the sample images, the first depth maps having higher accuracy than the second depth maps;

[0022] training a neural network to be trained using the training samples to obtain the target neural network.

[0023] In a possible implementation, in a case where the training samples further comprise sample normal maps and sample semantic maps corresponding to the sample images, and the target neural network comprises a multi-task network and a depth optimization network, the training the neural network to be trained using the training samples to obtain the target neural network comprises:

[0024] inputting the second depth maps and the sample images into the multi-task network to obtain predicted noise maps, predicted semantic maps, and predicted normal maps corresponding to the sample images;

[0025] inputting the predicted noise maps and the second depth maps into the depth optimization network to generate the predicted depth maps;

[0026] training the neural network to be trained based on at least one of combinations of the predicted semantic maps and the sample semantic maps, the predicted normal maps and the sample normal maps, the predicted depth maps and the first depth maps, and the predicted depth maps and the second depth maps to obtain the target neural network.

[0027] Here, the training samples comprise sample semantic maps and sample normal maps, so that the target neural network trained based on the sample semantic maps and the sample normal maps can accurately extract semantic information and normal information in images. Since there is a certain correlation between depth information and semantic information and normal information, the target neural network trained based on the semantic information and the normal information can generate a more accurate target depth map.

[0028] In a possible implementation, the training the neural network to be trained based on at least one of combinations of the predicted semantic maps and the sample semantic maps, the predicted normal maps and the sample normal maps, the predicted depth maps and the first depth maps, and the predicted depth maps and the second depth maps to obtain the target neural network comprises:

[0029] determine, based on the predicted depth map and the first depth map, a first loss value for characterizing a depth information bias, and a second loss value for characterizing a depth gradient bias; determine, based on the predicted semantic map and the sample semantic map, a third loss value for characterizing a semantic information bias; determine, based on the predicted normal map and the sample normal map, a fourth loss value for characterizing a normal information bias; and determine, based on a second depth map and the predicted depth map, a fifth loss value for characterizing a depth distortion information;

[0030] determine, based on one or more of the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value, a target loss value;

[0031] train the neural network to be trained based on the target loss value to obtain the target neural network.

[0032] Here, multiple loss values are set, and one or more loss values can be used to determine a target loss value. When the neural network is trained using the target loss value, the accuracy of the obtained target neural network can be higher.

[0033] In one possible implementation, when the target loss value includes the fourth loss value, the determination of the fourth loss value for characterizing the normal information bias based on the predicted normal map and the sample normal map includes:

[0034] determining, based on the predicted normal map and the sample normal map, a cosine similarity between first pixel information of each first pixel point in the predicted normal map and second pixel information of a second pixel point in the sample normal map that matches the first pixel point;

[0035] determining the fourth loss value based on the cosine similarity corresponding to each first pixel point and the number of first pixel points.

[0036] In one possible implementation, when the target loss value includes the second loss value, the determination of the second loss value for characterizing the depth gradient bias based on the predicted depth map and the first depth map includes:

[0037] determining a first gradient of each pixel point in the predicted depth map in a horizontal axis direction and a second gradient of the pixel point in a vertical axis direction; and

[0038] determining a third gradient of each pixel point in the first depth map in the horizontal axis direction and a fourth gradient of the pixel point in the vertical axis direction;

[0039] determine the second loss value based on the first gradient and the second gradient corresponding to each third pixel in the predicted depth map, and the third gradient and the fourth gradient corresponding to a fourth pixel matched with the third pixel in the first depth map.

[0040] In a possible implementation, when the target loss value includes a fifth loss value, determining the fifth loss value for representing depth distortion information based on the second depth map and the predicted depth map includes:

[0041] perform feature extraction on the predicted depth map and the second depth map respectively by using the trained feature extraction network to generate a first target feature map corresponding to the predicted depth map and a second target feature map corresponding to the second depth map;

[0042] determine the fifth loss value based on the first target feature map and the second target feature map.

[0043] In a possible implementation, determining the fifth loss value based on the first target feature map and the second target feature map includes:

[0044] determining a difference value between a first feature value of each first feature point in the first target feature map and a second feature value of a second feature point matched with the first feature point in the second target feature map;

[0045] determining a sum of squares of the difference values corresponding to each first feature point as the fifth loss value.

[0046] In a possible implementation, before inputting the second depth map and the sample image into the multi-task network to obtain a predicted noise map, a predicted semantic map, and a predicted normal map corresponding to the sample image, the method further includes:

[0047] performing an alignment operation on the sample image and the second depth map to obtain an aligned sample image and an aligned second depth map, wherein pixel points with the same pixel positions in the aligned sample image and the aligned second depth map correspond to the same position in a real scene.

[0048] In a possible implementation, when the target image includes a real image collected by a movable device, after obtaining the target depth map, the method further includes:

[0049] determining depth information of at least one first target object included in the target image based on the target depth map;

[0050] determine an occlusion state between the virtual object and the first target object based on depth information of each of the first target objects and a predetermined display pose of the virtual object;

[0051] generate a display image added with the virtual object based on the occlusion state and the target image, and control the movable device to display the display image.

[0052] Here, since the target depth map has high accuracy, the occlusion state between the virtual object and the first target object can be determined more accurately by using the target depth map with high accuracy, and when generating a display image added with the virtual object based on the occlusion state and the target image, the display effect of the generated display image can be more realistic.

[0053] In a possible implementation, in the case where the target image includes at least one frame of scene image, after the target depth map is obtained, the method further includes:

[0054] construct a three-dimensional scene map based on the at least one frame of scene image and the target depth map corresponding to each frame of scene image.

[0055] Here, since the target depth map has high accuracy, the three-dimensional scene map constructed by using the target depth map with high accuracy has high accuracy.

[0056] In a possible implementation, in the case where the target image includes a road image collected by an image collection device arranged on a driving device, after the target depth map is obtained, the method further includes:

[0057] determine depth information of at least one second target object included in the target image based on the target depth map;

[0058] control the driving device based on the depth information of each of the second target objects.

[0059] Here, since the target depth map has high accuracy, the depth information of the second target object can be determined more accurately by using the target depth map with high accuracy, and the driving device can be controlled more accurately based on the depth information of each of the second target objects, thereby improving the safety performance of the driving device.

[0060] The effects of the following devices, electronic devices, and the like are described in the above method, which will not be repeated here.

[0061] In a second aspect, the present disclosure provides a depth map optimization device, including:

[0062] an acquisition module configured to acquire a target image and a to-be-optimized depth map corresponding to the target image;

[0063] extracting a feature of the to-be-optimized depth map and the target image by using the trained target neural network for depth map optimization, to generate a noise map; wherein the noise map is used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map;

[0064] optimizing the to-be-optimized depth map by using the target neural network and the noise map, to obtain a target depth map.

[0065] In a possible implementation, when the optimization module optimizes the to-be-optimized depth map by using the target neural network and the noise map, to obtain a target depth map, the optimization module is configured to:

[0066] extracting a feature of the noise map and the to-be-optimized depth map by using the target neural network, to generate a target feature map; wherein the target feature map is used to eliminate noise information in the to-be-optimized depth map, and / or is used to supplement depth information in the to-be-optimized depth map;

[0067] optimizing the to-be-optimized depth map by using the target feature map, to generate the target depth map.

[0068] In a possible implementation, when the optimization module extracts a feature of the noise map and the to-be-optimized depth map by using the target neural network, to generate a target feature map, the optimization module is configured to:

[0069] extracting a feature of the noise map and the to-be-optimized depth map by using the target neural network, to generate a plurality of first feature maps with different sizes;

[0070] taking a first feature map with a minimum size as a to-be-processed feature map, and sampling a feature of the to-be-processed feature map, to generate a second feature map; wherein the feature sampling includes convolution processing and / or feature up-sampling;

[0071] generating a third feature map based on the second feature map and a first feature map with a size consistent with that of the second feature map;

[0072] taking the third feature map as an updated to-be-processed feature map, and returning to the step of sampling a feature of the to-be-processed feature map, to generate a second feature map, until the size of the generated third feature map is consistent with a maximum size corresponding to the first feature map, and the third feature map corresponding to the maximum size is determined as the target feature map.

[0073] In a possible implementation, the apparatus further includes a training module configured to train the target neural network according to the following steps:

[0074] obtain training samples, wherein the training samples include sample images, first depth maps corresponding to the sample images, and second depth maps corresponding to the sample images, and the first depth maps have higher accuracy than the second depth maps;

[0075] train a neural network to be trained using the training samples to obtain the target neural network.

[0076] In a possible implementation, in a case where the training samples further include sample normal maps and sample semantic maps corresponding to the sample images, and the target neural network includes a multi-task network and a depth optimization network, the training module, when training the neural network to be trained using the training samples to obtain the target neural network, is configured to:

[0077] input the second depth maps and the sample images into the multi-task network to obtain predicted noise maps, predicted semantic maps, and predicted normal maps corresponding to the sample images;

[0078] input the predicted noise maps and the second depth maps into the depth optimization network to generate the predicted depth maps;

[0079] train the neural network to be trained based on at least one of combinations of the predicted semantic maps and the sample semantic maps, the predicted normal maps and the sample normal maps, the predicted depth maps and the first depth maps, and the predicted depth maps and the second depth maps, to obtain the target neural network.

[0080] In a possible implementation, when the training module trains the neural network to be trained based on at least one of combinations of the predicted semantic maps and the sample semantic maps, the predicted normal maps and the sample normal maps, the predicted depth maps and the first depth maps, and the predicted depth maps and the second depth maps, to obtain the target neural network, the training module is configured to:

[0081] determine, based on the predicted depth maps and the first depth maps, a first loss value for representing a depth information deviation and a second loss value for representing a depth gradient deviation; determine, based on the predicted semantic maps and the sample semantic maps, a third loss value for representing a semantic information deviation; determine, based on the predicted normal maps and the sample normal maps, a fourth loss value for representing a normal information deviation; and determine, based on the second depth maps and the predicted depth maps, a fifth loss value for representing a depth distortion information;

[0082] determine a target loss value based on one or more of the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value;

[0083] train the neural network to be trained based on the target loss value, to obtain the target neural network.

[0084] In a possible implementation, when the target loss value comprises a fourth loss value, the training module, when determining the fourth loss value for representing the normal information bias based on the predicted normal map and the sample normal map, is configured to:

[0085] determine, based on the predicted normal map and the sample normal map, a cosine similarity between first pixel information of each first pixel point in the predicted normal map and second pixel information of a second pixel point matched with the first pixel point in the sample normal map;

[0086] determine the fourth loss value based on the cosine similarity corresponding to each first pixel point and the number of the first pixel points.

[0087] In a possible implementation, when the target loss value comprises a second loss value, the training module, when determining the second loss value for representing the depth gradient bias based on the predicted depth map and the first depth map, is configured to:

[0088] determine a first gradient of each pixel point in the predicted depth map in a horizontal axis direction and a second gradient of the pixel point in a vertical axis direction; and

[0089] determine a third gradient of each pixel point in the first depth map in the horizontal axis direction and a fourth gradient of the pixel point in the vertical axis direction;

[0090] determine the second loss value based on the first gradient and the second gradient corresponding to each third pixel point in the predicted depth map and the third gradient and the fourth gradient corresponding to a fourth pixel point matched with the third pixel point in the first depth map.

[0091] In a possible implementation, when the target loss value comprises a fifth loss value, the training module, when determining the fifth loss value for representing the depth distortion information based on a second depth map and the predicted depth map, is configured to:

[0092] perform feature extraction on the predicted depth map and the second depth map respectively by using the trained feature extraction network, to generate a first target feature map corresponding to the predicted depth map and a second target feature map corresponding to the second depth map;

[0093] determine the fifth loss value based on the first target feature map and the second target feature map.

[0094] In a possible implementation, the training module is configured to, when determining the fifth loss value based on the first target feature map and the second target feature map, determine a difference between a first feature value of each first feature point in the first target feature map and a second feature value of a second feature point in the second target feature map that matches the first feature point.

[0095] determine a difference between a first feature value of each first feature point in the first target feature map and a second feature value of a second feature point in the second target feature map that matches the first feature point.

[0096] determine a difference between a first feature value of each first feature point in the first target feature map and a second feature value of a second feature point in the second target feature map that matches the first feature point.

[0097] In a possible implementation, before inputting the second depth map and the sample image into the multi-task network to obtain a sample noise map, a predicted semantic map, and a predicted normal map corresponding to the sample image, the apparatus further includes an alignment module configured to:

[0098] perform an alignment operation on the sample image and the second depth map to obtain an aligned sample image and an aligned second depth map, where pixel points at the same pixel position in the aligned sample image and the aligned second depth map correspond to the same position in a real scene.

[0099] In a possible implementation, when the target image includes a real image captured by a movable device, after the target depth map is obtained, the apparatus further includes a first generation module configured to:

[0100] determine depth information of at least one first target object included in the target image based on the target depth map.

[0101] determine an occlusion state between the virtual object and the first target object based on the depth information of each first target object and a predetermined display pose of the virtual object.

[0102] generate a display image in which the virtual object is added based on the occlusion state and the target image, and control the movable device to display the display image.

[0103] In a possible implementation, when the target image includes at least one frame of scene image, after the target depth map is obtained, the apparatus further includes a second generation module configured to:

[0104] construct a three-dimensional scene map based on the at least one frame of scene image and the target depth map corresponding to each frame of scene image.

[0105] In a possible implementation, in a case where the target image comprises a road image captured by an image capturing device arranged on the travel device, after the target depth map is obtained, the device further comprises a third generation module configured to:

[0106] determine depth information of at least one second target object included in the target image based on the target depth map;

[0107] control the travel device based on the depth information of each second target object.

[0108] In a third aspect, the present disclosure provides an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the depth map optimization method according to the first aspect or any one of the embodiments.

[0109] In a fourth aspect, the present disclosure provides a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the steps of the depth map optimization method according to the first aspect or any one of the embodiments are performed.

[0110] In order to make the above objectives, features and advantages of the present disclosure more apparent, clear and easy to understand, the following will specifically describe the preferred embodiments with reference to the accompanying drawings, and the specific description is as follows. BRIEF DESCRIPTION OF DRAWINGS

[0111] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments, the drawings herein are incorporated into the description and form a part of the description, the drawings show the embodiments consistent with the present disclosure, and are used to illustrate the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0112] Figure 1 A flowchart of a depth map optimization method provided by an embodiment of the present disclosure is shown;

[0113] Figure 2 A structure diagram of a target neural network in a depth map optimization method provided by an embodiment of the present disclosure is shown;

[0114] Figure 3 An architecture diagram of a depth map optimization device provided by an embodiment of the present disclosure is shown;

[0115] Figure 4 A structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0116] To make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings for the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure generally described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present disclosure.

[0117] Generally, a depth image can be acquired by using a depth camera, or can also be acquired by using a three-dimensional vision calculation method. However, due to the influence of factors such as texture, reflection condition, object material and the like, the accuracy of the acquired depth image is low. In order to improve the accuracy of the depth image, the embodiments of the present disclosure provide a depth map optimization method and device, an electronic device and a storage medium.

[0118] The defects of the above solutions are the results obtained by the inventors after practice and careful research, and therefore, the discovery process of the above problems and the solutions proposed by the present disclosure to solve the above problems in the following should be the contributions made by the inventors to the present disclosure in the process of the present disclosure.

[0119] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0120] For the convenience of understanding the embodiments of the present disclosure, first, a depth map optimization method disclosed by the embodiments of the present disclosure is introduced in detail. The execution subject of the depth map optimization method provided by the embodiments of the present disclosure is generally a computer device with certain computing power, which includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the depth map optimization method can be realized by a processor calling computer readable instructions stored in a memory.

[0121] Referring to Figure 1 The flowchart of the depth map optimization method provided by the embodiments of the present disclosure is shown, and the method includes S101-S103, wherein:

[0122] S101, obtaining a target image and a to-be-optimized depth map corresponding to the target image;

[0123] S102, using a trained target neural network for depth map optimization to perform feature extraction on the to-be-optimized depth map and the target image, and generate a noise map; wherein the noise map is used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map;

[0124] S103, using the target neural network and the noise map to perform optimization processing on the to-be-optimized depth map, and obtain a target depth map.

[0125] In the above method, by using the target neural network, the to-be-optimized depth map and the target image are subjected to feature extraction, and a noise map is generated. The noise map can be used to represent noise information and / or depth missing information contained in the to-be-optimized depth map, and then the target neural network and the noise map can be used to perform optimization processing on the to-be-optimized depth map, such as eliminating noise information in the to-be-optimized depth map and / or supplementing missing depth information in the to-be-optimized depth map, to obtain a target depth map with high accuracy.

[0126] The following specifically describes S101-S103.

[0127] In S101, the target image can be any frame image in a real scene, wherein the target image can be a color image or a grayscale image. The to-be-optimized depth map can be any frame depth image obtained, such as a depth image corresponding to the target image collected by a depth camera, or a depth image corresponding to the target image obtained by using a stereo matching algorithm.

[0128] Here, the to-be-optimized depth map and the target image are aligned images. For example, the to-be-optimized depth map can be aligned so that the pixels with the same pixel positions in the target image and the to-be-optimized depth map after the alignment correspond to the same position in the real scene.

[0129] In S102 and S103, the target depth network can include a multi-task network and a depth optimization network. The to-be-optimized depth map and the target image can be regarded as a first image with multiple channels. The multi-task network in the target neural network can be used to extract features of the first image to generate a noise map. The noise map can be used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map. Then, the noise map and the to-be-optimized depth map can be regarded as a second image with multiple channels. The depth optimization network in the target neural network can be used to extract features of the second image to obtain a target depth map after optimization processing.

[0130] For example, the multi-task network can include a feature extraction module and a multi-task decoding module. The feature extraction module can be a convolutional neural network with encoding-decoding. The feature extraction module can include an encoding network and a decoding network. The encoding network can be used to extract feature information of the input to-be-optimized depth map and the target image. For example, the encoding network can include a convolutional layer and / or a down-sampling layer. The to-be-optimized depth map and the target image can be extracted by the encoding network to obtain a first intermediate feature map. The convolutional layer and the down-sampling layer can increase the number of channels of the first intermediate feature map and reduce the resolution of the features.

[0131] The decoding network can include a convolutional layer and / or an up-sampling layer. The decoding network can be used to perform convolution extraction and / or up-sampling processing on the obtained first intermediate feature map to obtain a second intermediate feature map with the same resolution as the target image.

[0132] The multi-task decoding module can include three detection branches. The three detection branches can be used to detect a noise map, a semantic map, and a normal map, respectively. Each detection branch can be composed of one or more convolutional layers in series. For example, the convolutional layer in each detection branch can have a step size of 1. A predicted image (for example, the predicted image can be a predicted noise map, a predicted semantic map, or a predicted normal map) with a channel number of 1 and the same resolution as the target image can be obtained. During training of the target neural network, the three detection branches can be retained. During inference of the target neural network, the detection branch corresponding to the semantic map and the detection branch corresponding to the normal map can be removed, and only the detection branch corresponding to the noise map can be retained.

[0133] In an optional implementation, in S103, the to-be-optimized depth map is optimized by using the target neural network and the noise map to obtain a target depth map, which can include the following steps.

[0134] In S1031, the target feature map is generated by using the target neural network to perform feature extraction on the noise map and the to-be-optimized depth map. The target feature map is used to eliminate noise information in the to-be-optimized depth map and / or is used to supplement depth information in the to-be-optimized depth map.

[0135] In S1032, the target depth map is generated by using the target feature map to perform optimization processing on the to-be-optimized depth map.

[0136] By generating the target feature map, the target feature map can eliminate noise information in the to-be-optimized depth map and / or can supplement depth information in the to-be-optimized depth map. Then, the target depth map with high precision is generated by using the target feature map to perform optimization processing on the to-be-optimized depth map.

[0137] In S1031, the target feature map can be generated by using a depth optimization network in the target neural network to perform feature extraction on the noise map and the to-be-optimized depth map.

[0138] In an optional implementation, in S1031, generating the target feature map by using the target neural network to perform feature extraction on the noise map and the to-be-optimized depth map can include steps A1-A4, where:

[0139] In step A1, a plurality of first feature maps with different sizes are generated by using the target neural network to perform feature extraction on the noise map and the to-be-optimized depth map.

[0140] In step A2, a second feature map is generated by performing feature sampling on a to-be-processed feature map, where the to-be-processed feature map is the first feature map with the smallest size. The feature sampling includes convolution processing and / or feature up-sampling.

[0141] In step A3, a third feature map is generated based on the second feature map and the first feature map with the same size as the second feature map.

[0142] In step A4, the third feature map is taken as an updated to-be-processed feature map, and the step of generating the second feature map by performing feature sampling on the to-be-processed feature map is returned to, until the size of the third feature map is consistent with the maximum size corresponding to the first feature map, and the third feature map with the maximum size is determined as the target feature map.

[0143] In implementation, the noise map and the to-be-optimized depth map can be regarded as a second image with multiple channels. The depth optimization network in the target neural network is used to perform feature extraction on the second image to obtain a first feature map with a first size. Then, the first feature map with the first size can be used to perform feature extraction to obtain a first feature map with a second size, and so on. For example, the convolution layer can be used to perform the feature extraction process.

[0144] For example, the plurality of first feature maps of different sizes can include a first feature map of a first size, a first feature map of a second size, and a first feature map of a third size, wherein the first size is greater than the second size, and the second size is greater than the third size.

[0145] For example, the third size of the first feature map (i.e., the smallest size of the first feature map) can be taken as a to-be-processed feature map, and the to-be-processed feature map (i.e., the third size of the first feature map) can be subjected to feature sampling (for example, convolution processing can be performed by using a convolution layer, or feature up-sampling can be performed by using an up-sampling layer), to generate a first processed second feature map, and the size of the first processed second feature map can be the third size. The first processed second feature map and the first feature map of the second size (here, the third size of the first feature map) can be fused to generate a first processed third feature map, and the size of the first processed third feature map can be the third size.

[0146] For example, the first processed second feature map (i.e., the third size of the second feature map) and the first feature map of the third size can be concatenated to generate the first processed third feature map; or the second feature map of the third size and the first feature map of the third size can be concatenated, and the concatenated feature map can be subjected to convolution processing to generate the first processed third feature map; or the difference between the feature values at the same feature positions in the second feature map of the third size and the first feature map of the third size can be determined to obtain a residual feature map, and the residual feature map can be determined as the first processed third feature map.

[0147] The first processed third feature map can be taken as an updated to-be-processed feature map, and the updated to-be-processed feature map can be subjected to feature sampling (for example, the feature sampling can include convolution processing and / or feature up-sampling processing) to generate a second processed second feature map, and the size of the second processed second feature map can be the second size. The second processed second feature map and the first feature map of the second size can be fused to generate a second processed third feature map, and the size of the second processed third feature map can be the second size.

[0148] Further, the third feature map after the second processing can be taken as an updated feature map to be processed, and feature sampling is performed on the updated feature map to be processed to generate a second feature map after the third processing, and at this time, the size of the second feature map after the third processing can be the first size. Further, fusion can be performed based on the second feature map after the third processing and the first feature map of the first size to generate a third feature map after the third processing. The size of the third feature map after the third processing can be the first size. It can be known that the first size is the maximum size corresponding to the first feature map, and the third feature map of the maximum size is determined as the target feature map, that is, the third feature map of the first size is determined as the target feature map.

[0149] For example, after generating a plurality of first feature maps of different sizes, the first feature map of the minimum size (third size) can also be subjected to feature sampling to generate a second feature map of the second size; the second feature map of the second size can be concatenated with the first feature map of the second size, and the concatenated second size feature map can be subjected to feature extraction to generate a second feature map of the first size, and the first size (maximum size) second feature map can be the target feature map.

[0150] Here, after generating a plurality of first feature maps of different sizes, the feature map to be processed can be subjected to feature sampling to generate a second feature map, and based on the second feature map and the first feature map consistent with the size of the second feature map, a third feature map can be generated. Since the first feature map consistent with the size of the second feature map has rich feature information, the information loss caused by the down-sampling process is alleviated, so that the information in the third feature map is rich, and further, the information in the generated target feature map is rich, so that based on the target feature map with rich information, a target depth map with high precision can be obtained.

[0151] In S1032, the target feature map can be fused with the depth map to be optimized to generate a target depth map. Alternatively, at least one feature extraction can be performed on the target feature map, and the feature map after the at least one feature extraction can be fused with the depth map to be optimized to generate a target depth map. For example, the process of feature fusion can be to add the feature values at the same feature position in the target feature map and the depth map to be optimized; or the target feature map and the depth map to be optimized can be concatenated, and the concatenated feature map can be subjected to convolution processing by using a convolution layer.

[0152] Referring to Figure 2A schematic diagram of a target neural network is shown, which can include a multi-task network and a depth optimization network. Among them, the network structure of the depth optimization network in the target neural network can be a residual U-shaped convolutional neural network. That is, the depth optimization network can use convolutional layers and / or down-sampling layers to increase the number of channels of the feature maps, reduce the resolution of the features, and obtain feature maps of different resolutions, denoted as a feature map set {F0, F1, …, Fn}. Then the extracted feature maps can be convoluted and / or up-sampled using convolutional layers and / or up-sampling layers. Before each up-sampling and convolutional processing, the current feature map can be connected with the feature map of the same resolution in the feature map set {F0, F1, …, Fn} to reduce the information loss caused by down-sampling. The depth optimization network can obtain a single-channel picture with the same resolution (size) as the target image, which is used as a residual map (i.e., a target feature map). Finally, the input original depth map (to-be-optimized depth map) is added to the residual map to obtain the optimized target depth map.

[0153] In combination Figure 2 The depth map optimization method is illustrated by example. The to-be-optimized depth map and the target image are input into the multi-task network of the target neural network. The feature extraction module in the multi-task network extracts features from the to-be-optimized depth map and the target image to generate a second intermediate feature map. The second intermediate feature map is input into the first detection branch corresponding to the noise map to obtain the noise map. During the training process of the target neural network, the second detection branch corresponding to the semantic map and the third detection branch corresponding to the normal map can also be included.

[0154] The noise map and the to-be-optimized depth map are then input into the depth optimization network. The depth optimization network can extract features from the noise map and the to-be-optimized depth map to generate a plurality of first feature maps of different sizes, i.e., a first feature map 21 of a first size, a first feature map 22 of a second size, and a first feature map 23 of a third size in the depth optimization network. Figure 2 The third size of the second feature map 24 is generated by further extracting features from the third size of the first feature map 23. The third size of the second feature map 24 is concatenated with the third size of the first feature map 23. The concatenated third size of the feature map is convoluted and up-sampled to generate a second size of the second feature map 25. The second size of the second feature map 25 is concatenated with the second size of the first feature map 22. The concatenated second size of the feature map is convoluted and up-sampled to generate a first size of the second feature map 26. The first size of the second feature map 26 is concatenated with the first size of the first feature map 21. The concatenated first size of the feature map is convoluted and up-sampled to generate a target feature map. Finally, the target feature map is fused with the to-be-optimized depth map to obtain a target depth map.

[0155] In an optional implementation, the target neural network can be trained according to the following steps:

[0156] In step B1, a training sample is obtained, wherein the training sample includes a sample image, a first depth map corresponding to the sample image, and a second depth map corresponding to the sample image, and the accuracy of the first depth map is higher than that of the second depth map.

[0157] In step B2, the training sample is used to train the neural network to be trained to obtain the target neural network.

[0158] The sample image can be a grayscale image or a color image, and the first depth map and the second depth map corresponding to the sample image, wherein the accuracy of the first depth map is higher than that of the second depth map, that is, the second depth map can be regarded as the true value data of the first depth map.

[0159] Here, the sample image can also include a sample normal map and a sample semantic map corresponding to the sample image. The pixel information of each pixel point in the sample normal map represents the normal vector of the pixel point at the same pixel position in the sample image.

[0160] For example, the sample image can be input into the trained semantic segmentation neural network to obtain a sample semantic map corresponding to the sample image.

[0161] In an optional implementation, in the case that the training sample further includes a sample normal map and a sample semantic map corresponding to the sample image, and the target neural network includes a multi-task network and a depth optimization network, step B2 of training the training sample to train the neural network to be trained to obtain the target neural network includes:

[0162] In step B21, the second depth map and the sample image are input into the multi-task network to obtain a predicted noise map, a predicted semantic map, and a predicted normal map corresponding to the sample image.

[0163] In step B22, the predicted noise map and the second depth map are input into the depth optimization network to generate a predicted depth map.

[0164] In step B23, at least one of the predicted semantic map and the sample semantic map, the predicted normal map and the sample normal map, the predicted depth map and the first depth map, and the predicted depth map and the second depth map is combined to train the neural network to be trained to obtain the target neural network.

[0165] Here, the sample semantic map and the sample normal map are included in the training sample, by increasing the sample semantic map and the sample normal map, the target neural network after training can accurately extract the semantic information and the normal information in the image. Since there is a certain correlation between the depth information and the semantic information and the normal information, the target neural network after training can generate a more accurate target depth map based on the semantic information and the normal information.

[0166] The processes of steps B21 and B22 can refer to the aforementioned description of the target neural network, and will not be described in detail here.

[0167] In step B23, the predicted semantic map and the sample semantic map form a map combination, the predicted normal map and the sample normal map form a map combination, the predicted depth map and the first depth map form a map combination, and the predicted depth map and the second depth map form a map combination. Four map combinations are obtained, and the target neural network can be obtained by training the neural network to be trained based on at least one of the four map combinations.

[0168] In an optional implementation, in step B23, the target neural network can be obtained by training the neural network to be trained based on at least one of the predicted semantic map and the sample semantic map, the predicted normal map and the sample normal map, the predicted depth map and the first depth map, and the predicted depth map and the second depth map, which can include:

[0169] In step C1, based on the predicted depth map and the first depth map, a first loss value for representing a depth information deviation and a second loss value for representing a depth gradient deviation are determined; based on the predicted semantic map and the sample semantic map, a third loss value for representing a semantic information deviation is determined; based on the predicted normal map and the sample normal map, a fourth loss value for representing a normal information deviation is determined; and based on the second depth map and the predicted depth map, a fifth loss value for representing a depth distortion information is determined.

[0170] In step C2, the target loss value is determined based on one or more of the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value.

[0171] In step C3, the target neural network is obtained by training the neural network to be trained based on the target loss value.

[0172] Here, multiple loss values are set, and one or more loss values can be used to determine the target loss value. When the neural network is trained using the target loss value, the accuracy of the obtained target neural network can be higher.

[0173] In step C1, the first loss value loss depth :

[0174]

[0175] wherein n is the number of pixel points in the predicted depth map, d pred (i) is the pixel information of the pixel point i in the predicted depth map (i.e., the depth value corresponding to the pixel point i), d gt (i) is the pixel information of the pixel point i in the first depth map. Wherein the pixel position of the pixel point i in the predicted depth map is consistent with the pixel position of the pixel point i in the first depth map.

[0176] In an optional implementation, when the target loss value includes the second loss value, determining the second loss value for representing the depth gradient deviation based on the predicted depth map and the first depth map can include:

[0177] Step D1, determining the first gradient of each pixel point in the predicted depth map in the horizontal axis direction and the second gradient in the vertical axis direction; and

[0178] Step D2, determining the third gradient of each pixel point in the first depth map in the horizontal axis direction and the fourth gradient in the vertical axis direction;

[0179] Step D3, determining the second loss value based on the first gradient and the second gradient corresponding to each third pixel point in the predicted depth map, and the third gradient and the fourth gradient corresponding to the fourth pixel point matched with the third pixel point in the first depth map.

[0180] In implementation, the second loss value loss edge :

[0181]

[0182] wherein, is the first gradient of the pixel point i in the predicted depth map in the horizontal axis direction, is the second gradient of the pixel point i in the predicted depth map in the vertical axis direction; is the third gradient of the pixel point i in the first depth map in the horizontal axis direction; is the fourth gradient of the pixel point i in the first depth map in the vertical axis direction. Exemplarily, the first gradient, the second gradient, the third gradient, and the fourth gradient can be calculated using Sobel operator.

[0183] Exemplarily, the third loss value can be calculated using the cross-entropy of multi-classification. Wherein the third loss value can be used to measure the consistency between the output predicted semantic map and the semantic label true value (i.e., the sample semantic map).

[0184] In an optional implementation, when the target loss value includes the fourth loss value, determining the fourth loss value for representing the normal information bias based on the predicted normal map and the sample normal map can include:

[0185] Step E1, determining, based on the predicted normal map and the sample normal map, a cosine similarity between first pixel information of each first pixel point in the predicted normal map and second pixel information of a second pixel point matched with the first pixel point in the sample normal map;

[0186] Step E2, determining the fourth loss value based on the cosine similarity corresponding to each first pixel point respectively and the number of the first pixel points.

[0187] The fourth loss value can be used to measure the consistency between the predicted normal map output in the normal prediction task and the normal ground truth (the sample normal map).

[0188] The fourth loss value loss can be determined according to the following formula (3) normal :

[0189]

[0190] Wherein, n pred (i) is the first pixel information of the pixel point i (i.e., the first pixel point) in the predicted normal map, and n gt (i) is the second pixel information of the pixel point i (i.e., the second pixel point matched with the first pixel point) in the sample normal map.

[0191] Here, the first pixel point can be each pixel point in the predicted normal map, and the second pixel point can be a pixel point in the sample normal map consistent with the pixel position of the first pixel point.

[0192] In an optional implementation, when the target loss value includes the fifth loss value, determining the fifth loss value for representing the depth distortion information based on the second depth map and the predicted depth map can include:

[0193] Step F1, using the trained feature extraction network to perform feature extraction on the predicted depth map and the second depth map respectively to generate a first target feature map corresponding to the predicted depth map and a second target feature map corresponding to the second depth map;

[0194] Step F2, determining the fifth loss value based on the first target feature map and the second target feature map.

[0195] Wherein, in step F2, determining the fifth loss value based on the first target feature map and the second target feature map can include:

[0196] Step F21: Determine the difference between the first feature value of each first feature point in the first target feature map and the second feature value of the second feature point in the second target feature map that matches the first feature point;

[0197] Step F22: The sum of squares of the differences corresponding to each first feature point is determined as the fifth loss value.

[0198] In practice, the trained feature extraction network can be a VGG network. The trained VGG network is used to extract features from the predicted depth map and the second depth map respectively, generating a first target feature map corresponding to the predicted depth map and a second target feature map corresponding to the second depth map.

[0199] Then, the difference between the first feature value of each first feature point in the first target feature map and the second feature value of the second feature point in the second target feature map that matches the first feature point can be determined. The sum of squares of the differences corresponding to each first feature point (or second feature point) is calculated, and this sum of squares is determined as the fifth loss value. Here, the first feature point can be any feature point in the first target feature map, and the second feature point can be any feature point in the second target feature map whose feature position information is consistent with that of the first feature point.

[0200] In steps C2 and C3, during implementation, any one of the first, second, third, fourth, and fifth loss values ​​can be used as the target loss value; alternatively, multiple loss values ​​can be added together to obtain the target loss value. For example, the sum of the first and second loss values ​​can be used as the target loss value, or the sum of the first, second, third, fourth, and fifth loss values ​​can be used as the target loss value. The target loss value can then be used to train the neural network to obtain the target neural network.

[0201] In one optional implementation, before inputting the second depth map and the sample image into a multi-task network to obtain the prediction noise map, prediction semantic map, and prediction normal map corresponding to the sample image, the method further includes: performing an alignment operation on the sample image and the second depth map to obtain an aligned sample image and a second depth map, wherein pixels with the same pixel position in the aligned sample image and the second depth map correspond to the same position in the real scene.

[0202] For example, the RANSAC algorithm, image registration algorithm, etc., can be used to align the sample image and the second depth map to obtain the aligned sample image and the second depth map. There are various algorithms for the alignment operation; this is only an example.

[0203] In an implementation, in a case where the target image comprises a real image captured by the movable device, after the target depth map is obtained, the method further comprises:

[0204] Step one, determining depth information of at least one first target object included in the target image based on the target depth map;

[0205] Step two, determining an occlusion state between the virtual object and the first target object based on the depth information of each first target object and a display pose of the virtual object;

[0206] Step three, generating a display image with the virtual object added based on the occlusion state and the target image, and controlling the movable device to display the display image.

[0207] In implementation, the target depth map can be applied in an AR scene. After the target depth map is obtained, the depth information of each first target object in the target image can be determined based on the target depth map, and then the occlusion state between the virtual object and the first target object can be determined according to the depth information of the first target object and the display pose of the virtual object, wherein the depth information of the first target object and the display pose of the virtual object are in the same coordinate system.

[0208] Further, the display image with the virtual object added can be generated based on the occlusion state and the target image, and the generated display image can be controlled to be displayed. For example, if the occlusion state is that the first target object occludes the virtual object, the AR special effect in the generated display image is the effect that the first target object occludes the virtual object; if the occlusion state is that the virtual object occludes the first target object, the AR special effect in the generated display image is the effect that the virtual object occludes the first target object.

[0209] Here, since the target depth map has high accuracy, the occlusion state between the virtual object and the first target object can be more accurately determined by using the target depth map with high accuracy, and then the display image with the virtual object added can be generated based on the occlusion state and the target image, so that the display effect of the generated display image is more realistic.

[0210] In an optional implementation, in a case where the target image comprises at least one frame of scene image, after the target depth map is obtained, the method further comprises: constructing a three-dimensional scene map based on the at least one frame of scene image and the target depth map corresponding to each frame of scene image.

[0211] In implementation, a plurality of feature points can be extracted from each frame of scene image, and a depth value of each feature point can be determined according to a target depth map corresponding to the frame of scene image, and then feature information and depth values of the plurality of feature points corresponding to at least one frame of scene image can be obtained; and the feature information and depth values of the plurality of feature points are used to construct a three-dimensional scene map.

[0212] Here, since the target depth map has high accuracy, the three-dimensional scene map constructed by using the target depth map with high accuracy has high accuracy.

[0213] In an optional implementation, when the target image includes a road image captured by an image capturing device arranged on the driving device, after the target depth map is obtained, the method further includes: determining depth information of at least one second target object included in the target image based on the target depth map; and controlling the driving device based on the depth information of each second target object.

[0214] When the target image includes a road image captured by an image capturing device arranged on the driving device, a target depth map corresponding to the road image can be obtained, depth information of at least one second target object included in the target image can be determined based on the target depth map corresponding to the road image, and the driving device can be controlled based on the depth information of each second target object. For example, when the driving device is controlled, the driving device can be controlled to accelerate, decelerate, turn, brake, etc., or voice prompt information can be played to prompt the driver to control the driving device to accelerate, decelerate, turn, brake, etc.

[0215] Here, since the target depth map has high accuracy, the depth information of the second target object can be determined more accurately by using the target depth map with high accuracy, and the driving device can be controlled more accurately based on the depth information of each second target object, thereby improving the safety performance of the driving device.

[0216] Those skilled in the art can understand that the order of writing each step in the above method of the specific embodiment does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0217] Based on the same concept, the embodiment of the disclosure also provides a depth map optimization device, as shown in Figure 3 The architecture schematic diagram of the depth map optimization device provided by the embodiment of the disclosure includes an acquisition module 301, an extraction module 302, and an optimization module 303, and specifically:

[0218] The acquisition module 301 is configured to acquire a target image and a to-be-optimized depth map corresponding to the target image.

[0219] The extraction module 302 is configured to perform feature extraction on the to-be-optimized depth map and the target image by using the trained target neural network for depth map optimization, and generate a noise map; the noise map is used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map.

[0220] The optimization module 303 is configured to perform optimization processing on the to-be-optimized depth map by using the target neural network and the noise map, and obtain a target depth map.

[0221] In a possible implementation, when the optimization module 303 performs optimization processing on the to-be-optimized depth map by using the target neural network and the noise map, and obtains a target depth map, the optimization module 303 is configured to:

[0222] perform feature extraction on the noise map and the to-be-optimized depth map by using the target neural network, and generate a target feature map; the target feature map is used to eliminate noise information in the to-be-optimized depth map, and / or is used to supplement depth information in the to-be-optimized depth map;

[0223] perform optimization processing on the to-be-optimized depth map by using the target feature map, and generate the target depth map.

[0224] In a possible implementation, when the optimization module 303 performs feature extraction on the noise map and the to-be-optimized depth map by using the target neural network, and generates a target feature map, the optimization module 303 is configured to:

[0225] perform feature extraction on the noise map and the to-be-optimized depth map by using the target neural network, and generate a plurality of first feature maps with different sizes;

[0226] take a first feature map with a minimum size as a to-be-processed feature map, perform feature sampling on the to-be-processed feature map, and generate a second feature map; the feature sampling includes convolution processing and / or feature up-sampling;

[0227] generate a third feature map based on the second feature map and a first feature map with a size consistent with that of the second feature map;

[0228] take the third feature map as an updated to-be-processed feature map, return to the step of performing feature sampling on the to-be-processed feature map to generate a second feature map, until the size of the generated third feature map is consistent with a maximum size corresponding to the first feature map, and determine the third feature map with the maximum size as the target feature map.

[0229] In a possible implementation, the apparatus further includes a training module 304 configured to train the target neural network according to the following steps:

[0230] obtaining training samples, wherein the training samples include sample images, first depth maps corresponding to the sample images, and second depth maps corresponding to the sample images, and the first depth maps have higher accuracy than the second depth maps;

[0231] training a neural network to be trained using the training samples to obtain the target neural network.

[0232] In a possible implementation, in a case where the training samples further include sample normal maps and sample semantic maps corresponding to the sample images, and the target neural network includes a multi-task network and a depth optimization network, the training module 304, when training the neural network to be trained using the training samples to obtain the target neural network, is configured to:

[0233] inputting the second depth maps and the sample images into the multi-task network to obtain predicted noise maps, predicted semantic maps, and predicted normal maps corresponding to the sample images;

[0234] inputting the predicted noise maps and the second depth maps into the depth optimization network to generate the predicted depth maps;

[0235] training the neural network to be trained based on at least one of combinations of the predicted semantic maps and the sample semantic maps, the predicted normal maps and the sample normal maps, the predicted depth maps and the first depth maps, and the predicted depth maps and the second depth maps to obtain the target neural network.

[0236] In a possible implementation, when training the neural network to be trained based on at least one of combinations of the predicted semantic maps and the sample semantic maps, the predicted normal maps and the sample normal maps, the predicted depth maps and the first depth maps, and the predicted depth maps and the second depth maps to obtain the target neural network, the training module 304 is configured to:

[0237] determining, based on the predicted depth maps and the first depth maps, a first loss value for representing a depth information deviation and a second loss value for representing a depth gradient deviation; determining, based on the predicted semantic maps and the sample semantic maps, a third loss value for representing a semantic information deviation; determining, based on the predicted normal maps and the sample normal maps, a fourth loss value for representing a normal information deviation; and determining, based on the second depth maps and the predicted depth maps, a fifth loss value for representing a depth distortion information;

[0238] determining a target loss value based on one or more of the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value;

[0239] train the neural network to be trained based on the target loss value, to obtain the target neural network.

[0240] In a possible implementation, when the target loss value includes a fourth loss value, the training module 304, when determining the fourth loss value for representing the normal information bias based on the predicted normal map and the sample normal map, is configured to:

[0241] determine a cosine similarity between first pixel information of each first pixel point in the predicted normal map and second pixel information of a second pixel point matched with the first pixel point in the sample normal map based on the predicted normal map and the sample normal map.

[0242] determine the fourth loss value based on the cosine similarity corresponding to each first pixel point and the number of first pixel points.

[0243] In a possible implementation, when the target loss value includes a second loss value, the training module 304, when determining the second loss value for representing the depth gradient bias based on the predicted depth map and the first depth map, is configured to:

[0244] determine a first gradient in a horizontal axis direction and a second gradient in a vertical axis direction of each pixel point in the predicted depth map; and

[0245] determine a third gradient in the horizontal axis direction and a fourth gradient in the vertical axis direction of each pixel point in the first depth map.

[0246] determine the second loss value based on the first gradient and the second gradient corresponding to each third pixel point in the predicted depth map, and the third gradient and the fourth gradient corresponding to a fourth pixel point matched with the third pixel point in the first depth map.

[0247] In a possible implementation, when the target loss value includes a fifth loss value, the training module 304, when determining the fifth loss value for representing the depth distortion information based on a second depth map and the predicted depth map, is configured to:

[0248] perform feature extraction on the predicted depth map and the second depth map respectively by using the trained feature extraction network, to generate a first target feature map corresponding to the predicted depth map and a second target feature map corresponding to the second depth map.

[0249] determine the fifth loss value based on the first target feature map and the second target feature map.

[0250] In a possible implementation, the training module 304 is configured to, when determining the fifth loss value based on the first target feature map and the second target feature map, determine a difference between a first feature value of each first feature point in the first target feature map and a second feature value of a second feature point in the second target feature map that matches the first feature point.

[0251] determine a difference between a first feature value of each first feature point in the first target feature map and a second feature value of a second feature point in the second target feature map that matches the first feature point.

[0252] determine a difference between a first feature value of each first feature point in the first target feature map and a second feature value of a second feature point in the second target feature map that matches the first feature point.

[0253] In a possible implementation, before inputting the second depth map and the sample image into the multi-task network to obtain a predicted noise map, a predicted semantic map, and a predicted normal map corresponding to the sample image, the apparatus further includes an alignment module 305, configured to:

[0254] perform an alignment operation on the sample image and the second depth map to obtain an aligned sample image and an aligned second depth map, where pixel points at the same pixel position in the aligned sample image and the aligned second depth map correspond to the same position in a real scene.

[0255] In a possible implementation, when the target image includes a real image captured by a movable device, after the target depth map is obtained, the apparatus further includes a first generation module 306, configured to:

[0256] determine depth information of at least one first target object included in the target image based on the target depth map.

[0257] determine an occlusion state between the virtual object and the first target object based on the depth information of each first target object and a predetermined display pose of the virtual object.

[0258] generate a display image in which the virtual object is added based on the occlusion state and the target image, and control the movable device to display the display image.

[0259] In a possible implementation, when the target image includes at least one frame of scene image, after the target depth map is obtained, the apparatus further includes a second generation module 307, configured to:

[0260] construct a three-dimensional scene map based on the at least one frame of scene image and the target depth map corresponding to each frame of scene image.

[0261] In a possible implementation, in a case where the target image comprises a road image captured by an image capturing device arranged on the travel device, after the target depth map is obtained, the apparatus further comprises a third generation module 308 configured to:

[0262] determine depth information of at least one second target object included in the target image based on the target depth map;

[0263] control the travel device based on the depth information of each second target object.

[0264] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes templates that can be used to perform the methods described in the above method embodiments, and specific implementations can refer to the descriptions of the above method embodiments. For brevity, they will not be repeated here.

[0265] Based on the same technical concept, the embodiments of the present disclosure also provide an electronic device. Referring to FIG. 4, Figure 4 The electronic device 400 provided by the embodiments of the present disclosure comprises a processor 401, a memory 402, and a bus 403. The memory 402 is configured to store execution instructions, including an internal memory 4021 and an external memory 4022. The internal memory 4021 is also called an internal memory, which is used to temporarily store operation data in the processor 401 and exchange data with the external memory 4022 such as a hard disk. The processor 401 exchanges data with the external memory 4022 through the internal memory 4021. When the electronic device 400 is running, the processor 401 and the memory 402 communicate through the bus 403, so that the processor 401 executes the following instructions:

[0266] obtain a target image and a to-be-optimized depth map corresponding to the target image;

[0267] perform feature extraction on the to-be-optimized depth map and the target image by using the trained target neural network for depth map optimization, to generate a noise map; wherein the noise map is used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map;

[0268] perform optimization processing on the to-be-optimized depth map by using the target neural network and the noise map, to obtain a target depth map.

[0269] In addition, the embodiments of the present disclosure also provide a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the depth map optimization method described in the above method embodiments are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0270] The embodiment of the present disclosure further provides a computer program product carrying program codes, the program codes comprising instructions for executing the steps of the depth map optimization method described in the above method embodiments, which can be specifically referred to the above method embodiments and will not be repeated here.

[0271] The present disclosure relates to the field of augmented reality, by acquiring image information of a target object in a real environment, and then detecting or recognizing the related features, states and attributes of the target object by means of various visual related algorithms, so as to obtain an AR effect combining virtual and real objects matching a specific application. Exemplarily, the target object can involve a face, a limb, a gesture, a motion, etc. related to a human body, or an identification object, a marker, or a sand table, a display area or a display object related to a venue or a place. The visual related algorithm can involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of an object, pose or depth detection of an object, etc. The specific application can not only involve an interactive scene such as a guide, a navigation, an explanation, a reconstruction, a virtual effect superimposed display, etc. related to a real scene or an object, but also can involve an interactive scene such as a makeup beautification, a limb beautification, a special effect display, a virtual model display, etc. related to a person.

[0272] The detection or recognition of the related features, states and attributes of the target object can be achieved by a convolutional neural network. The convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0273] The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0274] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system and the device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0275] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0276] In addition, the functional units in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0277] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, and various media that can store program codes.

[0278] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method of depth map optimization, the method comprising: The method comprises the following steps: obtaining a target image and a to-be-optimized depth map corresponding to the target image; extracting features of the to-be-optimized depth map and the target image by using a target neural network trained for depth map optimization, and generating a noise map; wherein the noise map is used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map; optimizing the to-be-optimized depth map by using the target neural network and the noise map, and obtaining a target depth map. The target neural network comprises a multi-task network and a depth optimization network, and the process of generating the target depth map comprises the following steps: regarding the to-be-optimized depth map and the target image as a first image with multiple channels, extracting features of the first image by using the multi-task network in the target neural network, and generating a noise map; regarding the noise map and the to-be-optimized depth map as a second image with multiple channels, extracting features of the second image by using the depth optimization network in the target neural network, and obtaining the target depth map after optimization processing.

2. The method of claim 1, wherein, The process of optimizing the to-be-optimized depth map by using the target neural network and the noise map to obtain the target depth map comprises the following steps: extracting features of the noise map and the to-be-optimized depth map by using the target neural network, and generating a target feature map; wherein the target feature map is used to eliminate noise information in the to-be-optimized depth map, and / or is used to supplement depth information in the to-be-optimized depth map; optimizing the to-be-optimized depth map by using the target feature map, and generating the target depth map.

3. The method of claim 2, wherein, The process of extracting features of the noise map and the to-be-optimized depth map by using the target neural network to generate a target feature map comprises the following steps: extracting features of the noise map and the to-be-optimized depth map by using the target neural network to generate a plurality of first feature maps with different sizes; taking a first feature map with the smallest size as a to-be-processed feature map, performing feature sampling on the to-be-processed feature map to generate a second feature map; wherein the feature sampling comprises convolution processing and / or feature up-sampling; generating a third feature map based on the second feature map and a first feature map with a size consistent with that of the second feature map; taking the third feature map as an updated to-be-processed feature map, returning to the step of performing feature sampling on the to-be-processed feature map to generate a second feature map, until the size of the generated third feature map is consistent with the maximum size corresponding to the first feature map, and determining the third feature map with the maximum size as the target feature map.

4. The method according to any one of claims 1 to 3, characterized in that, The target neural network is trained according to the following steps: obtaining training samples, wherein the training samples comprise sample images, first depth maps and second depth maps corresponding to the sample images, and the accuracy of the first depth maps is higher than that of the second depth maps; training a to-be-trained neural network by using the training samples, and obtaining the target neural network.

5. The method of claim 4, wherein, In the case that the training sample further comprises a sample normal map and a sample semantic map corresponding to the sample image, and the target neural network comprises a multi-task network and a depth optimization network, the training of the to-be-trained neural network based on the training sample to obtain the target neural network comprises: inputting the second depth map and the sample image into the multi-task network to obtain a predicted noise map, a predicted semantic map, and a predicted normal map corresponding to the sample image; inputting the predicted noise map and the second depth map into the depth optimization network to generate a predicted depth map; training the to-be-trained neural network based on at least one of a combination of the predicted semantic map and the sample semantic map, the predicted normal map and the sample normal map, the predicted depth map and the first depth map, and the predicted depth map and the second depth map to obtain the target neural network.

6. The method of claim 5, wherein, The training of the to-be-trained neural network based on at least one of a combination of the predicted semantic map and the sample semantic map, the predicted normal map and the sample normal map, the predicted depth map and the first depth map, and the predicted depth map and the second depth map to obtain the target neural network comprises: determining a first loss value for representing a depth information deviation and a second loss value for representing a depth gradient deviation based on the predicted depth map and the first depth map; determining a third loss value for representing a semantic information deviation based on the predicted semantic map and the sample semantic map; determining a fourth loss value for representing a normal information deviation based on the predicted normal map and the sample normal map; and determining a fifth loss value for representing a depth distortion information based on the second depth map and the predicted depth map; determining a target loss value based on one or more of the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value; training the to-be-trained neural network based on the target loss value to obtain the target neural network.

7. The method of claim 6, wherein, In the case that the target loss value comprises the fourth loss value, the determination of the fourth loss value for representing the normal information deviation based on the predicted normal map and the sample normal map comprises: determining a cosine similarity between first pixel information of each first pixel point in the predicted normal map and second pixel information of a second pixel point matching the first pixel point in the sample normal map based on the predicted normal map and the sample normal map; determining the fourth loss value based on the cosine similarity corresponding to each first pixel point and the number of first pixel points.

8. The method of claim 6, wherein, In the case that the target loss value comprises the second loss value, the determination of the second loss value for representing the depth gradient deviation based on the predicted depth map and the first depth map comprises: determining a first gradient of each pixel point in the predicted depth map in a horizontal axis direction and a second gradient of each pixel point in a vertical axis direction; and determining a third gradient of each pixel point in the first depth map in the horizontal axis direction and a fourth gradient of each pixel point in the vertical axis direction. determine the second loss value based on the first gradient and the second gradient corresponding to each third pixel point in the predicted depth map, and the third gradient and the fourth gradient corresponding to a fourth pixel point matched with the third pixel point in the first depth map.

9. The method of claim 6, wherein, In the case where the target loss value includes a fifth loss value, the determination of the fifth loss value for representing depth distortion information based on the second depth map and the predicted depth map includes: performing feature extraction on the predicted depth map and the second depth map respectively by using the trained feature extraction network to generate a first target feature map corresponding to the predicted depth map and a second target feature map corresponding to the second depth map; determine the fifth loss value based on the first target feature map and the second target feature map.

10. The method of claim 9, wherein, The determination of the fifth loss value based on the first target feature map and the second target feature map includes: determining the difference between a first feature value of each first feature point in the first target feature map and a second feature value of a second feature point matched with the first feature point in the second target feature map; determining the sum of squares of the difference corresponding to each first feature point as the fifth loss value.

11. The method of any one of claims 5-10, wherein, Before the input of the second depth map and the sample image into the multi-task network to obtain the predicted noise map, the predicted semantic map and the predicted normal map corresponding to the sample image, the method further includes: performing alignment operation on the sample image and the second depth map to obtain an aligned sample image and an aligned second depth map, wherein the pixel points with the same pixel position in the aligned sample image and the aligned second depth map correspond to the same position in the real scene.

12. The method of any one of claims 1-3, wherein, In the case where the target image includes a real image collected by a movable device, after the target depth map is obtained, the method further includes: determining depth information of at least one first target object included in the target image based on the target depth map; determining an occlusion state between the virtual object and the first target object based on the depth information of each first target object and a predetermined display pose of the virtual object; generating a display image added with the virtual object based on the occlusion state and the target image, and controlling the movable device to display the display image.

13. The method of any one of claims 1-3, wherein, In the case where the target image includes at least one frame of scene image, after the target depth map is obtained, the method further includes: constructing a three-dimensional scene map based on the at least one frame of scene image and the target depth map corresponding to each frame of scene image.

14. The method of any one of claims 1-3, wherein, In the case where the target image includes a road image collected by an image collection device arranged on a running device, after the target depth map is obtained, the method further includes: determining depth information of at least one second target object included in the target image based on the target depth map; controlling the running device based on the depth information of each second target object.

15. A depth map optimization apparatus, characterized by, includes: an acquisition module configured to acquire a target image and a to-be-optimized depth map corresponding to the target image; extracting a feature of the to-be-optimized depth map and the target image by using the trained target neural network for depth map optimization, to generate a noise map; wherein the noise map is used to represent noise information and / or depth missing information corresponding to the to-be-optimized depth map; optimizing the to-be-optimized depth map by using the target neural network and the noise map, to obtain a target depth map; wherein the target neural network comprises a multi-task network and a depth optimization network, and in the process of generating the target depth map, the extracting module is configured to take the to-be-optimized depth map and the target image as a first image of multiple channels, and extract a feature of the first image by using the multi-task network in the target neural network, to generate a noise map; the optimizing module is configured to take the noise map and the to-be-optimized depth map as a second image of multiple channels, and extract a feature of the second image by using the depth optimization network in the target neural network, to obtain the target depth map after optimization processing.

16. An electronic device, comprising: comprise: a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the depth map optimization method in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to execute the steps of the depth map optimization method in any one of claims 1 to 14.

Images