An image information determination method, device and computer readable storage medium

Abstract

The embodiment of the present application discloses a kind of image information determination methods, comprising: obtaining sample image and the sample depth map of sample image, the sample occlusion edge binary graph of sample image and the sample occlusion edge direction graph of sample image;Different levels of sample feature map of each sample image are obtained, based on different levels of sample feature map, sample depth map, sample occlusion edge binary graph and sample occlusion edge direction graph, model training is carried out to initial occlusion edge detection model and initial depth prediction model, and target model with occlusion edge detection and depth estimation function is obtained;The feature of image to be processed is extracted to obtain different levels of image to be processed feature map of image to be processed;Based on target model, the depth map of image to be processed, the occlusion edge binary graph of image to be processed and the occlusion edge direction graph of image to be processed are obtained by processing image to be processed feature map.The embodiment of the present application also discloses a kind of image information determination equipment and computer readable storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the image information determination technology in the communication field, and in particular to an image information determination method, device and computer readable storage medium. BACKGROUND

[0002] Estimating the scene depth as the most basic task in scene understanding, the current industry commonly uses binocular cameras or high-quality laser radars to obtain. However, the binocular camera is limited by the binocular baseline, and the range of depth estimation is limited; the high-quality laser radar is also high in price, and it is difficult to be used in various scenes on a large scale. Based on this, a computer vision basic task-monocular depth estimation task is currently proposed. And the main idea of the monocular depth estimation method of the depth learning in the related art is basically based on the network architecture of the encoder-decoder, and the difference between the predicted depth map and the actual depth map is constrained by using the loss function to determine the depth value. At the same time, the occlusion edge detection is a more challenging task based on image edge detection, which specifically needs to locate the edge of the foreground object of interest, and express the front-back relationship of the foreground-background or foreground-foreground on both sides of the edge in space.

[0003] However, the related art cannot fully utilize the correlation between the occlusion edge and the depth direction in the occlusion edge detection scheme, and the feature extractor cannot fully extract the related features. Moreover, the existing monocular depth estimation scheme has a depth gap (that is, an occlusion edge), which causes the depth map to have a relatively serious distortion. SUMMARY

[0004] To solve the above technical problems, the embodiments of the present application expect to provide an image information determination method, device and computer readable storage medium, which solve the problems that the related art cannot fully utilize the correlation between the occlusion edge and the depth direction, and the depth map has a relatively serious distortion, fully utilize the correlation between the occlusion edge and the depth direction, reduce the occurrence of false detection and missed detection of the occlusion edge, and improve the regression accuracy of the depth map.

[0005] The technical scheme of the present application is implemented as follows:

[0006] An image information determination method, the method comprising:

[0007] obtaining a sample image and a sample depth map of the sample image, a sample occlusion edge binary map of the sample image and a sample occlusion edge direction map of the sample image;

[0008] obtain different levels of sample feature maps of each sample image, and perform model training on an initial occlusion edge detection model and an initial depth prediction model based on the different levels of sample feature maps, the sample depth map, the sample occlusion edge binary map and the sample occlusion edge direction map, to obtain a target model; wherein the target model has occlusion edge detection and depth estimation functions;

[0009] perform feature extraction on the to-be-processed image to obtain different levels of to-be-processed feature maps of the to-be-processed image;

[0010] perform processing on the to-be-processed feature maps based on the target model to obtain a depth map of the to-be-processed image, an occlusion edge binary map of the to-be-processed image and an occlusion edge direction map of the to-be-processed image.

[0011] In the above scheme, the different levels of sample feature maps of each sample image are obtained by:

[0012] The residual network is used to perform feature extraction on each sample image to obtain different levels of sample feature maps of each sample image.

[0013] In the above scheme, the model training on the initial occlusion edge detection model and the initial depth prediction model based on the different levels of sample feature maps, the sample depth map, the sample occlusion edge binary map and the sample occlusion edge direction map to obtain the target model comprises:

[0014] Based on a first feature decoder, a second feature decoder and a feature map of a target level in the sample feature maps, first intermediate feature maps and second intermediate feature maps are obtained; wherein the first feature decoder is included in the initial occlusion edge detection model; the second feature decoder is included in the initial depth prediction model;

[0015] The feature fusion module in the initial occlusion edge detection model is used to process each level of feature map of the sample image to obtain a fusion feature map;

[0016] Based on the fusion feature map, the first intermediate feature map and the second intermediate feature map, a predicted depth map, a predicted occlusion edge binary map and a predicted occlusion edge direction map are determined;

[0017] Based on the predicted depth map, the predicted occlusion edge binary map, the predicted occlusion edge direction map, the sample depth map, the sample occlusion edge binary map, the sample occlusion edge direction map and a target loss function, model training is performed to update parameters in the initial occlusion edge detection model and the initial depth prediction model until a target condition is met, to obtain the target model.

[0018] In the scheme, the first intermediate feature map and the second intermediate feature map are obtained by decoding the target level feature map in the sample feature map based on the first feature decoder and the second feature decoder.

[0019] The deepest level feature map of the sample feature map is processed by using the target instance segmentation network to obtain a deep feature map.

[0020] The deep feature map is processed by using the first feature decoder and the second feature decoder to obtain the first intermediate feature map and the second intermediate feature map; wherein the first feature decoder and the second feature decoder include a plurality of deconvolution modules.

[0021] In the scheme, the first intermediate feature map and the second intermediate feature map are obtained by decoding the target level feature map in the sample feature map based on the first feature decoder and the second feature decoder.

[0022] The first parameter and the second parameter of the first feature decoder are determined, and the third parameter and the fourth parameter of the second feature decoder are determined.

[0023] The deep feature map is processed by using the first layer deconvolution layer of the first feature decoder and the first layer deconvolution layer of the second feature decoder to obtain the first layer first feature map and the first layer second feature map.

[0024] The first input feature map of the i+1 layer is determined based on the first parameter, the first feature map of the i layer, the third parameter and the second feature map of the i layer; wherein i is a positive integer greater than 1.

[0025] The second input feature map of the i+1 layer is determined based on the second parameter, the first feature map of the i layer, the fourth parameter and the second feature map of the i layer.

[0026] The first input feature map of the i+1 layer is processed by using the i+1 layer deconvolution layer of the first feature decoder, and the second input feature map of the i+1 layer is processed by using the i+1 layer deconvolution layer of the second feature decoder, until the last layer deconvolution layer of the first feature decoder and the second feature decoder is processed to obtain the first intermediate feature map and the second intermediate feature map.

[0027] In the scheme, the feature fusion module in the initial occlusion edge detection model is used to process each level feature map of the sample image to obtain a fusion feature map.

[0028] The feature fusion module is used to perform convolution processing on each level of feature maps of the sample image, and the each level of feature maps subjected to the convolution processing is subjected to deconvolution processing to obtain a plurality of third intermediate feature maps.

[0029] The plurality of third intermediate feature maps are connected in a channel dimension to obtain the fusion feature map.

[0030] In the above scheme, the fusion feature map, the first intermediate feature map and the second intermediate feature map are used to determine a predicted depth map, a predicted occlusion edge binary map and a predicted occlusion edge direction map, which comprises:

[0031] The fusion feature map is connected with the first intermediate feature map and the second intermediate feature map in a channel dimension to obtain a first connection feature map and a second connection feature map.

[0032] The second connection feature map is processed by a convolution layer to obtain the predicted occlusion edge binary map and the predicted occlusion edge direction map, and the second connection feature map is processed by the convolution layer to obtain the predicted depth map, wherein the convolution layer does not have an activation function.

[0033] In the above scheme, the method further comprises:

[0034] A first loss function of an occlusion edge detection task and a second loss function of a depth estimation task are determined.

[0035] A first weight of the occlusion edge detection task and a second weight of the depth estimation task are determined.

[0036] The target loss function is determined based on the first weight, the first loss function, the second weight and the second loss function.

[0037] An image information determination device, the device comprising: a processor, a memory and a communication bus;

[0038] The communication bus is used to realize the communication connection between the processor and the memory;

[0039] The processor is used to execute the image information determination program stored in the memory to realize the steps of the above-mentioned image information determination method.

[0040] A computer readable storage medium, the computer readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors to realize the steps of the above-mentioned image information determination method.

[0041] The image information determination method, device, and computer-readable storage medium provided in this application acquire sample images and their sample depth maps, sample occlusion edge binary maps, and sample occlusion edge direction maps. Different levels of sample feature maps are acquired for each sample image. Based on these different levels of feature maps, depth maps, occlusion edge binary maps, and occlusion edge direction maps, an initial occlusion edge detection model and an initial depth prediction model are trained to obtain a target model with occlusion edge detection and depth estimation functions. Feature extraction is performed on the image to be processed, resulting in different levels of feature maps to be processed. These feature maps are then processed based on the target model. By obtaining the depth map, the binary map of the occlusion edges, and the orientation map of the occlusion edges of the image to be processed, a target model with occlusion edge detection and depth estimation functions can be generated. This target model is then used to process the feature map of the image to be processed, yielding the depth map, the binary map of the occlusion edges, and the orientation map of the occlusion edges. This solves the problems in related technologies, such as the inability to fully utilize the correlation between occlusion edges and depth direction, and the severe distortion of the depth map. It fully utilizes the correlation between occlusion edges and depth direction, reduces false detections and false negatives of occlusion edges, and improves the regression accuracy of the depth map. Attached Figure Description

[0042] Figure 1 A flowchart illustrating an image information determination method provided for an embodiment of this application;

[0043] Figure 2 A flowchart illustrating another image information determination method provided for an embodiment of this application;

[0044] Figure 3 A schematic diagram illustrating the determination of a predicted occlusion edge binary map, a predicted occlusion edge orientation map, and a predicted depth map in an image information determination method provided in an embodiment of this application;

[0045] Figure 4 A flowchart illustrating yet another image information determination method provided for an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the structure of an image information determination device provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0048] It should be understood that the specific embodiments described herein are merely exemplary and do not limit the application.

[0049] Embodiments of the present application provide an image information determination method, which can be applied to an image information determination device, as shown in Figure 1 The method comprises the following steps:

[0050] Step 101, obtaining a sample image and a sample depth map of the sample image, a sample occlusion edge binary map of the sample image and a sample occlusion edge direction map of the sample image.

[0051] In embodiments of the present application, the sample image can be a plurality of images with depth obtained in advance; each sample depth map can refer to an image representing the depth information of each sample image; each sample occlusion edge binary map can be a black and white image representing the occlusion edge of each sample image; and each sample occlusion edge direction map can be an image representing the direction information of the occlusion edge of each sample image.

[0052] Step 102, obtaining a sample feature map of different levels of each sample image, and performing model training on an initial occlusion edge detection model and an initial depth prediction model based on the sample feature map of different levels, the sample depth map, the sample occlusion edge binary map and the sample occlusion edge direction map, to obtain a target model.

[0053] The target model has occlusion edge detection and depth estimation functions.

[0054] In embodiments of the present application, the sample feature map can be an image capable of representing the feature information of different levels of the sample image; and the sample feature map can be obtained by performing feature extraction on each sample image. The target model can be obtained by performing model training on the initial occlusion edge detection model and the initial depth prediction model based on the sample feature map of different levels, the sample depth map, the sample occlusion edge binary map, the sample occlusion edge direction map and a target loss function, using a neural network algorithm.

[0055] Step 103, performing feature extraction on the to-be-processed image to obtain a to-be-processed feature map of different levels of the to-be-processed image.

[0056] In embodiments of the present application, the method of performing feature extraction on the to-be-processed image can be the same as the method of performing feature extraction on each sample image; that is, the to-be-processed feature map can be an image capable of representing the feature information of different levels of the to-be-processed image.

[0057] Step 104, processing the to-be-processed feature map based on the target model to obtain a depth map of the to-be-processed image, an occlusion edge binary map of the to-be-processed image and an occlusion edge direction map of the to-be-processed image.

[0058] In the embodiment of the present application, the feature map to be processed can be input into the target model, and the target model can obtain the depth map of the image to be processed, the binary image of the occlusion edge of the image to be processed and the direction image of the occlusion edge of the image to be processed after depth estimation and occlusion edge detection on the feature map to be processed.

[0059] The image information determination method provided by the embodiment of the present application obtains a sample image and a sample depth map of the sample image, a sample binary image of the occlusion edge of the sample image and a sample direction image of the occlusion edge of the sample image, obtains a sample feature map of different levels of each sample image, and performs model training on an initial occlusion edge detection model and an initial depth prediction model based on the sample feature map of different levels, the sample depth map, the sample binary image of the occlusion edge and the sample direction image of the occlusion edge, to obtain a target model having the functions of occlusion edge detection and depth estimation; feature extraction is performed on an image to be processed to obtain a feature map to be processed of different levels of the image to be processed, and the target model is used to process the feature map to be processed to obtain a depth map of the image to be processed, a binary image of the occlusion edge of the image to be processed and a direction image of the occlusion edge of the image to be processed. In this way, the target model having the functions of occlusion edge detection and depth estimation can be generated, and the target model is used to process the feature map to be processed of the image to be processed, so that the depth map of the image to be processed, the binary image of the occlusion edge of the image to be processed and the direction image of the occlusion edge of the image to be processed can be obtained. Thus, the problem that the correlation between the occlusion edge and the depth direction cannot be fully utilized and the depth map is seriously distorted in the related art is solved. The correlation between the occlusion edge and the depth direction is fully utilized, the false detection and missed detection of the occlusion edge are reduced, and the regression accuracy of the depth map is improved.

[0060] Based on the foregoing embodiment, the embodiment of the present application provides an image information determination method, as shown in Figure 2 The method comprises the following steps:

[0061] Step 201, an image information determination device obtains a sample image and a sample depth map of the sample image, a sample binary image of the occlusion edge of the sample image and a sample direction image of the occlusion edge of the sample image.

[0062] Step 202, the image information determination device uses a residual network to perform feature extraction on each sample image to obtain a sample feature map of different levels of each sample image.

[0063] In the embodiment of the present application, the residual network can refer to a residual network (Residual Network, ResNet) 50; based on this, as Figure 3As shown, the image information determining device can employ ResNet50 to perform feature extraction on each sample image to obtain at least five levels of sample feature maps of each sample image. In a feasible implementation, the dimensions of each level of sample feature maps are 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the size of the corresponding sample image, and the number of channels are 128, 256, 512, 1024 and 2048 respectively.

[0064] Step 203: The image information determining device obtains a first intermediate feature map and a second intermediate feature map based on the first feature decoder, the second feature decoder and the feature map of the target level in the sample feature maps.

[0065] In the initial occlusion edge detection model, the first feature decoder is included; and in the initial depth prediction model, the second feature decoder is included.

[0066] In the embodiments of the present application, the image information determining device can obtain the first intermediate feature map based on the first feature decoder in the initial occlusion edge detection model and the feature map of the target level; and obtain the second intermediate feature map based on the second feature decoder in the initial depth prediction model and the feature map of the target level. It should be noted that the first feature decoder and the second feature decoder are both stacked by multiple deconvolution modules.

[0067] Step 204: The image information determining device processes each level of feature maps of the sample image by employing the feature fusion module in the initial occlusion edge detection model to obtain a fused feature map.

[0068] In the embodiments of the present application, the image information determining device can perform convolution and deconvolution processing on each level of feature maps of the sample image by employing the feature fusion module in the initial occlusion edge detection model, and connect the processed feature maps to obtain the fused feature map.

[0069] It should be noted that steps 203 and 204 can not be executed in a specific order, that is, steps 203 and 204 can be executed simultaneously; or, step 204 can be executed first and step 203 can be executed later; or, step 203 can be executed first and step 204 can be executed later.

[0070] Step 205: The image information determining device determines a predicted depth map, a predicted occlusion edge binary map and a predicted occlusion edge direction map based on the fused feature map, the first intermediate feature map and the second intermediate feature map.

[0071] The image information determination device can process the fusion feature map and the first intermediate feature map to obtain a predicted occlusion edge binary graph and a predicted occlusion edge direction graph; meanwhile, the image information determination device can process the fusion feature map and the second intermediate feature map to obtain a predicted depth graph.

[0072] In step 206, the image information determination device performs model training on the parameters in the initial occlusion edge detection model and the initial depth prediction model based on the predicted depth graph, the predicted occlusion edge binary graph, the predicted occlusion edge direction graph, the sample depth graph, the sample occlusion edge binary graph, the sample occlusion edge direction graph, and a target loss function until a target condition is met, to obtain a target model.

[0073] The target model has the functions of occlusion edge detection and depth estimation.

[0074] In the embodiments of the present application, the image information determination device can use a neural network algorithm, use a target loss function to compare and process the predicted depth graph and the sample depth graph, and determine whether the parameters in the initial depth prediction model need to be updated based on the comparison result; meanwhile, the image information determination device can use the target loss function to compare and process the predicted occlusion edge binary graph and the sample occlusion edge binary graph, and compare and process the predicted occlusion edge direction graph and the sample occlusion edge direction graph, and determine whether the parameters in the initial occlusion edge detection model need to be updated based on the comparison result; if the comparison result shows that the error of the predicted depth graph and the sample depth graph is large, and the error of the predicted occlusion edge binary graph and the sample occlusion edge binary graph is large, and the error of the predicted occlusion edge direction graph and the sample occlusion edge direction graph is large, the image information determination device continues to perform model training on the initial occlusion edge detection model and the initial depth prediction model to update the parameters in the initial occlusion edge detection model and the initial depth prediction model, until the error of the predicted depth graph and the sample depth graph is within a target threshold range, and the error of the predicted occlusion edge binary graph and the sample occlusion edge binary graph is within the target threshold range, and the error of the predicted occlusion edge direction graph and the sample occlusion edge direction graph is within the target threshold range; at this time, it can be considered that the model training is completed and a target model is obtained. The target condition refers to that the error of the predicted depth graph and the sample depth graph is within the target threshold range, and the error of the predicted occlusion edge binary graph and the sample occlusion edge binary graph is within the target threshold range, and the error of the predicted occlusion edge direction graph and the sample occlusion edge direction graph is within the target threshold range.

[0075] In step 207, the image information determination device performs feature extraction on the to-be-processed image to obtain to-be-processed feature graphs of different levels of the to-be-processed image.

[0076] It should be noted that the to-be-processed feature graph can be obtained by the image information determination device using ResNet50 for feature extraction.

[0077] In step 208, the image information determination device processes the to-be-processed feature map based on the target model to obtain a depth map of the to-be-processed image, a binary occlusion edge map of the to-be-processed image, and an occlusion edge direction map of the to-be-processed image.

[0078] It should be noted that the descriptions of the same steps and the same contents in the embodiments can refer to the descriptions in other embodiments, and will not be described here.

[0079] The image information determination method provided in the embodiments of the present application can generate a target model with occlusion edge detection and depth estimation functions, and process a to-be-processed feature map of a to-be-processed image by using the target model, so as to obtain a depth map of the to-be-processed image, a binary occlusion edge map of the to-be-processed image, and an occlusion edge direction map of the to-be-processed image. Thus, the problem that the correlation between the occlusion edge and the depth direction cannot be fully utilized and the depth map is seriously distorted in the related art is solved. The correlation between the occlusion edge and the depth direction is fully utilized, the false detection and missed detection of the occlusion edge are reduced, and the regression accuracy of the depth map is improved.

[0080] Based on the foregoing embodiments, the embodiments of the present application provide an image information determination method, which refers to FIG. 3, and includes the following steps. Figure 4 The method includes the following steps.

[0081] In step 301, an image information determination device acquires a sample image, a sample depth map of the sample image, a sample binary occlusion edge map of the sample image, and a sample occlusion edge direction map of the sample image.

[0082] In step 302, the image information determination device extracts features of each sample image by using a residual network to obtain sample feature maps of different levels of each sample image.

[0083] In step 303, the image information determination device processes a feature map of the deepest level of the sample feature maps by using a target instance segmentation network to obtain a depth feature map.

[0084] In the embodiments of the present application, the target instance segmentation network can refer to a spatial pyramid pooling (SPP) module. That is, the SPP module can be used to perform expression ability enhancement processing on a feature map of the deepest level in the sample feature maps to obtain the depth feature map.

[0085] In step 304, the image information determination device processes the depth feature map by using a first feature decoder and a second feature decoder to obtain a first intermediate feature map and a second intermediate feature map.

[0086] The first feature decoder and the second feature decoder include a multi-layer deconvolution module. The first feature decoder is included in the initial occlusion boundary detection model, and the second feature decoder is included in the initial depth prediction model.

[0087] In the embodiment of the present application, the image information determination device can combine the first feature decoder and the second feature decoder to process the depth feature map to obtain the first intermediate feature map and the second intermediate feature map. That is, as shown in the figure, for each layer of feature map, the parameters of the first feature decoder and the parameters of the second feature decoder can be combined for training, and then the first intermediate feature map and the second intermediate feature map are obtained. Figure 3

[0088] It should be noted that step 304 can be implemented by the following steps:

[0089] Step 304a, the image information determination device determines the first parameter and the second parameter of the first feature decoder, and determines the third parameter and the fourth parameter of the second feature decoder.

[0090] In the embodiment of the present application, the first parameter can be represented by α1, and the second parameter can be represented by β1; the third parameter can be represented by α2, and the fourth parameter can be represented by β2.

[0091] Step 304b, the image information determination device uses the first layer deconvolution layer of the first feature decoder and the first layer deconvolution layer of the second feature decoder to perform deconvolution processing on the depth feature map to obtain the first layer first feature map and the first layer second feature map.

[0092] The first layer deconvolution layer of the first feature decoder can be used to perform deconvolution processing on the depth feature map to obtain the first layer first feature map; at the same time, the first layer deconvolution layer of the second feature decoder is used to perform deconvolution processing on the depth feature map to obtain the first layer second feature map.

[0093] Step 304c, the image information determination device determines the (i+1)th layer first input feature map based on the first parameter, the i-th layer first feature map, the third parameter and the i-th layer second feature map.

[0094] Wherein, i is a positive integer greater than 1.

[0095] Step 304d, the image information determination device determines the (i+1)th layer second input feature map based on the second parameter, the i-th layer first feature map, the fourth parameter and the i-th layer second feature map.

[0096] ​Step 304e, the image information determining device processes the i+1th first input feature map by using the i+1th deconvolution layer of the first feature decoder, and processes the i+1th second input feature map by using the i+1th deconvolution layer of the second feature decoder, until the last deconvolution layer of the first feature decoder and the second feature decoder is processed, to obtain the first intermediate feature map and the second intermediate feature map.

[0097] In the embodiment of the present application, as shown in Figure 3 The first parameter and the product of the first layer first feature map are added to the product of the third parameter and the first layer second feature map to obtain the second layer first input feature map. Meanwhile, the second parameter and the product of the first layer first feature map are added to the product of the fourth parameter and the first layer second feature map to obtain the second layer second input feature map. Then, the second layer first input feature map is deconvolved by using the second layer deconvolution layer of the first feature decoder to obtain the third layer first input feature map. Meanwhile, the second layer second input feature map is deconvolved by using the second layer deconvolution layer of the second feature decoder to obtain the third layer second input feature map. In turn, the fifth layer first input feature map and the fifth layer second input feature map are deconvolved by using the fifth layer deconvolution layer of the first feature decoder and the second feature decoder, so as to obtain the first intermediate feature map and the second intermediate feature map.

[0098] Step 305, the image information determining device convolves each level of feature map of the sample image by using the feature fusion module, and deconvolves each level of feature map which has been convolved to obtain a plurality of third intermediate feature maps.

[0099] It should be noted that each level of feature map can be input into the feature fusion module, and the feature fusion module first convolves the five different scale feature maps by using 1*1 convolution processing, so as to reduce the channel number of each level of feature map to 64, and then deconvolves each level of feature map which has been convolved to increase the resolution of the image to be consistent with the resolution of the sample image, so as to obtain a plurality of groups of five third intermediate feature maps.

[0100] Step 306, the image information determining device connects the plurality of third intermediate feature maps in the channel dimension to obtain a fusion feature map.

[0101] In the embodiment of the present application, each group of five third intermediate feature maps can be connected in the channel dimension to obtain a plurality of fusion feature maps. In addition, the channel number of the fusion feature map can be 320.

[0102] It should be noted that steps 303-304 and steps 305-306 can be executed in no particular order, that is, steps 303-304 and steps 305-306 can be executed simultaneously, or steps 305-306 can be executed first and steps 303-304 can be executed later, or steps 303-304 can be executed first and steps 305-306 can be executed later.

[0103] Step 307, the image information determining device connects the fusion feature map with the first intermediate feature map and the second intermediate feature map respectively in the channel dimension to obtain a first connection feature map and a second connection feature map.

[0104] It should be noted that the fusion feature map with a channel number of 320 can be connected with the first intermediate feature map in the channel dimension to obtain the first connection feature map, and at the same time, the fusion feature map with a channel number of 320 can be connected with the second intermediate feature map in the channel dimension to obtain the second connection feature map.

[0105] Step 308, the image information determining device processes the second connection feature map by using a convolution layer to obtain a predicted occlusion edge binary graph and a predicted occlusion edge direction graph, and processes the second connection feature map by using a convolution layer to obtain a predicted depth graph.

[0106] Among them, the convolution layer does not have an activation function.

[0107] Step 309, the image information determining device performs model training to update the parameters in the initial occlusion edge detection model and the initial depth prediction model based on the predicted depth graph, the predicted occlusion edge binary graph, the predicted occlusion edge direction graph, the sample depth graph, the sample occlusion edge binary graph, the sample occlusion edge direction graph, and the target loss function until a target condition is met to obtain a target model.

[0108] Step 310, the image information determining device extracts features of the to-be-processed image to obtain different levels of to-be-processed feature maps of the to-be-processed image.

[0109] Step 311, the image information determining device processes the to-be-processed feature map based on the target model to obtain a depth graph of the to-be-processed image, an occlusion edge binary graph of the to-be-processed image, and an occlusion edge direction graph of the to-be-processed image.

[0110] Based on the foregoing embodiments, in other embodiments of the present application, before step 309, the following step can also be included:

[0111] Step 312, the image information determining device determines a first loss function of the occlusion edge detection task and a second loss function of the depth estimation task.

[0112] Step 313, the image information determination device determines a first weight of the occlusion edge detection task and a second weight of the depth estimation task.

[0113] Step 314, the image information determination device determines a target loss function based on the first weight, the first loss function, the second weight and the second loss function.

[0114] In the embodiment of the present application, the first weight can be multiplied by the first loss function, and the second weight and the second loss function can be multiplied and summed to obtain the target loss function.

[0115] It should be noted that in order to better guide the model optimization, the loss functions of the occlusion edge detection task and the depth estimation task need to be combined; assuming that the loss function of the occlusion edge detection task is Locc, the corresponding first weight is λ1, and the loss function of the depth estimation task is Ldep, and the corresponding second weight is λ2. The target loss function can be defined as Ltotal = λ1Locc + λ2Ldep.

[0116] In the embodiment of the present application, the performance of the monocular depth estimation task is improved by using occlusion edge detection, and a multi-task model framework is designed. However, the related art only learns the two tasks separately, which depends on a large amount of data set to learn enough features, and lacks additional information for guidance, and can only achieve limited precision. Using the multi-task learning method, the features of the occlusion edge detection task and the depth estimation task can be complementary and fused, and guide each other, so that enough semantic features can be learned for subsequent prediction without a large amount of data set, and the performance of the target model can also be improved.

[0117] It should be noted that the same steps and the same content in the present embodiment and other embodiments are described with reference to the descriptions in other embodiments, which will not be described here.

[0118] The image information determination method provided by the embodiment of the present application can generate a target model with occlusion edge detection and depth estimation functions, and process the to-be-processed feature map of the to-be-processed image using the target model, so as to obtain the depth map of the to-be-processed image, the occlusion edge binary map of the to-be-processed image and the occlusion edge direction map of the to-be-processed image, thereby solving the problem that the related art cannot fully utilize the correlation between the occlusion edge and the depth direction, and the depth map is seriously distorted. The correlation between the occlusion edge and the depth direction is fully utilized, the false detection and missed detection of the occlusion edge are reduced, and the regression accuracy of the depth map is improved.

[0119] Based on the foregoing embodiments, an embodiment of the present application provides an image information determination device, which can be applied to Figure 1 、 2In the image information determination method provided in the embodiment corresponding to 4, refer to Figure 5 As shown, the device may include: a processor 41, a memory 42, and a communication bus 43, wherein:

[0120] Communication bus 43 is used to realize the communication connection between processor 41 and memory 42;

[0121] The processor 41 is used to execute the image information determination program stored in the memory 42 to perform the following steps:

[0122] Obtain the sample image and its sample depth map, sample occlusion edge binary map, and sample occlusion edge orientation map;

[0123] Obtain sample feature maps at different levels for each sample image, and train the initial occlusion edge detection model and the initial depth prediction model based on the sample feature maps, sample depth maps, sample occlusion edge binary maps, and sample occlusion edge orientation maps at different levels to obtain the target model;

[0124] The target model has occlusion edge detection and depth estimation functions;

[0125] Feature extraction is performed on the image to be processed to obtain feature maps of different levels of the image to be processed;

[0126] The target model is used to process the feature map to be processed, resulting in the depth map, the binary map of the occlusion edge of the image to be processed, and the orientation map of the occlusion edge of the image to be processed.

[0127] In other embodiments of this application, the processor 41 is configured to execute the acquisition of sample feature maps at different levels for each sample image stored in the memory 42 to perform the following steps:

[0128] A residual network is used to extract features from each sample image, resulting in sample feature maps at different levels for each sample image.

[0129] In other embodiments of this application, the processor 41 is used to execute the sample feature maps, sample depth maps, sample occlusion edge binary maps, and sample occlusion edge orientation maps stored in the memory 42 based on different levels, to train the initial occlusion edge detection model and the initial depth prediction model to obtain the target model, so as to implement the following steps:

[0130] Based on the first feature decoder, the second feature decoder, and the feature map of the target level in the sample feature map, the first intermediate feature map and the second intermediate feature map are obtained.

[0131] The initial occlusion edge detection model comprises a first feature decoder; and the initial depth prediction model comprises a second feature decoder.

[0132] The feature fusion module in the initial occlusion edge detection model is used to process the feature maps of each level of the sample image to obtain fused feature maps.

[0133] Based on the fused feature maps, the first intermediate feature maps and the second intermediate feature maps, a predicted depth map, a predicted occlusion edge binary map and a predicted occlusion edge direction map are determined.

[0134] Based on the predicted depth map, the predicted occlusion edge binary map, the predicted occlusion edge direction map, the sample depth map, the sample occlusion edge binary map, the sample occlusion edge direction map and a target loss function, model training is performed to update the parameters in the initial occlusion edge detection model and the initial depth prediction model until a target condition is met, so as to obtain a target model.

[0135] In other embodiments of the present application, the processor 41 is configured to execute the first feature decoder, the second feature decoder and the target level feature map in the sample feature map to obtain the first intermediate feature map and the second intermediate feature map, so as to implement the following steps:

[0136] The deepest level feature map of the sample feature map is processed by using the target instance segmentation network to obtain a depth feature map.

[0137] The depth feature map is processed by using the first feature decoder and the second feature decoder to obtain the first intermediate feature map and the second intermediate feature map.

[0138] The first feature decoder and the second feature decoder comprise a plurality of deconvolution modules.

[0139] In other embodiments of the present application, the processor 41 is configured to execute the first feature decoder and the second feature decoder to process the depth feature map to obtain the first intermediate feature map and the second intermediate feature map, so as to implement the following steps:

[0140] The first parameter and the second parameter of the first feature decoder are determined, and the third parameter and the fourth parameter of the second feature decoder are determined.

[0141] The first layer deconvolution layer of the first feature decoder and the first layer deconvolution layer of the second feature decoder are used to perform deconvolution processing on the depth feature map to obtain the first layer first feature map and the first layer second feature map.

[0142] Based on the first parameter, the i-th layer first feature map, the third parameter and the i-th layer second feature map, the i+1-th layer first input feature map is determined; wherein i is a positive integer greater than 1.

[0143] determining the i+1th layer second input feature map based on the second parameter, the i th layer first feature map, the fourth parameter and the i th layer second feature map;

[0144] processing the i+1th layer first input feature map by using the i+1th layer deconvolutional layer of the first feature decoder and processing the i+1th layer second input feature map by using the i+1th layer deconvolutional layer of the second feature decoder until processing is performed by using the last layer deconvolutional layer of the first feature decoder and the second feature decoder, to obtain the first intermediate feature map and the second intermediate feature map.

[0145] In other embodiments of the present application, the processor 41 is configured to execute the image information determination program stored in the memory 42 to perform the following steps:

[0146] performing convolutional processing on each level of feature map of the sample image by using the feature fusion module and performing deconvolutional processing on each level of feature map that has been subjected to convolutional processing to obtain a plurality of third intermediate feature maps;

[0147] connecting the plurality of third intermediate feature maps in the channel dimension to obtain the fusion feature map.

[0148] In other embodiments of the present application, the processor 41 is configured to execute the image information determination program stored in the memory 42 to perform the following steps:

[0149] connecting the fusion feature map with the first intermediate feature map and the second intermediate feature map in the channel dimension respectively to obtain the first connection feature map and the second connection feature map;

[0150] processing the second connection feature map by using the convolutional layer to obtain the predicted occlusion edge binary map and the predicted occlusion edge direction map, and processing the second connection feature map by using the convolutional layer to obtain the predicted depth map;

[0151] wherein the convolutional layer does not have an activation function.

[0152] In other embodiments of the present application, the processor 41 is configured to execute the image information determination program stored in the memory 42 to perform the following steps:

[0153] determining a first loss function of the occlusion edge detection task and a second loss function of the depth estimation task;

[0154] determining a first weight of the occlusion edge detection task and a second weight of the depth estimation task;

[0155] determine the target loss function based on the first weight, the first loss function, the second weight, and the second loss function.

[0156] It should be noted that the specific implementation process of the steps performed by the processor in this embodiment can refer to the implementation process in the image information determination method provided in the corresponding embodiments of Figure 1 、 2 and 4, which will not be described here in detail.

[0157] The image information determination device provided in the embodiments of the present application can generate a target model with occlusion edge detection and depth estimation functions, and process the to-be-processed feature map of the to-be-processed image by using the target model, so as to obtain the depth map of the to-be-processed image, the occlusion edge binary map of the to-be-processed image, and the occlusion edge direction map of the to-be-processed image, thereby solving the problem that the correlation between the occlusion edge and the depth direction cannot be fully utilized in the related art, and the depth map is seriously distorted, fully utilizing the correlation between the occlusion edge and the depth direction, reducing the occurrence of false detection and missed detection of the occlusion edge, and improving the regression accuracy of the depth map.

[0158] Based on the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the image information determination method provided in the corresponding embodiments of Figure 1 、 2 and 4.

[0159] It should be noted that the computer-readable storage medium can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash Memory, a magnetic surface memory, an optical disc, or a Compact Disc Read-Only Memory (CD-ROM) memory, etc. It can also be various electronic devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0160] It should be noted that, in the present document, the terms "comprising", "containing" or any other similar term are intended to encompass non-exclusive inclusions, such that a process, a method, an article or an apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. An element specified by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element, without more restrictions.

[0161] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0162] Those skilled in the art can clearly understand the above-mentioned embodiment method by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0163] The present application is described with reference to the flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0164] These computer program instructions can also be stored in a computer-readable storage medium that can guide the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0165] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide a process for implementing the flow Figure 1 The flow or flows and / or the block ​ The steps of the function specified in the flow

[0166] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image information determination method characterized by, The method comprises: obtaining a sample image and a sample depth map of the sample image, a sample occlusion edge binary map of the sample image and a sample occlusion edge direction map of the sample image; obtaining different levels of sample feature maps of each sample image, and based on the different levels of sample feature maps, the sample depth map, the sample occlusion edge binary map and the sample occlusion edge direction map, performing model training on an initial occlusion edge detection model and an initial depth prediction model to obtain a target model; wherein the target model has occlusion edge detection and depth estimation functions; performing feature extraction on a to-be-processed image to obtain different levels of to-be-processed feature maps of the to-be-processed image; based on the target model, processing the to-be-processed feature maps to obtain a depth map of the to-be-processed image, an occlusion edge binary map of the to-be-processed image and an occlusion edge direction map of the to-be-processed image; wherein, based on the different levels of sample feature maps, the sample depth map, the sample occlusion edge binary map and the sample occlusion edge direction map, performing model training on an initial occlusion edge detection model and an initial depth prediction model to obtain a target model, comprises: based on a first feature decoder, a second feature decoder and a feature map of a target level in the sample feature maps, obtaining a first intermediate feature map and a second intermediate feature map; wherein the first feature decoder is included in the initial occlusion edge detection model; the second feature decoder is included in the initial depth prediction model; using a feature fusion module in the initial occlusion edge detection model to process each level of feature map of the sample image to obtain a fusion feature map; based on the fusion feature map, the first intermediate feature map and the second intermediate feature map, determining a predicted depth map, a predicted occlusion edge binary map and a predicted occlusion edge direction map; based on the predicted depth map, the predicted occlusion edge binary map, the predicted occlusion edge direction map, the sample depth map, the sample occlusion edge binary map, the sample occlusion edge direction map and a target loss function, performing model training to update parameters in the initial occlusion edge detection model and the initial depth prediction model until a target condition is met to obtain the target model.

2. The method of claim 1, wherein, The method comprises: using a residual network to perform feature extraction on each sample image to obtain different levels of sample feature maps of the sample image.

3. The method of claim 1, wherein, The method comprises: using a target instance segmentation network to process a deepest level of feature map of the sample feature map to obtain a depth feature map; using the first feature decoder and the second feature decoder to process the depth feature map to obtain the first intermediate feature map and the second intermediate feature map; wherein the first feature decoder and the second feature decoder comprise a plurality of deconvolution modules.

4. The method of claim 3, wherein, The processing of the deep feature map by the first feature decoder and the second feature decoder to obtain the first intermediate feature map and the second intermediate feature map comprises: Step 1: determining a first parameter and a second parameter of the first feature decoder, and determining a third parameter and a fourth parameter of the second feature decoder; Step 2: performing deconvolution processing on the deep feature map by a first layer deconvolution layer of the first feature decoder and a first layer deconvolution layer of the second feature decoder to obtain a first layer first output feature map and a first layer second output feature map; Step 3: adding a product of the first parameter and an i-th layer first output feature map to a product of the third parameter and an i-th layer second output feature map to obtain an i+1-th layer first input feature map; wherein i is a positive integer greater than or equal to 1; Step 4: adding a product of the second parameter and the i-th layer first output feature map to a product of the fourth parameter and the i-th layer second output feature map to obtain an i+1-th layer second input feature map; Step 5: processing the i+1-th layer first input feature map by an i+1-th layer deconvolution layer of the first feature decoder to obtain an i+1-th layer first output feature map, and processing the i+1-th layer second input feature map by an i+1-th layer deconvolution layer of the second feature decoder to obtain an i+1-th layer second output feature map; The processing of the deep feature map by the first feature decoder and the second feature decoder to obtain the first intermediate feature map and the second intermediate feature map comprises:

5. The method of claim 1, wherein, The processing of the deep feature map by the first feature decoder and the second feature decoder to obtain the first intermediate feature map and the second intermediate feature map comprises: The processing of the deep feature map by the first feature decoder and the second feature decoder to obtain the first intermediate feature map and the second intermediate feature map comprises: The processing of the deep feature map by the first feature decoder and the second feature decoder to obtain the first intermediate feature map and the second intermediate feature map comprises:

6. The method according to claim 1 or 5, characterized in that, The processing of the deep feature map by the first feature decoder and the second feature decoder to obtain the first intermediate feature map and the second intermediate feature map comprises: The method further comprises: determining a first loss function of an occlusion edge detection task and a second loss function of a depth estimation task; 7. The method of claim 6, wherein, determining a first weight of the occlusion edge detection task and a second weight of the depth estimation task; ​ ​ Determine the target loss function based on the first weight, the first loss function, the second weight, and the second loss function.

8. An image information determining apparatus characterized by comprising: The device comprises a processor, a memory, and a communication bus; The communication bus is configured to realize the communication connection between the processor and the memory; The processor is configured to execute an image information determination program stored in the memory to realize the steps of the image information determination method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more programs, which can be executed by one or more processors to realize the steps of the image information determination method according to any one of claims 1-7.

Images