A depth map optimization method and device

Abstract

The present application discloses a depth map optimization method and device. The present application obtains a depth map of the image to be segmented based on the first-view image and the second-view image of the image to be segmented; obtains a first feature vector based on the first-view image in combination with a pre-trained image segmentation model; obtains several prompt points based on the depth map and the first feature vector, and obtains a second feature vector in combination with the image segmentation model; obtains an image segmentation result based on the first feature vector and the second feature vector in combination with the image segmentation model, and optimizes the depth map through the image segmentation result. Through the present application, the generalization ability and accuracy of the image segmentation process can be improved, thereby improving the accuracy of the subsequent optimized depth map and the generalization of the overall depth map generation process.

Description

Technical Field

[0001] The present application relates to the field of stereoscopic vision, and in particular to a depth map optimization method and device. Background Art

[0002] With the rapid development of fields such as autonomous driving and drones, the image processing requirements in the field of stereo vision are becoming more and more complex, and the accuracy requirements for depth maps are also increasing. However, the generation of depth maps often has technical disadvantages such as lack of texture and being greatly affected by occlusions and lighting changes.

[0003] In order to overcome the above technical shortcomings, the existing technology uses an end-to-end depth estimation model. Such a model uses a large number of stereo images for training and can handle various complex visual scenes. However, such a model is affected by the quality of training data and has poor generalization ability. When encountering unprecedented data, its data processing ability will be greatly reduced. In addition, some existing technologies use specific post-processing methods to fill or remove noise from the generated depth map. Such methods often cannot take into account both generalization ability and accuracy at the same time.

[0004] Therefore, how to improve the accuracy of depth maps and the generalization of the generation process has become a technical problem that needs to be solved. Summary of the invention

[0005] The present application provides a depth map optimization method and device to solve the technical problems that the depth map generated by the traditional method is not accurate enough and the generalization ability of the subsequent depth map optimization method is poor.

[0006] In order to solve the above technical problems, in a first aspect, an embodiment of the present application provides a depth map optimization method, including:

[0007] Acquire a depth map of the image to be segmented according to the first perspective image and the second perspective image of the image to be segmented;

[0008] Obtain a first feature vector based on the first perspective image and a pre-trained image segmentation model;

[0009] According to the depth map and the first feature vector, a plurality of prompt points are obtained, and combined with the image segmentation model, a second feature vector is obtained;

[0010] According to the first feature vector and the second feature vector, in combination with the image segmentation model, an image segmentation result is obtained, and the depth map is optimized through the image segmentation result.

[0011] Compared with the prior art, the embodiments of the present application have the following beneficial effects: since the pre-trained image segmentation large model has strong generalization ability, it can handle image segmentation tasks in zero-sample scenarios and obtain image segmentation results. The depth map obtained by the traditional method is optimized according to the segmentation results, which indirectly improves the generalization of the traditional method; at the same time, the depth map initially generated by the traditional method contains the geometric information of the object, which can be used to generate accurate prompt point information, thereby improving the image segmentation accuracy of the image segmentation large model, and the accurate image segmentation results will further improve the optimization results of the depth map.

[0012] In some embodiments of the first aspect of the present application, acquiring a plurality of cue points according to the depth map and the first feature vector includes:

[0013] Acquire a feature map of the image to be segmented according to the first feature vector;

[0014] According to the depth value of each pixel in the depth map, the depth map is graded to obtain depth graded area maps of several levels;

[0015] The feature map and all the depth classification area maps are combined to obtain a number of prompt points.

[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects: when there are problems such as occlusion between different objects in the image to be segmented, resulting in unclear texture, but since the depth value of each pixel is different, the depth value contains rich geometric information. Through the geometric information in the depth map, after further classification according to the depth value, the area map corresponding to the different depth value intervals is obtained. Through these area maps, accurate prompt point information can be obtained, so that the accuracy of subsequent image segmentation is improved.

[0017] In some embodiments of the first aspect of the present application, the step of combining the feature map and all the depth classification region maps to obtain a plurality of prompt points includes:

[0018] Sequentially multiplying and subtracting each depth classification region map from the feature map to obtain a multiplication map and a subtraction map of a corresponding level;

[0019] Determine the first prompt point closest to the particle in the connected area of ​​each of the multiplication graphs and the second prompt point at the center of each block in each of the subtraction graphs;

[0020] All the first cue points and all the second cue points are merged, and duplicate points are removed to obtain a plurality of the cue points.

[0021] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by multiplying and subtracting the area map containing the depth value with the feature map containing the feature information of the image to be segmented, the image information of the block connectivity information is obtained, and according to the connectivity information of the block, the accuracy of the prompt point generation can be improved, so that the subsequent image segmentation can focus on searching the connected area.

[0022] In some embodiments of the first aspect of the present application, acquiring a feature map of the image to be segmented according to the first feature vector includes:

[0023] Obtaining a feature representation of each pixel from the first feature vector;

[0024] In combination with the feature representation of each pixel, a first similarity between each pixel and its adjacent pixels is calculated using a cosine similarity function;

[0025] A feature map of the image to be segmented is generated according to the first similarity of each pixel.

[0026] Compared with the prior art, the embodiments of the present application have the following beneficial effects: through the similarity between the features of each pixel point, the correlation between the pixels in the image to be segmented can be preliminarily judged, and the feature map obtained based on this information can effectively improve the accuracy of subsequent image segmentation.

[0027] In some embodiments of the first aspect of the present application, obtaining an image segmentation result according to the first feature vector and the second feature vector in combination with the image segmentation large model includes:

[0028] According to the first feature vector and the second feature vector, in combination with the mask decoder of the image segmentation model, a segmentation mask corresponding to each of the prompt points is obtained;

[0029] According to the intersection ratio between the segmentation masks, all the segmentation masks are merged to obtain the image segmentation result.

[0030] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by including the first feature vector of the feature information of the original image to be segmented and the second feature vector of the depth value feature, the image information in the subsequently generated mask can be improved, the number of masks generated can be reduced, the computational complexity of subsequent mask merging can be reduced, and the efficiency of subsequent optimized depth map generation can be further improved.

[0031] In some embodiments of the first aspect of the present application, merging all the segmentation masks according to the intersection ratio between the segmentation masks to obtain the image segmentation result includes:

[0032] Calculating a second similarity between each of the segmentation masks, and performing a deduplication operation on all of the segmentation masks according to the second similarity;

[0033] Grouping all the segmentation masks after deduplication according to the block connection degree between the segmentation masks;

[0034] According to the depth value of each segmentation mask, subtraction and merging operations are performed on each segmentation mask in each group, and finally merged according to the intersection ratio between the segmentation masks to obtain the image segmentation result.

[0035] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by deduplicating masks, the efficiency of subsequent data processing can be improved; by grouping masks, similar masks can be subsequently processed within the group, thereby improving the image segmentation effect; the masks are subjected to intra-group subtraction and merging processing in order to achieve more detailed image segmentation, and the masks are merged according to the size of the intersection ratio between each segmentation mask, thereby effectively improving the accuracy of subsequent image segmentation.

[0036] In some embodiments of the first aspect of the present application, the second similarity is obtained by determining a ratio of an intersection to a union between two segmentation masks.

[0037] Compared with the prior art, the embodiments of the present application have the following beneficial effects: for two completely identical segmentation masks, their union and intersection are the same, and when the two segmentation masks do not have any intersection at all, the difference between the two segmentation masks is the largest. Therefore, by performing simple intersection and union operations, the similarity between the two segmentation masks can be quickly and objectively judged, thereby improving the deduplication effect and calculation efficiency.

[0038] In some embodiments of the first aspect of the present application, optimizing the depth map by using the image segmentation result includes:

[0039] Sequentially traverse the depth value of each pixel in the multiplication result of the image segmentation result and the depth image, and determine whether the depth value is a first preset value;

[0040] If it is not the first preset value, the coordinates and depth value of the current pixel are mapped to the depth image; otherwise, according to the distance between the pixel points whose depth values ​​are not the first preset value and the current pixel point and the depth value of the pixel points whose depth values ​​are not the first preset value, the completion value of the current pixel point is calculated, and the completion value is filled into the position corresponding to the current pixel point in the depth image.

[0041] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by optimizing the prompt points and the image segmentation results generated by the large image segmentation model, the depth map is completed and optimized, so that the depth map holes caused by the original image texture information being too complex or not clear enough are correctly filled, thereby effectively improving the accuracy of the depth map and at the same time improving the generalization of the overall optimized depth map generation process.

[0042] In some embodiments of the first aspect of the present application, the calculating the complement value of the current pixel includes:

[0043] Obtaining the reciprocal of the sum of squares of distances between each pixel whose depth value is not the first preset value and the current pixel;

[0044] According to the reciprocal of the distance, a weighted average is performed on the depth values ​​of the pixels whose depth values ​​are not the first preset value to obtain a completion value of the current pixel.

[0045] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by using the weighted average method, the empty areas in the depth map are accurately filled, thereby improving the smoothness of the subsequent optimized depth map.

[0046] In a second aspect, an embodiment of the present invention further provides a depth map optimization device, comprising: a depth map acquisition module, a first feature vector acquisition module, a second feature vector acquisition module, and a depth map optimization module;

[0047] The depth map acquisition module is used to acquire the depth map of the image to be segmented according to the first perspective image and the second perspective image of the image to be segmented;

[0048] The first feature vector acquisition module is used to acquire a first feature vector according to the first perspective image in combination with a pre-trained image segmentation model;

[0049] The second feature vector acquisition module is used to acquire a plurality of prompt points according to the depth map and the first feature vector, and acquire a second feature vector in combination with the image segmentation model;

[0050] The depth map optimization module is used to obtain an image segmentation result based on the first feature vector and the second feature vector in combination with the image segmentation model, and optimize the depth map through the image segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a flow chart of a depth map optimization method provided in some embodiments of the present application;

[0052] Figure 2 An image segmentation result diagram obtained by directly segmenting using an existing image segmentation model;

[0053] Figure 3 A feature map obtained by a depth map optimization method provided by some embodiments of the present application;

[0054] Figure 4 A prompt point display image obtained by a depth map optimization method provided by some embodiments of the present application;

[0055] Figure 5 A schematic flow chart of another depth map optimization method provided in some embodiments of the present application;

[0056] Figure 6 The depth map has a lot of hole noise before optimization;

[0057] Figure 7 A schematic structural diagram of a depth map optimization device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] Embodiment 1

[0060] Please refer to Figure 1 , a depth map optimization method provided in an embodiment of the present application includes: S10 to S40, specifically:

[0061] S10: Acquire a depth map of the image to be segmented according to the first-view image and the second-view image of the image to be segmented.

[0062] Preferably, the depth map of the image to be segmented can be obtained by combining the images taken at each perspective through the SGBM algorithm, or any other depth map generation method can be used, such as: predicting the depth map from a single or multiple images to be segmented through deep learning; or measuring the depth value of the object surface through a sensor; or estimating the physical surface depth value through the principle of signal reflection through sound wave signals or radar signals, thereby obtaining a depth map. The present application can optimize the depth image obtained by any of the above methods.

[0063] S20: Obtain a first feature vector based on the first perspective image in combination with a pre-trained image segmentation model.

[0064] In some embodiments of the present application, obtaining the first feature vector based on the first perspective image in combination with a pre-trained image segmentation model includes: inputting the first perspective image into the image encoding of the pre-trained image segmentation model, generating the first feature vector to obtain image feature information of the image to be segmented.

[0065] S30: Acquire a plurality of prompt points according to the depth map and the first feature vector, and acquire a second feature vector in combination with the image segmentation model.

[0066] Since the image segmentation model generates more masks when processing RGB input, it suggests that it may be overly dependent on texture information. When the texture differences between objects are not obvious, the model may have large errors. The main reason for this problem is that the image segmentation model overly relies on the differences in color and texture when performing classification and segmentation under the condition of automatically generating cue points. Figure 2 , is the image segmentation result obtained by automatically generating cue points. It can be seen that the schoolbag behind the bicycle wheel, the handlebars, and the bottom of the seat are not segmented. It can be seen that in RGB images, large models may not be segmented in the case of occlusion or complex scenes, especially in the case of complex textures. Therefore, it is necessary to give more accurate cue point information to the image segmentation large model.

[0067] In some embodiments of the present application, obtaining a plurality of cue points according to the depth map and the first feature vector includes: S301 to S303, specifically:

[0068] S301: Acquire a feature map of the image to be segmented according to the first feature vector;

[0069] S302: classifying the depth map according to the depth value of each pixel in the depth map to obtain depth classification area maps of several levels;

[0070] S303: Acquire a number of prompt points by combining the feature map and all the depth classification region maps.

[0071] When there are problems such as occlusion between different objects in the image to be segmented, resulting in unclear texture, but their depth values ​​are different, the depth values ​​of the depth map contain rich geometric information. Through the geometric information in the depth map, further classification is performed according to the depth values ​​to obtain regional maps corresponding to different depth value intervals. Through these regional maps, accurate cue point information can be obtained, thereby improving the accuracy of subsequent image segmentation.

[0072] In some embodiments of the present application, the above step S301 can be implemented by the following steps S3011 to S3013, including:

[0073] S3011: Obtaining a feature representation of each pixel from the first feature vector;

[0074] S3012: Calculate the first similarity between each pixel and its adjacent pixels by using a cosine similarity function in combination with the feature representation of each pixel;

[0075] S3013: Generate a feature map of the image to be segmented according to the first similarity of each pixel point.

[0076] refer to Figure 3 , is a feature map obtained through the embodiment of the present application. Through the similarity between the features of each pixel point, the correlation of each pixel point in the image to be segmented can be preliminarily determined. The feature map obtained based on this information can effectively improve the accuracy of subsequent image segmentation.

[0077] In some embodiments of the present application, the above step S303 may be implemented by the following steps S3031 to S3033, including:

[0078] S3031: performing multiplication and subtraction operations on each depth classification region map and the feature map in sequence, respectively, to obtain a multiplication map and a subtraction map of a corresponding level;

[0079] S3032: Determine the first prompt point closest to the particle in the connected area of ​​each of the multiplication graphs and the second prompt point at the center of each block in each of the subtraction graphs;

[0080] S3033: Merge all first cue points and all second cue points, and remove duplicate points to obtain a plurality of the cue points.

[0081] By multiplying and subtracting the area map containing the depth value with the feature map containing the feature information of the image to be segmented, the image information of the block connectivity information is obtained. According to the connectivity information of the block, the accuracy of prompt point generation can be improved, so that the connected area can be searched intensively during subsequent image segmentation.

[0082] Preferably, the above step S303 can also be obtained through the following preferred implementation method: first, the depth map obtained in S10 is graded, such as: the depth map is graded according to the standard of 10 depth value progressions, and the pixel points in the depth map whose depth values ​​are in the depth value range of 0 to 255 are traversed. If the pixel point is within the current progressive depth value interval, the depth value of the pixel point is set to a preset depth value, thereby obtaining 26 corresponding depth graded area maps; further, in combination with the feature map obtained in S301, the 26 classified graded depth maps and the feature map are multiplied and subtracted to obtain two lists of multiplication maps and subtraction maps, and then the point closest to the particle point in the connected area and the center point in each block of the subtraction map are obtained, the points of the obtained images in the two lists are merged, and the duplicate points are removed, thereby obtaining a display map containing prompt points. For details, please refer to Figure 4 The cue point display diagram shown. Through this method, the number of cue points generated can be effectively reduced, thereby reducing the complexity of subsequent calculations and improving the accuracy of the cue points. The above cue point information can also be obtained through neural networks or other methods.

[0083] S40: Obtain an image segmentation result according to the first feature vector and the second feature vector in combination with the image segmentation model, and optimize the depth map through the image segmentation result.

[0084] In some embodiments of the present application, the step of obtaining the image segmentation result according to the first feature vector and the second feature vector in combination with the image segmentation large model includes S401 to S402, specifically:

[0085] S401: Acquire a segmentation mask corresponding to each of the prompt points according to the first feature vector and the second feature vector in combination with a mask decoder of the large image segmentation model;

[0086] S402: Merge all the segmentation masks according to the intersection ratios between the segmentation masks to obtain an image segmentation result.

[0087] The embodiment of the present application, by including a first feature vector of feature information of the original image to be segmented and a second feature vector of depth value features, can improve the image information in the subsequently generated mask and reduce the number of masks generated, thereby improving the efficiency of image segmentation and further improving the efficiency of subsequent optimized depth map generation.

[0088] In some embodiments of the present application, the above S402 may be implemented by the following S4021 to S4023, including:

[0089] S4021: calculating a second similarity between each of the segmentation masks, and performing a deduplication operation on all the segmentation masks according to the second similarity; the second similarity is obtained by determining a ratio of an intersection to a union between two segmentation masks;

[0090] S4022: grouping all the segmentation masks after deduplication according to the block connection degree between the segmentation masks;

[0091] S4023: performing subtraction and merging operations on the segmentation masks in each group according to the depth value of each segmentation mask, and finally merging the segmentation masks according to the intersection ratio between the segmentation masks to obtain an image segmentation result.

[0092] In the embodiments of the present application, the efficiency of subsequent data processing can be improved by deduplicating masks; similar masks can be subsequently processed within the group by grouping, thereby improving the image segmentation effect; masks can be subtracted and merged within the group to segment the target in the image more finely, and masks can be merged according to the intersection ratio between each segmentation mask, thereby effectively improving the accuracy of subsequent image segmentation.

[0093] Preferably, the above S4021 can be implemented by the following preferred implementations:

[0094] Step 1: First calculate the area of ​​each mask and store the results in a list;

[0095] Step 2: Randomly select each pair of masks in the list, calculate their intersection and union, and merge the two masks if the intersection area of ​​the two masks exceeds 70% of their union area;

[0096] Step 3: Temporarily delete the merged mask so that it is no longer considered in the next cycle, and repeat step 2 until no mask can be further merged, completing the deduplication operation.

[0097] Preferably, the above S4023 can be implemented by the following preferred implementations:

[0098] Step 1: First, loop through two random masks in the current group and determine the overlap of the two masks; if the intersection between the two masks accounts for more than 50% of the union of the two masks, obtain the sub-mask. When the difference between the average depth value of the sub-mask and the mask with a smaller area is greater than 2, assign the sub-mask to the mask with a larger area. Otherwise, merge the two masks and assign them to the mask with a larger area, and the mask with a smaller area remains unchanged.

[0099] Step 2: Loop through all the masks of each group obtained in step 1, randomly select two masks, calculate the intersection, and when the intersection is not empty, calculate the proportion of the intersection in the two masks. The mask with a higher proportion retains the intersection part, and the intersection area of ​​the mask with a lower proportion is set to zero, and the value of the remaining area is retained until all masks no longer change.

[0100] In some embodiments of the present application, optimizing the depth map by using the image segmentation result includes:

[0101] Sequentially traverse the depth value of each pixel in the multiplication result of the image segmentation result and the depth image, and determine whether the depth value is a first preset value;

[0102] If it is not the first preset value, the coordinates and depth value of the current pixel are mapped to the depth image; otherwise, according to the distance between the pixel points whose depth values ​​are not the first preset value and the current pixel point and the depth value of the pixel points whose depth values ​​are not the first preset value, the completion value of the current pixel point is calculated, and the completion value is filled into the position corresponding to the current pixel point in the depth image.

[0103] refer to Figure 6 The depth map generated by the existing technology is often affected by factors such as sensor noise, occlusion of image objects, and light changes, resulting in a large amount of void noise, which leads to incomplete or inaccurate depth information. The embodiment of the present application optimizes the image segmentation results generated by the prompt points and the large image segmentation model to complete and optimize the depth map, so that the depth map holes caused by the original image texture information being too complex or not clear enough are correctly filled, which effectively improves the accuracy of the depth map and improves the generalization of the overall optimized depth map generation process.

[0104] In some embodiments of the present application, the calculating the complement value of the current pixel includes:

[0105] Obtaining the reciprocal of the sum of squares of distances between each pixel whose depth value is not the first preset value and the current pixel;

[0106] According to the reciprocal of the distance, a weighted average is performed on the depth values ​​of the pixels whose depth values ​​are not the first preset value to obtain a completion value of the current pixel.

[0107] Through the weighted average method, the empty areas in the depth map are accurately filled, which improves the smoothness of the subsequent optimized depth map.

[0108] Preferably, the above method for calculating the supplement value can be implemented by the following preferred embodiments:

[0109] First, the sum of the squares of the distances between pixels with a depth value of 0 and other pixels with a depth value other than 0 is calculated using the following formula:

[0110] tmp=(yy Py ) 2 +(xx Py ) 2

[0111] Among them, (x, y) is the coordinate of the pixel point with a depth value of 0; (x Py ,y Py ) are the coordinates of the pixel points whose depth value is not 0.

[0112] Next, the inverse of the weight is calculated by summing the squares of the distances between pixels with non-zero depth values ​​and pixels with zero depth values:

[0113]

[0114] Finally, the filling value is calculated by the depth value of each pixel whose depth value is not 0:

[0115]

[0116] refer to Figure 5 , which is a flow chart of another depth map optimization method provided in an embodiment of the present application. It can be seen that after the images of two perspectives are input into the SGBM algorithm, a depth map with low precision is obtained, and the cue points are obtained through the depth map and the feature information of the original image; then, the image segmentation is realized by combining the feature information obtained from the cue points and the feature information of the original image, and an accurate image segmentation result is obtained. Finally, the depth map is optimized according to the accurate image segmentation result to obtain a completed depth map, thereby improving the accuracy of the depth map.

[0117] In summary, a depth map optimization method provided by an embodiment of the present application has the following beneficial effects compared to the prior art: since the pre-trained image segmentation large model has strong generalization ability, it can handle image segmentation tasks in zero-sample scenarios and obtain image segmentation results. The depth map obtained by the traditional method is optimized according to the segmentation results, which indirectly improves the generalization of the traditional method; at the same time, the depth map initially generated by the traditional method contains the geometric information of the object, which can be used to generate accurate prompt point information, thereby improving the image segmentation accuracy of the image segmentation large model, and accurate image segmentation results will further improve the optimization results of the depth map.

[0118] Embodiment 2

[0119] refer to Figure 7 , which is a depth map optimization device provided in an embodiment of the present application, includes: a depth map acquisition module 11, a first feature vector acquisition module 12, a second feature vector acquisition module 13 and a depth map optimization module 14.

[0120] In some embodiments of the present application, the depth map acquisition module 11 is used to acquire a depth map of the image to be segmented according to the first perspective image and the second perspective image of the image to be segmented;

[0121] The first feature vector acquisition module 12 is used to acquire a first feature vector according to the first perspective image in combination with a pre-trained image segmentation model;

[0122] The second feature vector acquisition module 13 is used to acquire a plurality of prompt points according to the depth map and the first feature vector, and acquire a second feature vector in combination with the image segmentation model;

[0123] The depth map optimization module 14 is used to obtain an image segmentation result based on the first feature vector and the second feature vector in combination with the image segmentation model, and optimize the depth map through the image segmentation result.

[0124] In some embodiments of the present application, the second feature vector acquisition module 13 is used to acquire a number of cue points according to the depth map and the first feature vector, including:

[0125] Acquire a feature map of the image to be segmented according to the first feature vector;

[0126] According to the depth value of each pixel in the depth map, the depth map is graded to obtain depth graded area maps of several levels;

[0127] The feature map and all the depth classification area maps are combined to obtain a number of prompt points.

[0128] In some embodiments of the present application, the combining the feature map and all the depth classification area maps to obtain a number of prompt points includes:

[0129] Sequentially multiplying and subtracting each depth classification region map from the feature map to obtain a multiplication map and a subtraction map of a corresponding level;

[0130] Determine the first prompt point closest to the particle in the connected area of ​​each of the multiplication graphs and the second prompt point at the center of each block in each of the subtraction graphs;

[0131] All the first cue points and all the second cue points are merged, and duplicate points are removed to obtain a plurality of the cue points.

[0132] In some embodiments of the present application, acquiring a feature map of the image to be segmented according to the first feature vector includes:

[0133] Obtaining a feature representation of each pixel from the first feature vector;

[0134] In combination with the feature representation of each pixel, a first similarity between each pixel and its adjacent pixels is calculated using a cosine similarity function;

[0135] A feature map of the image to be segmented is generated according to the first similarity of each pixel.

[0136] In some embodiments of the present application, the depth map optimization module 14 is used to obtain an image segmentation result according to the first feature vector and the second feature vector in combination with the image segmentation model, including:

[0137] According to the first feature vector and the second feature vector, in combination with the mask decoder of the image segmentation model, a segmentation mask corresponding to each of the prompt points is obtained;

[0138] According to the intersection ratio between the segmentation masks, all the segmentation masks are merged to obtain the image segmentation result.

[0139] In some embodiments of the present application, merging all the segmentation masks according to the intersection ratio between the segmentation masks to obtain the image segmentation result includes:

[0140] Calculating a second similarity between each of the segmentation masks, and performing a deduplication operation on all of the segmentation masks according to the second similarity;

[0141] Grouping all the segmentation masks after deduplication according to the block connection degree between the segmentation masks;

[0142] According to the depth value of each segmentation mask, subtraction and merging operations are performed on each segmentation mask in each group, and finally merged according to the intersection ratio between the segmentation masks to obtain the image segmentation result.

[0143] In some embodiments of the present application, the second similarity is obtained by determining the ratio of the intersection and union between two segmentation masks.

[0144] In some embodiments of the present application, the depth map optimization module 14 is used to optimize the depth map according to the image segmentation result, including:

[0145] Sequentially traverse the depth value of each pixel in the multiplication result of the image segmentation result and the depth image, and determine whether the depth value is a first preset value;

[0146] If it is not the first preset value, the coordinates and depth value of the current pixel are mapped to the depth image; otherwise, according to the distance between the pixel points whose depth values ​​are not the first preset value and the current pixel point and the depth value of the pixel points whose depth values ​​are not the first preset value, the completion value of the current pixel point is calculated, and the completion value is filled into the position corresponding to the current pixel point in the depth image.

[0147] In some embodiments of the present application, the calculating the complement value of the current pixel includes:

[0148] Obtaining the reciprocal of the distance between each pixel whose depth value is not the first preset value and the current pixel;

[0149] According to the reciprocal of the distance, a weighted average is performed on the depth values ​​of the pixels whose depth values ​​are not the first preset value to obtain a completion value of the current pixel.

[0150] In summary, a depth map optimization device provided by an embodiment of the present application has the following beneficial effects compared with the prior art: since the pre-trained image segmentation large model has strong generalization ability, it can handle image segmentation tasks in zero-sample scenarios and obtain image segmentation results. The depth map obtained by the traditional method is optimized according to the segmentation results, which indirectly improves the generalization of the traditional method; at the same time, the depth map preliminarily generated by the traditional method contains the geometric information of the object, which can be used to generate accurate prompt point information, thereby improving the image segmentation accuracy of the image segmentation large model, and the accurate image segmentation results will further improve the optimization results of the depth map.

[0151] Embodiment 3

[0152] Based on the above-mentioned embodiments of the depth map optimization method, another embodiment of the present application provides a depth map optimization terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the depth map optimization method of any embodiment of the present application is implemented.

[0153] Exemplarily, in this embodiment, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the depth map optimization device.

[0154] The depth map optimization device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The depth map optimization terminal device may include, but is not limited to, a processor and a memory.

[0155] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the depth map optimization device, and uses various interfaces and lines to connect the various parts of the entire depth map optimization device. The memory may be used to store the computer program and / or module, and the processor implements various functions of the depth map optimization device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0156] Embodiment 4

[0157] Based on the above-mentioned embodiment of the depth map optimization method, another embodiment of the present application provides a storage medium, wherein the storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the depth map optimization method of any embodiment of the present application.

[0158] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0159] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A depth map optimization method, characterized in that: include: Acquire a depth map of the image to be segmented according to the first perspective image and the second perspective image of the image to be segmented; Obtain a first feature vector based on the first perspective image and a pre-trained image segmentation model; According to the depth map and the first feature vector, a plurality of cue points are obtained, and the plurality of cue points are combined with the image segmentation model to obtain a second feature vector; According to the first feature vector and the second feature vector, in combination with the image segmentation model, an image segmentation result is obtained, and the depth map is optimized through the image segmentation result; The obtaining a plurality of prompt points according to the depth map and the first feature vector includes: Acquire a feature map of the image to be segmented according to the first feature vector; According to the depth value of each pixel in the depth map, the depth map is graded to obtain depth graded area maps of several levels; Combining the feature map and all the depth classification area maps, obtaining a number of prompt points; The step of combining the feature map and all the depth classification area maps to obtain a plurality of prompt points includes: Sequentially multiplying and subtracting each depth classification region map from the feature map to obtain a multiplication map and a subtraction map of a corresponding level; Determine the first prompt point closest to the particle in the connected area of ​​each of the multiplication graphs and the second prompt point at the center of each block in each of the subtraction graphs; Merging all first prompt points and all second prompt points, and removing duplicate points, to obtain a plurality of the prompt points; The step of obtaining an image segmentation result according to the first feature vector and the second feature vector in combination with the image segmentation model includes: According to the first feature vector and the second feature vector, in combination with the mask decoder of the image segmentation model, a segmentation mask corresponding to each of the prompt points is obtained; According to the intersection ratio between the segmentation masks, all the segmentation masks are merged to obtain the image segmentation result; The step of optimizing the depth map by using the image segmentation result includes: Sequentially traverse the depth value of each pixel in the multiplication result of the image segmentation result and the depth image, and determine whether the depth value is a first preset value; If it is not the first preset value, the coordinates and depth value of the current pixel are mapped to the depth image; otherwise, according to the distance between the pixel points whose depth values ​​are not the first preset value and the current pixel point and the depth value of the pixel points whose depth values ​​are not the first preset value, the completion value of the current pixel point is calculated, and the completion value is filled into the position corresponding to the current pixel point in the depth image.

2. A depth map optimization method according to claim 1, characterized in that: The step of obtaining a feature map of the image to be segmented according to the first feature vector includes: Obtaining a feature representation of each pixel from the first feature vector; In combination with the feature representation of each pixel, a first similarity between each pixel and its adjacent pixels is calculated using a cosine similarity function; A feature map of the image to be segmented is generated according to the first similarity of each pixel.

3. A depth map optimization method as claimed in claim 1, characterized in that: The step of merging all the segmentation masks according to the intersection ratio between the segmentation masks to obtain the image segmentation result includes: Calculating a second similarity between each of the segmentation masks, and performing a deduplication operation on all of the segmentation masks according to the second similarity; Grouping all the segmentation masks after deduplication according to the block connection degree between the segmentation masks; According to the depth value of each segmentation mask, subtraction and merging operations are performed on each segmentation mask in each group, and finally merged according to the intersection ratio between the segmentation masks to obtain the image segmentation result.

4. A depth map optimization method as claimed in claim 3, characterized in that: The second similarity is obtained by determining the ratio of the intersection and the union between the two segmentation masks.

5. A depth map optimization method as claimed in claim 1, characterized in that: The calculating the complement value of the current pixel point includes: Obtaining the reciprocal of the sum of squares of distances between each pixel whose depth value is not the first preset value and the current pixel; According to the reciprocal of the distance, a weighted average is performed on the depth values ​​of the pixels whose depth values ​​are not the first preset value to obtain a completion value of the current pixel.

6. A depth map optimization device, characterized in that: include: A depth map acquisition module, a first feature vector acquisition module, a second feature vector acquisition module and a depth map optimization module; The depth map acquisition module is used to acquire the depth map of the image to be segmented according to the first perspective image and the second perspective image of the image to be segmented; The first feature vector acquisition module is used to acquire a first feature vector according to the first perspective image in combination with a pre-trained image segmentation model; The second feature vector acquisition module is used to acquire a number of cue points according to the depth map and the first feature vector, and combine the number of cue points with the image segmentation model to acquire a second feature vector; The depth map optimization module is used to obtain an image segmentation result according to the first feature vector and the second feature vector in combination with the image segmentation model, and optimize the depth map according to the image segmentation result; The second feature vector acquisition module is used to acquire a plurality of cue points according to the depth map and the first feature vector, including: Acquire a feature map of the image to be segmented according to the first feature vector; According to the depth value of each pixel in the depth map, the depth map is graded to obtain depth graded area maps of several levels; Combining the feature map and all the depth classification area maps, obtaining a number of prompt points; The step of combining the feature map and all the depth classification area maps to obtain a plurality of prompt points includes: Sequentially multiplying and subtracting each depth classification region map from the feature map to obtain a multiplication map and a subtraction map of a corresponding level; Determine the first prompt point closest to the particle in the connected area of ​​each of the multiplication graphs and the second prompt point at the center of each block in each of the subtraction graphs; Merging all first prompt points and all second prompt points, and removing duplicate points, to obtain a plurality of the prompt points; The depth map optimization module is used to obtain an image segmentation result according to the first feature vector and the second feature vector in combination with the image segmentation model, including: According to the first feature vector and the second feature vector, in combination with the mask decoder of the image segmentation model, a segmentation mask corresponding to each of the prompt points is obtained; According to the intersection ratio between the segmentation masks, all the segmentation masks are merged to obtain the image segmentation result; The depth map optimization module is used to optimize the depth map according to the image segmentation result, including: Sequentially traverse the depth value of each pixel in the multiplication result of the image segmentation result and the depth image, and determine whether the depth value is a first preset value; If it is not the first preset value, the coordinates and depth value of the current pixel are mapped to the depth image; otherwise, according to the distance between the pixel points whose depth values ​​are not the first preset value and the current pixel point and the depth value of the pixel points whose depth values ​​are not the first preset value, the completion value of the current pixel point is calculated, and the completion value is filled into the position corresponding to the current pixel point in the depth image.

Images