A method for processing parallax maps, an electronic device, a storage medium, and a product.

Abstract

This application provides a method for processing disparity maps, an electronic device, a storage medium, and a product. The processing method includes: acquiring a disparity map to be processed and a corresponding semantic segmentation result map; adjusting the disparity in the disparity map corresponding to the foreground region in the semantic segmentation result map to obtain an adjusted disparity map; and fusing the adjusted disparity map with the disparity map to be processed to obtain a fused disparity map. In this application, by adjusting the disparity in the disparity map corresponding to the foreground region in the semantic segmentation result map, the disparity of the foreground region is made nearly uniform. Fusing the adjusted disparity map with the disparity map to be processed results in a more natural disparity map and a clearer image. This application not only saves time in optimizing disparity maps and reduces the error rate of optimizing disparity maps, but also improves the accuracy of optimizing disparity maps.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to a method for processing parallax maps, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] With the rapid development of binocular 3D reconstruction technology, stereo matching is a crucial step in ensuring accuracy, as the quality of the matching directly impacts reconstruction precision. To improve the reconstruction accuracy of binocular 3D algorithms, a common approach is to obtain a disparity map through stereo matching of two images. This method calculates the matching cost within the maximum disparity range in the horizontal direction based on features such as neighborhood color and spatial relationships, and identifies the disparity map with the minimum matching cost. However, this method suffers from drawbacks. The limited neighborhood reference area and the potential for overlapping textures in large object regions make matching difficult, leading to poor consistency in the final disparity map and an increased error rate. For instance, in portrait mode photography, where high disparity consistency in the human body region is critical, directly using stereo matching results can result in slight blurring of the human body area, leading to poor disparity consistency. Summary of the Invention

[0003] This application provides a method for processing disparity maps, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem of disparity in object regions in related technologies. Figure 1 For applications requiring high consistency, directly using stereo matching results will result in object region parallax. Figure 1 The technical problem of poor consistency leading to an increased optimization error rate is addressed in this application. The technical solution is as follows:

[0004] According to a first aspect of the embodiments of this application, a method for processing disparity maps is provided, the method comprising:

[0005] Obtain the disparity map to be processed and the corresponding semantic segmentation result map. The disparity map is obtained by stereo correction and stereo matching based on a first image and a second image taken separately for the same scene. The semantic segmentation result map is obtained by semantic segmentation of the first image or the second image after stereo correction of the first image and the second image.

[0006] The disparity in the disparity map is adjusted to match the disparity of the foreground region in the semantic segmentation result map to obtain the adjusted disparity map.

[0007] The adjusted disparity map is fused with the disparity map to be processed to obtain a fused disparity map.

[0008] Optionally, when the semantic segmentation result image includes at least a single subject object, the disparity in the disparity map corresponding to the foreground region of the semantic segmentation result image is adjusted to obtain an adjusted disparity map, including:

[0009] Determine the mean disparity value of all pixels in the foreground region of the semantic segmentation result image;

[0010] Determine the difference between the disparity value and the mean value for each pixel;

[0011] The disparity values ​​with differences greater than a preset threshold are adjusted towards the mean to obtain an adjusted disparity map.

[0012] Optionally, when the semantic segmentation result graph includes at least two subject objects, the method further includes:

[0013] Construct the number of cluster categories and feature maps based on the disparity map to be processed and the semantic segmentation result map;

[0014] Clustering is performed based on the number of categories and the feature map to obtain a planar label map;

[0015] The step of adjusting the disparity of the foreground region corresponding to the disparity map in the disparity map to obtain the adjusted disparity map includes: adjusting the disparity of the foreground region corresponding to each category on the planar label map to obtain the adjusted disparity map.

[0016] Optionally, the step of constructing the number of cluster categories and feature maps based on the disparity map to be processed and the semantic segmentation result map includes:

[0017] The semantic segmentation result image is normalized to obtain a normalized semantic segmentation result image;

[0018] Based on the normalized semantic segmentation result map and the disparity map to be processed, a clustering feature map is constructed;

[0019] The number of cluster categories is determined based on the color types in the semantic segmentation result image.

[0020] Optionally, constructing the clustering input feature map based on the normalized semantic segmentation result map and the disparity map to be processed includes:

[0021] The normalized semantic segmentation result map is multiplied pixel by pixel with the disparity map to be processed to obtain the region disparity map;

[0022] Based on the number of pixels N in the region disparity map, the pixel value d of each pixel in the region disparity map, and the x-coordinate and y-coordinate of each pixel, a clustering feature map is constructed.

[0023] Optionally, if the semantic segmentation result image includes a binary image or a color-labeled image, then determining the number of cluster categories based on the color types in the semantic segmentation result image includes:

[0024] For the binary image, the number of cluster categories is set according to the pixel values ​​on the binary image;

[0025] For the color label image, the total number of color labels plus the background color is set as the number of cluster categories; or the total number of all color types on the color label image is set as the number of cluster categories.

[0026] Optionally, adjusting the disparity of the foreground region corresponding to each category on the planar marker map to obtain an adjusted disparity map includes:

[0027] Determine the mean disparity value of all pixels in the foreground region corresponding to each category on the planar marker map;

[0028] Determine the difference between the disparity value and the mean value of each pixel in the foreground region corresponding to each category;

[0029] The disparity values ​​with differences greater than a preset threshold are adjusted to the mean of the corresponding category to obtain the adjusted disparity map.

[0030] According to a second aspect of the embodiments of this application, a disparity map processing apparatus is provided, comprising:

[0031] The acquisition module is used to acquire the disparity map to be processed and the corresponding semantic segmentation result map. The disparity map is obtained by stereo correction and stereo matching based on a first image and a second image taken separately for the same scene. The semantic segmentation result map is obtained by semantic segmentation of the first image or the second image after stereo correction of the first image and the second image.

[0032] The first processing module is used to adjust the disparity in the disparity map corresponding to the foreground region in the semantic segmentation result map to obtain the adjusted disparity map;

[0033] The fusion module is used to fuse the adjusted disparity map with the disparity map to be processed to obtain a fused disparity map.

[0034] Optionally, when the semantic segmentation result graph includes a single subject object, the first processing module includes:

[0035] The first mean determination module is used to determine the mean disparity value of all pixels in the foreground region of the semantic segmentation result map;

[0036] The first difference determination module is used to determine the difference between the disparity value and the mean value of each pixel.

[0037] The first adjustment module is used to adjust the disparity values ​​whose difference is greater than a preset threshold to the mean value to obtain an adjusted disparity map.

[0038] Optionally, when the semantic segmentation result graph includes at least two subject objects, the apparatus further includes:

[0039] The construction module is used to construct the number of cluster categories and feature maps based on the disparity map to be processed and the semantic segmentation result map;

[0040] The clustering module is used to perform clustering based on the number of categories and the feature map to obtain a planar label map;

[0041] The second processing module is used to adjust the disparity of the foreground region corresponding to each category on the planar marker map to obtain an adjusted disparity map.

[0042] Optionally, the building module includes:

[0043] The normalization processing module is used to normalize the semantic segmentation result image to obtain a normalized semantic segmentation result image.

[0044] The feature construction module is used to construct a clustering feature map based on the normalized semantic segmentation result map and the disparity map to be processed; and / or

[0045] The category determination module is used to determine the number of cluster categories based on the color types in the semantic segmentation result image.

[0046] Optionally, the feature construction module includes:

[0047] The multiplication module is used to multiply the normalized semantic segmentation result map with the disparity map to be processed pixel by pixel to obtain the region disparity map;

[0048] The combination module is used to construct a clustered feature map based on the number of pixels N of the region disparity map, the pixel value d of each pixel in the region disparity map, and the x-coordinate and y-coordinate of each pixel.

[0049] Optionally, if the semantic segmentation result image includes: a binary image or a color-labeled image marked with each subject object, then the category determination module includes:

[0050] The first category determination module is used to set the number of clustering categories according to the pixel value features on the binary image; and / or

[0051] The second category determination module is used to set the total number of color labels on the color label map plus the total number of background colors as the number of cluster categories; or to set the total number of all color types on the color label map as the number of cluster categories.

[0052] Optionally, the second processing module includes:

[0053] The second mean determination module is used to determine the mean value of the disparity of all pixels in the foreground region corresponding to each category on the planar marker map;

[0054] The second difference determination module is used to determine the difference between the disparity value and the mean value of each pixel in the foreground region corresponding to each category;

[0055] The second adjustment module is used to adjust the disparity values ​​whose difference is greater than a preset threshold to the mean value of the corresponding category, so as to obtain an adjusted disparity map.

[0056] The fusion module is further configured to fuse the disparity map adjusted by the second adjustment module with the disparity map to be processed to obtain a fused disparity map.

[0057] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0058] processor;

[0059] Memory used to store the processor's executable instructions;

[0060] The processor is configured to execute the instructions to implement the disparity map processing method described above.

[0061] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the disparity map processing method as described above.

[0062] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the disparity map processing method as described above.

[0063] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0064] In this embodiment, a disparity map to be processed and a corresponding semantic segmentation result map are first obtained. The disparity in the disparity map corresponding to the foreground region of the semantic segmentation result map is adjusted to obtain an adjusted disparity map. The adjusted disparity map is then fused with the disparity map to be processed to obtain a fused disparity map. In other words, in this embodiment, by adjusting the disparity of the foreground region in the semantic segmentation result map, the disparity of the foreground region is made nearly uniform. The adjusted disparity map is then fused with the disparity map to be processed, resulting in a more natural fused disparity map and a clearer image. This application adjusts the disparity in the disparity map corresponding to the foreground region of the semantic segmentation result map and fuses the adjusted disparity map with the original disparity map. This not only saves time in optimizing disparity maps and reduces the error rate of optimizing disparity maps, but also improves the accuracy of optimizing disparity maps.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0067] Figure 1 This is a flowchart illustrating a method for processing disparity maps according to an embodiment of this application.

[0068] Figure 2 This is another flowchart illustrating a method for processing a disparity map according to an embodiment of this application.

[0069] Figure 3A This is a schematic diagram of an optimized process provided in this application.

[0070] Figure 3B This application provides a kind of Figure 3A The diagram after optimization.

[0071] Figure 4 This is a block diagram of a parallax map processing apparatus shown in an embodiment of this application.

[0072] Figure 5 This is a block diagram illustrating a first processing module in an embodiment of this application.

[0073] Figure 6 This is another block diagram of a parallax map processing apparatus shown in an embodiment of this application.

[0074] Figure 7 This is a block diagram illustrating a construction module in an embodiment of this application.

[0075] Figure 8 This is a block diagram illustrating an electronic device according to an embodiment of this application.

[0076] Figure 9 This is a block diagram illustrating a parallax map processing apparatus according to an embodiment of this application. Detailed Implementation

[0077] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0078] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0079] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0080] Figure 1 This is a flowchart of a disparity map processing method provided in an embodiment of this application, such as... Figure 1 As shown, this disparity map processing method is used in a terminal or server, where the terminal includes electronic devices such as mobile terminals. The method includes the following steps:

[0081] Step 101: Obtain the disparity map to be processed and the corresponding semantic segmentation result map. The disparity map is obtained by stereo correction and stereo matching of the first image and the second image taken for the same scene respectively. The semantic segmentation result map is obtained by semantic segmentation of the first image or the second image after stereo correction of the first image and the second image.

[0082] In this step, the first image and the second image are images captured by two cameras (such as binocular ranging or binocular portrait mode) of the same scene. The first image can be the left image, and the corresponding second image is the right image; conversely, if the first image is the right image, the corresponding second image is the left image. In this embodiment, the cameras can be arranged horizontally or vertically.

[0083] The disparity map to be processed in this step is the result of stereo correction and stereo matching based on the left and right images. The semantic segmentation result image is the result of semantic segmentation based on the stereo-corrected left image. Stereo matching, also known as disparity estimation, estimates the geometric relationships between pixels in the corresponding images based on multi-view image data (usually binocular image data) captured by the front-end camera. Using disparity estimation, the information of the corresponding viewpoint can be obtained from the information of one viewpoint and its depth (disparity) information, reducing the amount of original data and facilitating the transmission and storage of multi-view video. Stereo matching (i.e., binocular stereo matching) typically includes four steps: matching cost calculation, matching cost aggregation, binocular disparity calculation, and disparity post-processing. Disparity post-processing is a key step affecting stereo matching. The implementation process of these steps is well-known in the art and will not be elaborated here.

[0084] In this embodiment, stereo rectification is first performed on the left image and the input image, outputting the rectified left and right images. Then, stereo matching is performed on the stereo-rectified left and right images; that is, the input for stereo matching is two stereo-rectified images. Through the above matching steps, a disparity map (hereinafter referred to as the disparity map) is obtained as the binocular stereo matching result. This disparity map will show some non-uniform translation in the image space compared to the original image captured by the camera. Therefore, it is necessary to ensure that both the input of the semantic segmentation result image and the disparity map are in the original image space or in the stereo-rectified space. Only in this way will the obtained semantic segmentation result image and the disparity map be in the same image space.

[0085] It should be noted that, in this embodiment, to improve the disparity map optimization effect, the semantic segmentation result map and the disparity map should belong to the same image space. This is achieved by ensuring that the input of the semantic segmentation result map is consistent with the stereo-corrected left image as the input of the disparity map to be processed. In other words, the disparity map to be processed is the result of stereo correction and stereo matching based on the left and right images, while the semantic segmentation result map is the result of semantic segmentation based on the stereo-corrected left image.

[0086] Step 102: Adjust the disparity in the disparity map corresponding to the foreground region in the semantic segmentation result map to obtain the adjusted disparity map.

[0087] In this step, when there is a single main object in the semantic segmentation result image, the adjustment process includes: determining the mean disparity value of all pixels in the foreground region of the semantic segmentation result image; determining the difference between the disparity value of each pixel and the mean; adjusting the disparity values ​​with differences greater than a preset threshold towards the mean, to obtain the adjusted disparity image. Specifically:

[0088] First, the mean disparity value of all pixels in the foreground region of the semantic segmentation result image is calculated. Then, the difference between the disparity value of each pixel and the mean is calculated. Next, it is determined whether the difference is greater than a preset threshold, and disparity values ​​with differences greater than the preset threshold are adjusted towards the mean. This adjustment can be a consistent stretching, for example, for disparity values ​​with differences greater than the preset threshold, the mean can be uniformly reduced; conversely, the mean can be uniformly increased, etc., to obtain an adjusted disparity map. The preset threshold is typically set to 1, meaning that if the difference is greater than or equal to 1, it indicates a large difference, and the corresponding disparity value is considered an outlier.

[0089] In other words, in this step, the consistency stretching adjustment for outliers is to optimize the foreground region of the semantic segmentation result image so that the disparity values ​​of all pixels in the optimized foreground region are close to the mean. However, for the background region (i.e., the background region) of the semantic segmentation result image, since the disparity levels in the background region are rich, it cannot be forcibly stretched to be consistent. Therefore, the consistency stretching adjustment must remove the background region.

[0090] Step 103: Fuse the adjusted disparity map with the disparity map to be processed to obtain the fused disparity map.

[0091] The fusion process in this step can employ classic image fusion methods. Image fusion is the process of weighting and combining two images for display. Specifically, methods such as alpha linear fusion can be used, and the fusion formula is as follows:

[0092] refine_result=alpha*input_disparity+(1–alpha)*scale_disparity

[0093] In this formula, `refine_result` represents the fused result, specifically the refined disparity map in this implementation; `input_disparity` represents the disparity map to be processed; `scale_disparity` represents the adjusted disparity map; and `alpha` (abbreviated as 'a') is the fusion degree of the two disparity maps, also known as the weighting system, with a value ranging from 0 to 1. The specific implementation process of alpha fusion is well-known to those skilled in the art and will not be elaborated upon here.

[0094] In this embodiment, a disparity map to be processed and its corresponding semantic segmentation result map are first obtained. The disparity in the disparity map corresponding to the foreground region in the semantic segmentation result map is adjusted to obtain an adjusted disparity map. The adjusted disparity map is then fused with the disparity map to be processed to obtain a fused disparity map. In other words, in this embodiment, by adjusting the disparity of the foreground region in the semantic segmentation result map, the disparity of the foreground region is made nearly uniform. The adjusted disparity map is then fused with the disparity map to be processed, resulting in a more natural disparity map and a clearer image. This application adjusts the disparity in the disparity map corresponding to the foreground region in the semantic segmentation result map and fuses the adjusted disparity map with the original disparity map. This not only saves time in optimizing disparity maps and reduces the error rate of optimizing disparity maps, but also improves the accuracy of optimizing disparity maps.

[0095] Please also see Figure 2 This is another flowchart illustrating a disparity map processing method provided in this application. The difference between this embodiment and the previous one is that in the previous embodiment, the semantic segmentation result map included a single subject object, while in this embodiment, the semantic segmentation result map includes at least two subject objects. The method includes:

[0096] Step 201: Obtain the disparity map to be processed and the corresponding semantic segmentation result map. The disparity map is obtained by performing stereo correction and stereo matching on the first image and the second image taken for the same scene respectively. The semantic segmentation result map is obtained by performing semantic segmentation on the first image or the second image after stereo correction of the first image and the second image.

[0097] This step is the same as step 101, as detailed above, and will not be repeated here.

[0098] Step 202: Construct the number of cluster categories and feature map based on the disparity map to be processed and the semantic segmentation result map.

[0099] This step specifically includes: 1) Normalizing the semantic segmentation result graph to obtain a normalized semantic segmentation result graph.

[0100] One normalization process in this embodiment includes: if the semantic segmentation result image is a binary image (i.e., a grayscale image), the unsigned character (uchar) semantic segmentation mask in the semantic segmentation result image is first converted to a floating-point mask; then, it is divided by 255 to normalize the pixel values ​​in the semantic segmentation result image to 0-1. Here, grayscale image typically refers to uchar type. If the semantic segmentation result image is a color-marked image, the regions with pixel values ​​greater than 0 in the color-marked image are set to 1, thereby normalizing the pixel values ​​in the semantic segmentation result image to 0-1 as well.

[0101] 2) Construct a clustering feature map based on the normalized semantic segmentation result map and the disparity map to be processed.

[0102] In this step, firstly, the normalized semantic segmentation result image is multiplied pixel by pixel with the disparity image to be processed to obtain a semantically segmented region disparity image; that is, the pixels of the semantic segmentation (mask) result image with a value range of 0-1 are multiplied with the disparity image to obtain a semantically segmented region disparity image. A semantically segmented region disparity image can be represented as d, x, y, where d represents the pixel value f_img; x represents the horizontal coordinate of the pixel, and y represents the vertical coordinate of the pixel.

[0103] Secondly, based on the number of pixels N in the region disparity map, the pixel value d of each pixel in the region disparity map, and the x and y coordinates of each pixel, a clustering feature map is constructed.

[0104] In this step, the pixel d of the region disparity map and the position (x, y) of each pixel in the region disparity map are combined into an N*3 feature map, where N is the number of pixels in the region disparity map and 3 represents the three dimensions, namely (d, x, y). In this embodiment, multiplication is a pixel-to-pixel multiplication; for example, for a disparity map with width w and height h, each pixel has a pixel coordinate (x, y), x∈(0, w), y∈(0, h), and the processed region disparity map itself has a pixel value that can be used as the third dimension. Therefore, based on the number of pixels N in the region disparity map, the pixel value d of each pixel, and the x and y coordinates of the pixel, an N*3 feature map can be constructed, where N = h*w, and 3 represents the three dimensions d, x, and y.

[0105] 3) Determine the number of cluster categories K based on the color categories in the semantic segmentation result image.

[0106] In this step, if the semantic segmentation result image includes a binary image or a color-labeled image marked with each subject object, then the method for determining the number of cluster categories K is as follows:

[0107] Since the number of categories K in the KMeans clustering algorithm represents the number of categories that can be clustered based on given features (such as an N*3 feature map), in this embodiment it should be set to 3, that is, the background is one category, the human body region is one category, and the disputed areas in the middle are one category.

[0108] In this embodiment, for binary images, the number of clusters (kmeans) is set to 2. This is because the pixel values ​​in a binary image are only 0 and 255, and the corresponding colors are black and white, so the number of clusters is 2.

[0109] For color label images, the number of categories K in k-means is set to the number of categories in the color label image. For example, if there are 4 categories of all colors in the color label image (including the background color), then the number of categories K for this cluster is set to 4.

[0110] Of course, for the color label image, the total number of color labels plus the background color can also be set as the number of clusters. For example, if there are 3 color labels on the color label image, and 1 more background color is added, the total number is 4, meaning the number of clusters is set to 4.

[0111] Step 203: Cluster the data according to the number of categories and the feature map to obtain a planar label map.

[0112] In this embodiment, the number of categories K and the feature maps of each category are clustered using a K-means clustering algorithm to obtain a planar label map.

[0113] Among them, the K-means algorithm is the most classic partition-based clustering method. The basic idea of ​​this algorithm is to cluster objects using k points in space as centroids, grouping them according to their closest points. Then, through an iterative method, the centroid values ​​of each cluster are updated successively until the best clustering result is obtained. The centroids can be actual points or virtual points. The algorithm process includes: first, arbitrarily selecting k objects from n data objects as initial cluster centers; for the remaining objects, assigning them to the clusters most similar to these centers (represented by the cluster centers) based on their similarity (i.e., distance); then calculating the cluster center of each new cluster (the mean of all objects in that cluster); repeating this process until the standard measure function begins to converge. Typically, the mean squared error is used as the standard measure function. The advantage of using k clusters is that each cluster is as compact as possible, while the clusters themselves are as far apart as possible. It should be noted that the K-means algorithm is a well-known technique to those skilled in the art, and will not be elaborated upon here.

[0114] Step 204: Adjust the disparity of the foreground region corresponding to each category on the planar marker map to obtain the adjusted disparity map.

[0115] In this step, the mean disparity value of all pixels in the foreground region corresponding to each category on the planar marker map is first determined; then the difference between the disparity value and the mean value of each pixel in the foreground region corresponding to each category is determined; and the disparity values ​​with a difference greater than a preset threshold are adjusted to the mean value of the corresponding category to obtain the adjusted disparity map.

[0116] In other words, in this embodiment, when there are multiple categories on the planar marker map, the adjustment for each category is to optimize the foreground region corresponding to each category so that the disparity value of all pixels in the foreground region of each category after optimization is close to the average value of the foreground region of the current category. However, for the background region (i.e., the background area), since the disparity level of the background region is rich, it cannot be forcibly adjusted to be consistent. Therefore, the background region must be removed during the adjustment.

[0117] In this specific implementation, K-1 categories (excluding the background region) are first traversed, and the disparity distribution of each category is statistically analyzed to obtain the mean disparity value of all pixels in that category. Then, the region is traversed, and the difference between the disparity value of each pixel and the mean is calculated. Disparity values ​​with a difference greater than a preset threshold are adjusted towards the mean of the corresponding region to obtain an adjusted disparity map (scale_disparity). The preset threshold is usually set to 1, meaning that if the difference is greater than or equal to 1, it indicates a large difference, and the corresponding disparity value is considered an outlier.

[0118] Step 205: Fuse the adjusted disparity map with the disparity map to be processed to obtain the fused disparity map.

[0119] The specific implementation process of this step is similar to that of step 103. Step 103 involves fusing the disparity map to be processed after category adjustment to obtain a fused disparity map. In this embodiment, however, all the category-adjusted disparity maps are first fused with the disparity map to be processed to obtain the final fused disparity map. The implementation process of fusing all the category-adjusted disparity maps with the disparity map to be processed is detailed in step 103 above and will not be repeated here.

[0120] Optionally, in this embodiment, the adjusted disparity maps of K-1 categories and the disparity map to be processed are linearly fused using the following linear fusion formula to obtain the final refined disparity. The linear fusion formula is:

[0121] refine_result=alpha*input_disparity+(1–alpha)*scale_disparity.

[0122] In this formula, `refine_result` represents the fused result, specifically the refined disparity map in this implementation; `input_disparity` represents the disparity map to be processed; `scale_disparity` represents the adjusted disparity map; and `alpha` (abbreviated as 'a') is the degree of fusion between the two disparity maps, which can also be understood as a weighted value, ranging from 0 to 1. The specific implementation process of alpha fusion is well-known to those skilled in the art and will not be elaborated upon here.

[0123] For better understanding, please also refer to [the relevant documents / references]. Figure 3A and Figure 3B .in, Figure 3A A schematic diagram provided for implementation of this application, before optimization processing. Figure 3B A method provided for the implementation of this application Figure 3A The diagram after optimization.

[0124] like Figure 3A and Figure 3BAs shown, this embodiment takes optimizing the corresponding area of ​​the arm where the person is standing with their hands on their hips as an example. Before optimization, the disparity value of the corresponding area of ​​the arm where the person is standing with their hands on their hips is darker than the disparity value of the main area of ​​the person. If the existing optimization method (for example, directly using the stretching result based on the k-means label may sometimes have problems in uncertain areas) is applied to portrait mode and the focus is on the head, it is very likely that half of the area corresponding to the person will be clear, while the other half of the area corresponding to the person standing with their hands on their hips will be slightly blurry. However, after using the stretching and fusion optimization provided in this embodiment, the disparity consistency of the corresponding area of ​​the entire person will be much better, making the final result more natural.

[0125] In this embodiment, a disparity map to be processed and its corresponding semantic segmentation result map are first obtained. The number of cluster categories and feature maps are constructed based on the disparity map to be processed and the semantic segmentation result map. Clustering is performed based on the number of categories and the feature maps to obtain a planar label map. The disparity of the foreground region corresponding to each category on the planar label map is adjusted to obtain an adjusted disparity map. The adjusted disparity map is then fused with the disparity map to be processed to obtain a fused disparity map. In other words, in this embodiment, the disparity of the foreground region of each category in the semantic segmentation result map is first adjusted to make the disparity of the foreground region of each category nearly consistent. Then, the adjusted disparity map is fused with the disparity map to be processed, making the final fused disparity map more natural and the resulting image clearer. This application adjusts the foreground region of each category in the semantic segmentation result map and fuses all adjusted disparity maps with the original disparity map, which not only saves time in optimizing disparity maps and reduces the error rate of optimizing disparity maps, but also improves the accuracy of optimizing disparity maps.

[0126] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to this application.

[0127] Figure 4 This is a block diagram of a parallax map processing device provided in an embodiment of this application. (Refer to...) Figure 4 The device includes: an acquisition module 401, a first processing module 402, and a fusion module 403, wherein,

[0128] The acquisition module 401 is used to acquire the disparity map to be processed and the corresponding semantic segmentation result map. The disparity map is obtained by stereo correction and stereo matching based on a first image and a second image taken separately for the same scene. The semantic segmentation result map is obtained by semantic segmentation of the first image or the second image after stereo correction of the first image and the second image.

[0129] The first processing module 402 is used to adjust the disparity in the disparity map corresponding to the foreground region in the semantic segmentation result map to obtain an adjusted disparity map;

[0130] The fusion module 403 is used to fuse the adjusted disparity map with the disparity map to be processed to obtain a fused disparity map.

[0131] Optionally, in another embodiment, based on the above embodiment, when the semantic segmentation result image includes a single subject object, the first processing module 402 includes: a first mean determination module 501, a first difference determination module 502, and a first adjustment module 503, as shown in the schematic diagram below. Figure 5 As shown, where,

[0132] The first mean determination module 501 is used to determine the mean disparity value of all pixels in the foreground region of the semantic segmentation result map;

[0133] The first difference determination module 502 is used to determine the difference between the disparity value and the mean value of each pixel.

[0134] The first adjustment module 503 is used to adjust the disparity values ​​with differences greater than a preset threshold to the mean value to obtain an adjusted disparity map.

[0135] Optionally, in another embodiment, based on the above embodiments, when the semantic segmentation result graph includes at least two subject objects, the device further includes: a construction module 601, a clustering module 602, and a second processing module 603, the structural diagram of which is shown below. Figure 6 As shown, Figure 6 by Figure 4 Taking this as an example, where,

[0136] The construction module 601 is used to construct the number of cluster categories and feature map based on the disparity map to be processed and the semantic segmentation result map;

[0137] The clustering module 602 is used to perform clustering based on the number of categories and the feature map to obtain a planar label map;

[0138] The second processing module 603 is used to adjust the disparity of the foreground region corresponding to each category on the planar marker map to obtain an adjusted disparity map.

[0139] Optionally, in another embodiment, based on the above embodiments, the construction module 601 includes: a normalization processing module 701, a feature construction module 702, and / or a category determination module 703, the structural diagram of which is shown below. Figure 7 As shown, Figure 7 Taking the example of China and Israel including various modules, among which,

[0140] The normalization processing module 701 is used to normalize the semantic segmentation result map to obtain a normalized semantic segmentation result map.

[0141] The feature construction module 702 is used to construct a clustering input feature map based on the normalized semantic segmentation result map and the disparity map to be processed.

[0142] The category determination module 703 is used to determine the number of cluster categories based on the color types in the semantic segmentation result image.

[0143] Optionally, in another embodiment, based on the above embodiments, the feature construction module includes: a multiplication module and a combination module, wherein,

[0144] The multiplication module is used to multiply the normalized semantic segmentation result map with the disparity map to be processed pixel by pixel to obtain the region disparity map;

[0145] This combined module is used to construct a clustered feature map based on the number of pixels N in the region disparity map, the pixel value d of each pixel in the region disparity map, and the x-coordinate and y-coordinate of each pixel.

[0146] Optionally, in another embodiment, based on the above embodiments, if the semantic segmentation result image includes: a binary image or a color-labeled image marked with each subject object, then the category determination module includes: a first category determination module and / or a second category determination module, wherein,

[0147] The first category determination module is used to set the number of clustering categories according to the pixel values ​​on the binary image;

[0148] The second category determination module is used to set the total number of color labels on the color label map plus the total number of background colors as the number of cluster categories; or to set the total number of all color types on the color label map as the number of cluster categories.

[0149] Optionally, in another embodiment, based on the above embodiment, the second processing module includes: a second mean determination module, a second difference determination module, and a second adjustment module, wherein,

[0150] The second mean determination module is used to determine the mean value of the disparity of all pixels in the foreground region corresponding to each category on the planar marker map;

[0151] The second difference determination module is used to determine the difference between the disparity value and the mean value of each pixel in the foreground region corresponding to each category;

[0152] The second adjustment module is used to adjust the disparity values ​​with differences greater than a preset threshold to the mean value of the corresponding category, so as to obtain an adjusted disparity map;

[0153] The fusion module is further configured to fuse the disparity map adjusted by the second adjustment module with the disparity map to be processed to obtain a fused disparity map.

[0154] Optionally, embodiments of this application also provide an electronic device, including:

[0155] processor;

[0156] Memory used to store the processor's executable instructions;

[0157] The processor is configured to execute the instructions to implement the disparity map processing method described above.

[0158] Optionally, embodiments of this application also provide a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the disparity map processing method described above. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0159] Optionally, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the disparity map processing method described above.

[0160] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0161] Please also see Figure 8This is a block diagram of an electronic device 800 provided in an embodiment of this application. For example, the electronic device 800 can be a mobile terminal or a server; in this embodiment, a mobile terminal is used as an example for explanation. For example, the electronic device 800 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0162] Reference Figure 8 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0163] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0164] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0165] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0166] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0167] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0168] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0169] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0170] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0171] In an embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the disparity map processing method described above.

[0172] In one embodiment, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device 800, enables the electronic device 800 to perform the disparity map processing method as described above.

[0173] In one embodiment, a computer program product is also provided, including a computer program or instructions that, when executed by a processor 820, implement the disparity map processing method as described above.

[0174] Figure 9 This is a block diagram of an apparatus 900 for processing disparity maps according to an embodiment of this application. For example, apparatus 900 can be provided as a server. See also... Figure 9 The apparatus 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by memory 932 for storing instructions, such as application programs, that can be executed by the processing component 922. The application programs stored in memory 932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 922 is configured to execute instructions to perform the disparity map processing method described above.

[0175] Device 900 may also include a power supply component 926 configured to perform power management of device 900, a wired or wireless network interface 950 configured to connect device 900 to a network, and an input / output (I / O) interface 958. Device 900 can operate on an operating system stored in memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0176] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0177] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method of processing a parallax map, characterized by, The method comprises the following steps: obtaining a disparity map to be processed and a corresponding semantic segmentation result map, wherein the disparity map is obtained based on stereo rectification and stereo matching of a first image and a second image respectively taken for the same scene, and the semantic segmentation result map is obtained by performing semantic segmentation on the first image or the second image after stereo rectification; adjusting the disparity corresponding to the foreground region in the semantic segmentation result map in the disparity map to obtain an adjusted disparity map; fusing the adjusted disparity map with the disparity map to be processed to obtain a fused disparity map; when the semantic segmentation result map includes at least two main objects, the method further comprises: constructing a class number and a feature map of clustering according to the disparity map to be processed and the semantic segmentation result map; performing clustering according to the class number and the feature map to obtain a plane label map; the step of adjusting the disparity corresponding to the foreground region in the semantic segmentation result map in the disparity map to obtain an adjusted disparity map comprises: adjusting the disparity of the foreground region corresponding to each class on the plane label map respectively to obtain an adjusted disparity map; the step of constructing a feature map of clustering according to the normalized semantic segmentation result map and the disparity map to be processed comprises: performing normalization processing on the semantic segmentation result map to obtain a normalized semantic segmentation result map; constructing a feature map of clustering according to the normalized semantic segmentation result map and the disparity map to be processed.

2. The method of processing a parallax map according to claim 1, wherein, when the semantic segmentation result map includes a single main object, the step of adjusting the disparity corresponding to the foreground region in the semantic segmentation result map in the disparity map to obtain an adjusted disparity map comprises: determining the mean value of the disparity values of all pixel points of the foreground region in the semantic segmentation result map; determining the difference between the disparity value of each pixel point and the mean value; adjusting the disparity value greater than the preset threshold to the mean value to obtain an adjusted disparity map.

3. The method of processing a parallax map according to claim 1, wherein, the step of constructing a class number of clustering according to the semantic segmentation result map comprises: determining the class number of clustering according to the color category in the semantic segmentation result map.

4. The method of processing a parallax map according to claim 3, wherein, the step of constructing a feature map of clustering according to the normalized semantic segmentation result map and the disparity map to be processed comprises: multiplying the normalized semantic segmentation result map and the disparity map to be processed pixel by pixel to obtain a region disparity map; constructing a feature map of clustering based on the pixel point number N of the region disparity map, the pixel value d of each pixel point of the region disparity map, and the horizontal coordinate x and the vertical coordinate y of each pixel point.

5. The method of processing a parallax map according to claim 3, wherein, if the semantic segmentation result map includes a binary image or a color label image, the step of determining the class number of clustering according to the color category in the semantic segmentation result map comprises: for the binary image, setting the class number of clustering according to the pixel value on the binary image; and / or For the color label graph, the total number of color labels on the color label graph plus the total number of background colors after the background colors is set as the number of categories of clustering; or the number of all color categories on the color label graph is set as the number of categories of clustering.

6. The method of processing a parallax map according to claim 1, wherein, The adjusting processing of the disparity of the foreground region corresponding to each category on the plane mark graph respectively to obtain an adjusted disparity map comprises: determining the mean value of the disparity values of all pixel points of the foreground region corresponding to each category on the plane mark graph; determining the difference between the disparity value and the mean value of each pixel point of the foreground region corresponding to each category; adjusting the disparity value greater than the preset threshold value to the mean value of the corresponding category respectively to obtain an adjusted disparity map.

7. An electronic device, comprising: comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the disparity map processing method of any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the disparity map processing method of any one of claims 1 to 6.

9. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the disparity map processing method of any one of claims 1 to 6.

Images