Image processing method, apparatus, device, and computer-readable storage medium

Abstract

The application discloses an image processing method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring a binocular image pair to be processed, wherein the binocular image pair comprises a first image and a second image; calculating a first disparity map of the first image and the second image; and processing the first disparity map according to a disparity map processing model and the first image to obtain a second disparity map. According to the application, a high-quality disparity map can be generated.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of smart terminal technology, the use of electronic devices (such as smartphones and tablets) is becoming increasingly widespread. The vast majority of electronic devices have built-in cameras, and with the increasing processing power of mobile terminals and the development of camera technology, users are demanding higher and higher image quality.

[0003] For example, more and more electronic devices are now equipped with multiple cameras. To achieve better shooting results, dual-camera bokeh can be used to focus the image on the subject, such as in portrait mode on mobile phones. To achieve dual-camera bokeh, it is necessary to calculate the disparity map between the main camera image and the secondary camera image, then estimate the depth based on the disparity map, and finally blur the background of the image based on the depth information.

[0004] However, due to the presence of repetitive textures, weak textures, overexposure, noise, distortion, and other issues, the disparity map obtained through stereo matching is of poor quality and cannot meet the requirements of subsequent image processing. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, and computer-readable storage medium capable of generating high-quality disparity maps.

[0006] In a first aspect, embodiments of this application provide an image processing method, including:

[0007] Obtain a pair of stereo images to be processed, the pair of stereo images including a first image and a second image;

[0008] Calculate the first disparity map between the first image and the second image;

[0009] The first disparity map is processed according to the disparity map processing model and the first image to obtain the second disparity map.

[0010] Secondly, embodiments of this application also provide an image processing apparatus, comprising:

[0011] The first acquisition module is used to acquire a pair of stereo images to be processed, the pair of stereo images including a first image and a second image;

[0012] The first calculation module is used to calculate a first disparity map between the first image and the second image;

[0013] The image processing module is used to process the first disparity map according to the disparity map processing model and the first image to obtain a second disparity map.

[0014] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the image processing method provided in any embodiment of this application.

[0015] Fourthly, embodiments of this application also provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the image processing method provided in any embodiment of this application.

[0016] The technical solution provided in this application embodiment obtains a first disparity map of the first image and the second image in a binocular image pair, and then processes the first disparity map using a disparity map processing model and the first image to obtain a second disparity map of higher quality. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a first flowchart of an image processing method provided in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram illustrating the image processing process according to the disparity map processing model provided in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram illustrating the training process of the disparity map processing model in the image processing method provided in this embodiment of the invention.

[0021] Figure 4 This diagram illustrates the process of optimizing the quality of the third disparity map during the training of the disparity map processing model.

[0022] Figure 5 This diagram illustrates the classification of pixels in an image patch of a human face mask during the training of a disparity map processing model.

[0023] Figure 6 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.

[0024] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] This application provides an image processing method, the execution subject of which can be a computer device. The computer device can be a smartphone, tablet, laptop, or desktop computer, etc. Alternatively, the computer device can also be a server.

[0028] Please see Figure 1 , Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The specific flow of the image processing method provided in this embodiment of the application can be as follows:

[0029] 101. Obtain the stereo image pair to be processed. The stereo image pair includes a first image and a second image.

[0030] The binocular image pair can be obtained by capturing images using a binocular camera, or it can be two images of the same scene captured by two cameras from different perspectives, with some content overlapping in the first and second images. Generally, to facilitate subsequent image matching, the horizontal positions of the two cameras can be set to be the same, so the parallax between the first and second images is the horizontal parallax.

[0031] For example, in one embodiment, the first image is captured by the main camera in the camera module, and the second image is captured by the secondary camera in the camera module. Furthermore, in subsequent processing, the first image captured by the main camera is used as the reference image for calculating the disparity map.

[0032] 102. Calculate the first disparity map between the first image and the second image.

[0033] After acquiring the first and second images, stereo matching is performed on them to calculate the first disparity map of the two images. Stereo matching obtains the corresponding pixel in the other image for each pixel in one image, thus generating the disparity map between the two images.

[0034] For example, in one embodiment, a first disparity map of the first image and the second image is calculated using the SGBM (semi-global block matching) algorithm. The size of the first disparity map is the same as that of the first image and the second image. Alternatively, in other embodiments, the first disparity map can also be calculated using an algorithm such as BM (block matching).

[0035] 103. Process the first disparity map according to the disparity map processing model and the first image to obtain the second disparity map.

[0036] After obtaining the first disparity map, the first disparity map is processed according to the disparity map processing model and the first image provided in the embodiments of this application to obtain the second disparity map.

[0037] The disparity map processing model is trained according to the model training method provided in the embodiments of this application. The trained disparity map processing model can be used to optimize the quality of the disparity map and improve its quality.

[0038] For example, in one embodiment, processing a first disparity map according to a disparity map processing model and a first image to obtain a second disparity map includes: stitching the first image and the first disparity map together in the channel direction to obtain a multi-channel first stitched image; and processing the first stitched image according to the disparity map processing model to obtain the second disparity map.

[0039] In this embodiment, the first image, the first disparity map, and the second disparity map are of the same size. The first image and the first disparity map are stitched together along the channel direction. Each pixel in the first disparity map represents a disparity value, making it a single-channel image. The first image, however, is typically a multi-channel image. For example, if the first image is in RGB format, it is a three-channel image. Stitching the first image and the first disparity map along the channel direction yields a four-channel stitched image. This first stitched image is used as input data to a disparity map processing model. The model processes the first stitched image and outputs a single-channel second disparity map. Compared to the first disparity map, the second disparity map has better quality.

[0040] In one embodiment, processing the first stitched image according to the disparity map processing model to obtain the second disparity map includes: performing convolution processing on the first stitched image according to the encoding module to obtain a multi-scale feature map of the first stitched image; and performing convolution processing and upsampling processing on the multi-scale feature map according to the decoding module to obtain the second disparity map.

[0041] In this embodiment, the disparity map processing model is a U-shaped convolutional neural network model, which includes an encoding module and a decoding module. The encoding module extracts features from the input data and includes multiple convolutional layers. After processing by these layers, a multi-scale feature map is obtained. The decoding module then upsamples the multi-scale features using convolutional kernels to obtain a second disparity map with the same size as the input data.

[0042] It should be noted that the embodiments of this application do not limit the specific results of the U-shaped convolutional neural network model. As long as the network structure conforms to the above processing method, it can be trained to obtain the disparity map processing model of this application.

[0043] To facilitate the reader's understanding of the solution in this application, the following example illustrates a network structure for a disparity map processing model. For instance... Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the image processing process according to the disparity map processing model provided in the embodiments of this application. The numbers below the image indicate the number of channels in the corresponding image or image map. A first stitched image with 4 channels is input into the model. The encoding module performs layer-by-layer convolution processing on the first stitched image to obtain multi-scale feature maps. Here, scale refers to the size on the image plane. Both the first image and the first disparity map are H×W in size. After processing by the encoding module, feature maps with scales of H×W, H / 2×W / 2, H / 4×W / 4, H / 8×W / 8, and H / 16×W / 16 are obtained, as shown below. Figure 2 As shown, these feature maps are processed by convolution and upsampling in the decoding module to obtain feature maps that gradually increase in size, eventually resulting in a feature map with 32+32 channels. Then, convolution and dimensionality reduction are performed on this feature map to obtain a second disparity map with 1 channel.

[0044] In the decoding module, the obtained feature map is concatenated with the feature map of the same size obtained in the encoding module in the channel direction, and then input into the next layer for computation.

[0045] It should be noted that, Figure 2 The processing procedure shown is only an example of a U-shaped convolutional neural network model and does not constitute a limitation on the scheme of this application.

[0046] In practice, this application is not limited by the execution order of the described steps. Without causing conflicts, some steps may be performed in other orders or simultaneously.

[0047] As can be seen from the above, the image processing method provided in this application obtains the first disparity map of the first image and the second image in the binocular image pair, and then processes the first disparity map using the disparity map processing model and the first image to obtain a second disparity map of higher quality.

[0048] Next, the training process of the disparity map processing model will be explained.

[0049] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the training process of the disparity map processing model in the image processing method provided in this embodiment of the invention. The training process includes:

[0050] 201. Obtain the first sample image and the second sample image of the same scene taken from different perspectives.

[0051] The first and second sample images can be obtained by using a binocular camera, or by using two cameras to capture the same scene from different perspectives, resulting in two images with some overlap. Generally, to facilitate subsequent image matching, the horizontal positions of the two cameras can be set to be the same, so the parallax between the first and second sample images is the horizontal parallax.

[0052] 202. Calculate the third disparity map between the first sample image and the second sample image.

[0053] After obtaining the first and second sample images, stereo matching is performed on them to calculate a third disparity map. Stereo matching obtains the corresponding pixel in the other image for each pixel in one image, thus yielding the disparity values ​​of all pixels and forming the third disparity map.

[0054] 203. Process the third disparity map according to the preset image processing algorithm to obtain the fourth disparity map.

[0055] The preset image processing algorithm can improve the quality of the third disparity map, thereby achieving optimized processing of the third disparity map. For example, in one embodiment, the preset image processing algorithm can be at least one of several algorithms, such as filtering algorithm, noise reduction algorithm, and image partial region erosion algorithm.

[0056] Alternatively, in another embodiment, the third disparity map is processed according to a preset image processing algorithm to obtain a fourth disparity map, including: filtering the third disparity map using a least-squares filtering algorithm to obtain a first intermediate disparity map; obtaining a first subject mask of the first sample image and determining the disparity mean of the subject region based on the first intermediate disparity map; performing erosion processing on the first intermediate disparity map to obtain a second intermediate disparity map; using the disparity mean to fill the subject region of the second intermediate disparity map with disparity to obtain a third intermediate disparity map; filtering the non-subject region of the third intermediate disparity map using a joint bilateral filtering algorithm to obtain a fourth intermediate disparity map; and performing noise reduction processing on the fourth intermediate disparity map to obtain a fourth disparity map.

[0057] The following text uses the example of photographing people, so the main area in the image is the portrait area, and the main mask is the portrait mask.

[0058] like Figure 4 As shown, Figure 4 This diagram illustrates the process of quality optimization of the third disparity map during the training of the disparity map processing model. In this embodiment, after obtaining the third disparity map, it is first filtered using a least-squares filtering algorithm to enhance the image, resulting in a first intermediate disparity map. Figure 4 Only the first sample image is shown; the second sample image is not shown.

[0059] This processing may result in edge diffusion and inaccurate human figure edges. Therefore, after obtaining the first intermediate disparity map, it is eroded to obtain the second intermediate disparity map, thus eliminating the influence of disparity diffusion. Furthermore, this embodiment uses a portrait mask to process the first intermediate disparity map, determining the human figure region within it. Then, based on the pixel differences in the first disparity map, the mean disparity value of the human figure region is calculated. This mean disparity value is then used to fill the main body region of the second intermediate disparity map, resulting in the third intermediate disparity map. The third intermediate disparity map obtained through this process exhibits high consistency of the human figure and high edge accuracy.

[0060] Next, the non-subject areas in the third intermediate disparity map are filtered using a joint bilateral filtering algorithm to obtain a smoother fourth intermediate disparity map. Since joint bilateral filtering is based on color and spatial similarity, it may introduce some noise. This noise can be reduced by applying median filtering to the fourth intermediate disparity map. Finally, Gaussian filtering can be applied to the denoised disparity map to eliminate jagged edges on the portrait, resulting in the final fourth disparity map.

[0061] 204. The first sample image and the third disparity map are stitched together in the channel direction to obtain the second stitched image. The second stitched image and the fourth disparity map constitute a sample pair.

[0062] After the above series of processing steps on the third disparity map, a quality-optimized third disparity map is obtained. Next, sample pairs are constructed based on the first sample image, the third disparity map, and the fourth disparity map to train the initial disparity map processing model.

[0063] The model takes the original first sample image and the unoptimized third disparity map as input. For example, the first sample image and the third disparity map are stitched together in the channel direction to obtain a four-channel second stitched image. The second stitched image and the fourth disparity map constitute a set of sample pairs.

[0064] The above text describes how to obtain a set of sample pairs. Multiple sets of sample pairs can be obtained in the same way, for example, 1000 sets of sample pairs can be obtained as training data.

[0065] 205. Train the initial disparity map processing model based on multiple sets of samples to obtain the disparity map processing model.

[0066] After obtaining multiple sets of sample pairs, the initial disparity map processing model is iteratively trained based on these sets of sample pairs. For example, in one embodiment, training the initial disparity map processing model based on multiple sets of sample pairs to obtain the disparity map processing model includes: obtaining target sample pairs from the multiple sets of sample pairs; inputting the second stitched image from the target sample pair into the initial disparity map processing model for processing to obtain a predicted disparity map; calculating the comprehensive loss of the initial disparity map processing model based on the predicted disparity map and the fourth disparity map from the target sample pair; updating the model parameters of the initial disparity map processing model based on the comprehensive loss, and returning to execute the step of obtaining target sample pairs from the multiple sets of sample pairs until the model iterative training is completed.

[0067] In this embodiment, in one iteration, an unused sample pair is selected from multiple sample pairs as the target sample pair. The second stitched image from the target sample pair is then used as input data to the model for processing. After processing, the model outputs a predicted disparity map. The comprehensive loss generated during model training is calculated based on the predicted disparity map and the fourth disparity map from the target sample pair. The model parameters are then updated based on this comprehensive loss. This iterative training is repeated until an iteration termination condition is met. For example, the iteration termination condition could be that the number of iterations reaches a preset number, or the comprehensive loss meets a preset condition. After the iteration is complete, the model parameters obtained from the last iteration are determined, and the initial disparity map processing model with these parameters is identified as the disparity map processing model.

[0068] There are several ways to calculate the overall loss of the initial disparity map processing model during training. For example, in one embodiment, the overall loss of the initial disparity map processing model is calculated based on the predicted disparity map, the fourth disparity map in the target sample pair, and the L1 loss function. Another example is that in one embodiment, the overall loss of the initial disparity map processing model is calculated based on the predicted disparity map, the fourth disparity map in the target sample pair, and the L2 loss function. Yet another example is that in one embodiment, the target sample image in the second stitched image of the target sample pair is obtained, and the second subject mask of the target sample image is obtained; the overall loss of the initial disparity map processing model is calculated based on the predicted disparity map, the fourth disparity map in the target sample pair, the second subject mask, and the depth consistency loss function.

[0069] Alternatively, in another embodiment, the comprehensive loss of the initial disparity map processing model is calculated based on the predicted disparity map and the fourth disparity map in the target sample pair, including: calculating a first loss of the initial disparity map processing model based on the predicted disparity map, the fourth disparity map in the target sample pair, and the L1 loss function; obtaining the target sample image in the second stitched image of the target sample pair, and obtaining a second subject mask of the target sample image; calculating a second loss of the initial disparity map processing model based on the predicted disparity map, the fourth disparity map in the target sample pair, the second subject mask, and the depth consistency loss function; obtaining a multi-channel feature map output by the initial decoding module with the same size as the second subject mask; calculating a third loss of the initial disparity map processing model based on the second subject mask and the multi-channel feature map; and determining the sum of the first loss, the second loss, and the third loss as the comprehensive loss of the initial disparity map processing model.

[0070] In this embodiment, three loss functions are used to calculate the loss value, and the sum of the three loss values ​​is determined as the comprehensive loss during the model training process.

[0071] The first type of loss function is the L1 loss function, and its calculation formula is as follows:

[0072] L d =L1(d pred ,d t arg et )

[0073] Among them, L d d represents the first loss value, and d represents the predicted disparity map output by the model. t arg et This is the fourth disparity map in the target sample pair.

[0074] The second type of loss function is the depth consistency loss function based on the subject mask. Its calculation formula is as follows:

[0075]

[0076] Among them, L consThe second loss value is represented by the mask, where mask is the main body mask. For differentiation operations.

[0077] Next, the third loss function will be explained. In this embodiment, the third loss function adopts the triplet loss function of human image edge.

[0078] Obtain the multi-channel feature map output by the initial decoding module that matches the size of the second main body mask, from Figure 2 It can be seen that the feature map output by the penultimate layer of the decoding module has a size on the plane that is similar to that of the fourth disparity layer. Figure 1 However, the number of channels is 32, meaning that each pixel in this multi-channel feature map has a 32-dimensional vector along the channel direction.

[0079] Next, a third loss of the initial disparity map processing model is calculated based on the second subject mask and the multi-channel feature map. For example, in one embodiment, calculating the third loss of the initial disparity map processing model based on the second subject mask and the multi-channel feature map includes: dividing the multi-channel feature map into multiple image blocks; for each image block, determining the center pixel of the image block, and determining, based on the second subject mask, the same type of pixels in the image block that are of the same category as the center pixel, and the different type of pixels that are of different categories from the center pixel; determining subject edge image blocks from the multiple image blocks based on a first number of the same type of pixels and a second number of the different type of pixels in each image block; for each subject edge image block, calculating a first similarity between the center pixel of the subject edge image block and its same type of pixels, and calculating a second similarity between the center pixel of the subject edge image block and its different type of pixels; and calculating the third loss of the initial disparity map processing model based on the first and second similarities of all subject edge image blocks.

[0080] The multi-channel feature map is divided into multiple image blocks. For each image block, the center pixel is determined, and based on the second main body mask, pixels of the same category as the center pixel and pixels of different categories are identified. For example... Figure 5 As shown, Figure 5 This diagram illustrates pixel classification within an image patch of a human face mask during the training of a disparity map processing model. The second main mask, the human face mask, is divided into H×W image patches of size K×K, with a step size of 1. For each image patch, the center point is denoted as i. Pixels within this image patch that belong to the same category as i (both belonging to the human face region or both belonging to the background region) are denoted as... Quantity is recorded as Pixels within an image patch that belong to a different category from i are denoted as . Quantity is recorded as For pixel i, when and When all values ​​are greater than the threshold T, it indicates that there are pixels of different categories around i, and i can be determined to be located in the segmentation boundary region, for example, T = K-1.

[0081] Based on the above principles, for multi-channel feature maps, the size of the multi-channel feature map on the plane is the same as that of the portrait mask. Therefore, the multi-channel feature map can be segmented in the same way as the portrait mask, into K×K image blocks. Then, the pixels within each image block are classified according to the portrait mask. The pixels within each image block can be divided into three categories: These three categories are the center point of the image patch, the point that belongs to the same category as i in the portrait mask, and the point that does not belong to the same category as i in the portrait mask.

[0082] In the multi-channel feature map, pixels belonging to the same category have higher similarity, while pixels belonging to different categories have lower similarity. The similarity of pixels belonging to the same category is s. + The similarity between pixels belonging to different categories is s. - The calculation method is as follows: cosine similarity calculation:

[0083] First follow The values ​​at each pixel are normalized. Then, the first similarity s is calculated. + Second similarity s - .

[0084]

[0085]

[0086] Where j is The j-th pixel among a set of pixels of the same type, or j is... The j-th pixel among a set of heterogeneous pixels.

[0087] The goal of training is s + The closer to 1, the better. - The closer to 0, the better, so the third function can be designed as follows:

[0088]

[0089] m is a hyperparameter, and the value of m is less than 1 and greater than 0. For example, in one embodiment, m = 0.5.

[0090] Since the solution in this application focuses more on the human face segmentation boundary region, the triplet loss function L can be expressed by the following formula: tm .

[0091]

[0092] Where T is a preset threshold for determining the human image segmentation boundary. For any image block among multiple image blocks, if the first number of similar pixels and the second number of dissimilar pixels in the image block are both greater than the preset threshold T, then the image block is determined as a subject edge image block.

[0093] The final overall loss L of the model can be expressed as:

[0094] L = L d +L cons +L tm .

[0095] The third loss L is calculated based on the above similarity. tm Then, by combining the first loss and the second loss, the sum of the three losses is obtained, and the overall loss of the model is determined.

[0096] The model parameters are updated based on the comprehensive loss, and this iterative training is repeated until the iteration termination condition is met to complete the training of the model and obtain the disparity map processing model.

[0097] In the above model training, the calculated initial third disparity map and the first sample image used as the reference image are stitched together in the channel direction to form a second stitched image, which is used as the input data of the model. The third disparity map is then optimized using an image processing algorithm to obtain a high-quality fourth disparity map. Sample image pairs are formed by the second stitched image and the fourth disparity map. Multiple sets of sample image pairs are used to train the initial disparity map processing model to obtain the disparity map processing model. The trained disparity map processing model can be used to optimize the quality of the disparity map and improve its quality.

[0098] In one embodiment, an image processing apparatus is also provided. See also... Figure 6 , Figure 6 This is a schematic diagram of the structure of an image processing apparatus 300 provided in an embodiment of this application. The image processing apparatus 300 is applied to a computer device and includes:

[0099] The first acquisition module 301 is used to acquire a pair of stereo images to be processed, the pair of stereo images including a first image and a second image;

[0100] The first calculation module 302 is used to calculate a first disparity map between the first image and the second image;

[0101] Image processing module 303 is used to process the first disparity map according to the disparity map processing model and the first image to obtain a second disparity map.

[0102] In some embodiments, the first disparity map, the first image, and the second disparity map are of the same size; the image processing module 303 is used to stitch the first image and the first disparity map in the channel direction to obtain a multi-channel first stitched image; and to process the first stitched image according to the disparity map processing model to obtain a second disparity map.

[0103] In some embodiments, the disparity map processing model is a U-shaped convolutional neural network model, which includes an encoding module and a decoding module; the image processing module 303 is used to perform convolution processing on the first stitched image according to the encoding module to obtain a multi-scale feature map of the first stitched image; and to perform convolution processing and upsampling processing on the multi-scale feature map according to the decoding module to obtain a second disparity map.

[0104] In some embodiments, the first image is captured by the main camera in the camera module, and the second image is captured by the secondary camera in the camera module.

[0105] In some embodiments, the device further includes:

[0106] The model training module is used to acquire first and second sample images of the same scene taken from different perspectives.

[0107] Calculate the third disparity map between the first sample image and the second sample image;

[0108] The third disparity map is processed according to a preset image processing algorithm to obtain a fourth disparity map;

[0109] The first sample image and the third disparity map are stitched together in the channel direction to obtain a second stitched image. The second stitched image and the fourth disparity map constitute a set of sample pairs.

[0110] The initial disparity map processing model is trained based on multiple sets of sample pairs to obtain the disparity map processing model.

[0111] In some embodiments, the model training module is further configured to:

[0112] The third disparity map is filtered using the least squares filtering algorithm to obtain the first intermediate disparity map.

[0113] Obtain the first subject mask of the first sample image, and determine the mean disparity of the subject region based on the first intermediate disparity map;

[0114] The first intermediate disparity map is subjected to erosion processing to obtain the second intermediate disparity map;

[0115] The disparity mean is used to fill the main area of ​​the second intermediate disparity map with disparity to obtain the third intermediate disparity map;

[0116] The non-subject regions in the third intermediate disparity map are filtered using a joint bilateral filtering algorithm to obtain the fourth intermediate disparity map.

[0117] The fourth intermediate disparity map is denoised to obtain the fourth disparity map.

[0118] In some embodiments, the model training module is further configured to:

[0119] Obtain the target sample pair from the multiple sample pairs;

[0120] The second stitched image from the target sample pair is input into the initial disparity map processing model for processing to obtain the predicted disparity map.

[0121] The comprehensive loss of the initial disparity map processing model is calculated based on the predicted disparity map and the fourth disparity map in the target sample pair;

[0122] The model parameters of the initial disparity map processing model are updated according to the comprehensive loss, and the step of obtaining the target sample pair from the multiple sample pairs is returned until the model iterative training is completed.

[0123] In some embodiments, the initial disparity map processing model includes an initial encoding module and an initial decoding module; the model training module is further configured to:

[0124] In some embodiments, the model training module is further configured to:

[0125] The first loss of the initial disparity map processing model is calculated based on the predicted disparity map, the fourth disparity map in the target sample pair, and the L1 loss function.

[0126] Obtain the target sample image from the second stitched image of the target sample pair, and obtain the second main body mask of the target sample image;

[0127] The second loss of the initial disparity map processing model is calculated based on the predicted disparity map, the fourth disparity map in the target sample pair, the second subject mask, and the depth consistency loss function.

[0128] Obtain the multi-channel feature map output by the initial decoding module that is consistent with the size of the second main body mask;

[0129] The third loss of the initial disparity map processing model is calculated based on the second main body mask and the multi-channel feature map;

[0130] The sum of the first loss, the second loss, and the third loss is determined as the comprehensive loss of the initial disparity map processing model.

[0131] In some embodiments, the model training module is further configured to:

[0132] The multi-channel feature map is divided into multiple image blocks;

[0133] For each image block, the center pixel of the image block is determined, and according to the second main body mask, the same type of pixels in the image block that are the same as the center pixel and the different type of pixels that are different from the center pixel are determined;

[0134] Based on the first number of similar pixels and the second number of dissimilar pixels in each image block, a subject edge image block is determined from the plurality of image blocks;

[0135] For each subject edge image block, calculate the first similarity between the center pixel of the subject edge image block and its own type of pixel, and calculate the second similarity between the center pixel of the subject edge image block and its dissimilar type of pixel;

[0136] The third loss of the initial disparity map processing model is calculated based on the first similarity and the second similarity of all the subject edge image patches.

[0137] In some embodiments, the model training module is further configured to:

[0138] For any image block among the plurality of image blocks, if the first number of the same type of pixels and the second number of different type of pixels in the image block are both greater than a preset threshold, then the image block is determined as a subject edge image block.

[0139] As can be seen from the above, the image processing apparatus proposed in this application obtains the first disparity map of the first image and the second image in the binocular image pair, and then processes the first disparity map using the disparity map processing model and the first image to obtain a second disparity map of higher quality.

[0140] This application also provides a computer device, which can be a terminal, such as a smartphone, tablet, or laptop. Please refer to [link to relevant documentation]. Figure 7 , Figure 7This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 500 includes a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, and a computer program stored on the memory 502 and executable on the processor. The processor 501 and the memory 502 are electrically connected. Those skilled in the art will understand that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0141] The processor 501 is the control center of the computer device 500. It connects various parts of the computer device 500 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, it performs various functions of the computer device 500 and processes data, thereby monitoring the computer device 500 as a whole.

[0142] In this embodiment, the processor 501 in the computer device 500 loads the instructions corresponding to the processes of one or more applications into the memory 502 according to the following steps, and the processor 501 runs the applications stored in the memory 502 to perform the following steps:

[0143] Obtain a pair of stereo images to be processed, the pair of stereo images including a first image and a second image;

[0144] Calculate the first disparity map between the first image and the second image;

[0145] The first disparity map is processed according to the disparity map processing model and the first image to obtain the second disparity map.

[0146] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0147] Optional, such as Figure 7 As shown, the computer device 500 also includes: a touch screen display 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch screen display 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507. Those skilled in the art will understand that... Figure 7 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0148] The touch display screen 503 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 503 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the computer device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), and generate corresponding operation commands, which then execute the corresponding program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 501, and can receive and execute commands from the processor 501.

[0149] The radio frequency circuit 504 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0150] Audio circuitry 505 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 505 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 505, converted back into audio data, and output to processor 501 for processing. The audio data is then transmitted via radio frequency circuitry 504 to, for example, another computer device, or output to memory 502 for further processing. Audio circuitry 505 may also include an earphone jack to facilitate communication between peripheral headphones and the computer device.

[0151] The input unit 506 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0152] Power supply 507 is used to supply power to various components of computer device 500. Optionally, power supply 507 can be logically connected to processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 507 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0153] although Figure 7 As not shown in the diagram, the computer device 500 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0154] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0155] As can be seen from the above, the computer device provided in this embodiment obtains the first disparity map of the first image and the second image in the binocular image pair, and then processes the first disparity map using the disparity map processing model and the first image to obtain a second disparity map of higher quality.

[0156] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0157] Therefore, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the image processing methods provided in embodiments of this application.

[0158] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0159] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0160] Since the computer program stored in the storage medium can execute the steps of any of the image processing methods provided in the embodiments of this application, the beneficial effects that any of the image processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0161] The foregoing has provided a detailed description of an image processing method, apparatus, device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: Obtain a pair of stereo images to be processed, the pair of stereo images including a first image and a second image; Calculate the first disparity map between the first image and the second image; The first disparity map is processed according to the disparity map processing model and the first image to obtain the second disparity map; The step of processing the first disparity map according to the disparity map processing model and the first image to obtain the second disparity map includes: stitching the first image and the first disparity map in the channel direction to obtain a multi-channel first stitched image; and processing the first stitched image according to the disparity map processing model to obtain the second disparity map. The method further includes: acquiring a first sample image and a second sample image of the same scene captured from different perspectives; calculating a third disparity map between the first sample image and the second sample image; processing the third disparity map according to a preset image processing algorithm to obtain a fourth disparity map; stitching the first sample image and the third disparity map together in the channel direction to obtain a second stitched image, wherein the second stitched image and the fourth disparity map constitute a set of sample pairs; and training an initial disparity map processing model based on multiple sets of sample pairs to obtain a disparity map processing model. The step of processing the third disparity map according to a preset image processing algorithm to obtain a fourth disparity map includes: filtering the third disparity map using a least-squares filtering algorithm to obtain a first intermediate disparity map; obtaining a first subject mask of the first sample image and determining the disparity mean of the subject region based on the first intermediate disparity map; performing erosion processing on the first intermediate disparity map to obtain a second intermediate disparity map; using the disparity mean to perform disparity filling on the subject region of the second intermediate disparity map to obtain a third intermediate disparity map; filtering the non-subject region of the third intermediate disparity map using a joint bilateral filtering algorithm to obtain a fourth intermediate disparity map; and performing noise reduction processing on the fourth intermediate disparity map to obtain a fourth disparity map.

2. The method as described in claim 1, characterized in that, The first disparity map, the first image, and the second disparity map are all the same size.

3. The method as described in claim 2, characterized in that, The disparity map processing model is a neural network model, which includes an encoding module and a decoding module; the step of processing the first stitched image according to the disparity map processing model to obtain a second disparity map includes: The first stitched image is convolved according to the encoding module to obtain a multi-scale feature map of the first stitched image; The decoding module performs convolution and upsampling processing on the multi-scale feature map to obtain a second disparity map.

4. The method according to any one of claims 1 to 3, characterized in that, The first image was captured by the main camera in the camera module, and the second image was captured by the secondary camera in the camera module.

5. The method as described in claim 1, characterized in that, The step of training the initial disparity map processing model based on multiple sets of sample pairs to obtain the disparity map processing model includes: Obtain the target sample pair from the multiple sample pairs; The second stitched image from the target sample pair is input into the initial disparity map processing model for processing to obtain the predicted disparity map. The comprehensive loss of the initial disparity map processing model is calculated based on the predicted disparity map and the fourth disparity map in the target sample pair; The model parameters of the initial disparity map processing model are updated according to the comprehensive loss, and the step of obtaining the target sample pair from the multiple sample pairs is returned until the model iterative training is completed.

6. The method as described in claim 5, characterized in that, The initial disparity map processing model includes an initial encoding module and an initial decoding module; The step of calculating the comprehensive loss of the initial disparity map processing model based on the predicted disparity map and the fourth disparity map in the target sample pair includes: The first loss of the initial disparity map processing model is calculated based on the predicted disparity map, the fourth disparity map in the target sample pair, and the L1 loss function. Obtain the target sample image from the second stitched image of the target sample pair, and obtain the second main body mask of the target sample image; The second loss of the initial disparity map processing model is calculated based on the predicted disparity map, the fourth disparity map in the target sample pair, the second subject mask, and the depth consistency loss function. Obtain the multi-channel feature map output by the initial decoding module that is consistent with the size of the second main body mask; The third loss of the initial disparity map processing model is calculated based on the second main body mask and the multi-channel feature map; The sum of the first loss, the second loss, and the third loss is determined as the comprehensive loss of the initial disparity map processing model.

7. The method as described in claim 6, characterized in that, The calculation of the third loss of the initial disparity map processing model based on the second main mask and the multi-channel feature map includes: The multi-channel feature map is divided into multiple image blocks; For each image block, the center pixel of the image block is determined, and according to the second main body mask, the same type of pixels in the image block that are the same as the center pixel and the different type of pixels that are different from the center pixel are determined; Based on the first number of similar pixels and the second number of dissimilar pixels in each image block, a subject edge image block is determined from the plurality of image blocks; For each subject edge image block, calculate the first similarity between the center pixel of the subject edge image block and its own type of pixel, and calculate the second similarity between the center pixel of the subject edge image block and its dissimilar type of pixel; The third loss of the initial disparity map processing model is calculated based on the first similarity and the second similarity of all the subject edge image patches.

8. The method as described in claim 7, characterized in that, The step of determining the subject edge image block from the plurality of image blocks based on a first number of similar pixels and a second number of dissimilar pixels in each image block includes: For any image block among the plurality of image blocks, if the first number of the same type of pixels and the second number of different type of pixels in the image block are both greater than a preset threshold, then the image block is determined as a subject edge image block.

9. An image processing apparatus, characterized in that, include: The first acquisition module is used to acquire a pair of stereo images to be processed, the pair of stereo images including a first image and a second image; The first calculation module is used to calculate a first disparity map between the first image and the second image; The image processing module is used to process the first disparity map according to the disparity map processing model and the first image to obtain a second disparity map; The image processing module processes the first disparity map according to the disparity map processing model and the first image to obtain a second disparity map, including: stitching the first image and the first disparity map in the channel direction to obtain a multi-channel first stitched image; and processing the first stitched image according to the disparity map processing model to obtain the second disparity map. The image processing device is further configured to: acquire a first sample image and a second sample image of the same scene captured from different perspectives; calculate a third disparity map between the first sample image and the second sample image; process the third disparity map according to a preset image processing algorithm to obtain a fourth disparity map; stitch the first sample image and the third disparity map together in the channel direction to obtain a second stitched image, wherein the second stitched image and the fourth disparity map constitute a set of sample pairs; and train an initial disparity map processing model based on multiple sets of sample pairs to obtain a disparity map processing model. The image processing device processes the third disparity map according to a preset image processing algorithm to obtain a fourth disparity map, including: filtering the third disparity map using a least-squares filtering algorithm to obtain a first intermediate disparity map; obtaining a first subject mask of the first sample image and determining the disparity mean of the subject region based on the first intermediate disparity map; performing erosion processing on the first intermediate disparity map to obtain a second intermediate disparity map; using the disparity mean to perform disparity filling on the subject region of the second intermediate disparity map to obtain a third intermediate disparity map; filtering the non-subject region of the third intermediate disparity map using a joint bilateral filtering algorithm to obtain a fourth intermediate disparity map; and performing noise reduction processing on the fourth intermediate disparity map to obtain a fourth disparity map.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 8.

11. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the image processing method as described in any one of claims 1 to 8.

Images