An image processing method, apparatus, electronic device, and storage medium

Abstract

This application discloses an image processing method, apparatus, electronic device, and storage medium. The method includes: obtaining a depth map of a current frame image using a first model; determining the depth value of a focal plane based on the depth value corresponding to the main body region in the depth map of the current frame image; determining a bokeh map based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focal plane; and performing bokeh processing on the current frame image using the bokeh map. The method provided by this application improves accuracy and stability while simulating a more natural gradient bokeh effect from a DSLR lens based on the distance between people and scene objects.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Background blurring refers to blurring the background of an image to clearly display the foreground. It's a widely used photographic technique that effectively highlights the subject and expresses visual aesthetics. Some professional cameras can directly capture images with a blurred background, but this requires a high-resolution lens with a large aperture and zoom capability. For cameras on electronic devices such as smartphones or tablets, due to limitations in size and thickness, their lens performance cannot match that of professional cameras, so they cannot directly capture images with a blurred background. Therefore, image processing software is needed to achieve background blurring.

[0003] Several methods for digital image blurring have been proposed in the existing technology, such as object space-based depth-of-field rendering methods and image space-based depth-of-field rendering methods. Although object space-based depth-of-field rendering methods produce quite realistic results, they are computationally intensive and memory-intensive, making it difficult to obtain results quickly and directly. Image space-based depth-of-field rendering methods are computationally fast, but the blurred background image may exhibit problems such as blurred boundaries between the portrait and the background and unnatural transitions. Summary of the Invention

[0004] To address any of the aforementioned technical problems, embodiments of this application provide an image processing method, apparatus, electronic device, and storage medium.

[0005] To achieve the objectives of the embodiments of this application, the embodiments of this application provide an image processing method, including:

[0006] The depth map of the current frame image is obtained using the first model;

[0007] The depth value of the focus plane is determined based on the depth value corresponding to the main body region in the depth map of the current frame image.

[0008] A bokeh map is determined based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane.

[0009] The current frame image is blurred using the blur mapping map.

[0010] An image processing apparatus, comprising:

[0011] The acquisition module is set to obtain the depth map using the first model;

[0012] The first determining module is configured to determine the depth value of the focal plane based on the depth value corresponding to the main body region in the depth map of the current frame image.

[0013] The second determining module is configured to determine the bokeh map based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane.

[0014] The processing module is configured to perform blurring processing on the current frame image using the blurring mapping map.

[0015] A storage medium storing a computer program, wherein the computer program is configured to execute the method described above when run.

[0016] An electronic device includes a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the methods described above.

[0017] One of the above technical solutions has the following advantages or beneficial effects:

[0018] The depth map of the current frame image is obtained through the first model. Based on the depth value corresponding to the main body region in the depth map of the current frame image, the depth value of the focus plane is determined. Based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane, the region of the current frame image is clearly divided into the main body region and the non-main body region. A blur map is determined to achieve the purpose of blurring only the non-main body region. Then, the blur map is used to blur the current frame image, ensuring the privacy requirement of background blurring in the non-main body region and ensuring the use requirement of clear image in the main body region, thus taking into account the dual needs of privacy security and communication use.

[0019] Other features and advantages of the embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the embodiments of this application. The objects and other advantages of the embodiments of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0020] The accompanying drawings are used to provide a further understanding of the technical solutions of the embodiments of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0021] Figure 1 A schematic flowchart of the image processing method provided in the embodiments of this application;

[0022] Figure 2 A schematic diagram showing the distribution of the number of pixels corresponding to the depth values ​​provided in the embodiments of this application;

[0023] Figure 3 This is another schematic flowchart of the image processing method provided in the embodiments of this application;

[0024] Figure 4 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0026] Figure 1 This is a schematic flowchart illustrating the image processing method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0027] Step 10: Obtain the depth map of the current frame image using the first model;

[0028] The first model is a deep learning model used to output a corresponding depth map for the input image. Using the first model to generate the depth map of the current frame image can effectively improve the efficiency of depth map generation.

[0029] Step 20: Determine the depth value of the focal plane based on the depth value corresponding to the main body region in the depth map of the current frame image;

[0030] The main area can be a face area, a human body area, or an object (such as an animal or a piece of furniture).

[0031] The main body region can be obtained through semantic segmentation or instance segmentation. Semantic segmentation can be implemented based on face segmentation, portrait segmentation, or pet segmentation, etc.

[0032] Alternatively, the main area can be obtained through target detection technology, such as face detection, head detection, human body detection, pet detection, and target detection of other objects. The working principle of pet detection is similar to that of human detection, the difference being the different objects.

[0033] The depth value of the focal plane can be determined by reasonably selecting the depth value of the main body region. For example, after obtaining the mask of the main body region through semantic segmentation or instance segmentation, a depth value can be selected from the depth map region corresponding to the mask as the depth value of the focal plane; or, after the target detection technology determines the target detection box corresponding to the main body region, a depth value can be selected from the depth map region corresponding to the target detection box as the depth value of the focal plane.

[0034] Of course, in addition to determining the depth value of the focal plane in one method, you can also set a fixed value to be the depth value of the focal plane of the current frame image.

[0035] Step 30: Determine the bokeh map based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane;

[0036] Specifically, each pixel is identified as either in the main subject region or a non-main subject region, thereby determining all pixels that need background blurring. The non-main subject region is then obtained, and a blurring map corresponding to the non-main subject region is generated as the image required for background blurring of the main subject region.

[0037] By clearly dividing the current frame image into a main body region and a non-main body region, a blurring map is determined to achieve the goal of blurring only the non-main body region. Then, the current frame image is blurred using the blurring map, which ensures the privacy requirement of background blurring in the non-main body region and the usage requirement of clear image in the main body region. This takes into account both the dual needs of privacy and security and communication use, highlighting the main body effect and providing a better visual effect.

[0038] For example, taking video conferencing as an application scenario, privacy protection is an important user requirement in video conferencing. Privacy information includes other people in the background, the location in the background, etc., which need to be hidden in the video conference. Therefore, the method provided in the embodiments of this application can be used to blur the background of non-subject areas to achieve the purpose of ensuring privacy. Moreover, since the method provided in the embodiments of this application does not blur the subject area, it ensures that the image of the subject area in the video conference is clear and that the video conference can proceed normally.

[0039] Furthermore, with the increasing prevalence of mobile photography and video production, and the advancement of technology, many professional-level photography techniques can now be achieved using mobile phones. One such technique is portrait video blurring, also known as "movie mode." Movie mode utilizes software algorithms to simulate a blurred background, thereby focusing the viewer's attention on the main part of the video. Applying the technology provided in this application's embodiments to this scenario can highlight the subject, such as a person or object, while blurring the background. This makes the video more professional and cinematic, enhancing its visual appeal.

[0040] Step 40: Use the bokeh mapping map to blur the current frame image;

[0041] Specifically, the bokeh map can be fused with the current frame image to obtain the bokeh processing result.

[0042] The method provided in this application embodiment obtains the depth map of the current frame image through a first model, and determines the depth value of the focus plane based on the depth value corresponding to the main body region in the depth map of the current frame image. Based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane, the region of the current frame image is clearly divided into the main body region and the non-main body region, and a blur map is determined to achieve the purpose of blurring only the non-main body region. Then, the blur map is used to blur the current frame image, ensuring the privacy requirement of background blurring in the non-main body region and ensuring the use requirement of clear image in the main body region, thus taking into account the dual needs of privacy and security and communication use.

[0043] The method provided in the embodiments of this application is described below:

[0044] In step 10, the first model is typically deployed on electronic devices such as mobile phones or tablets, so the deployment scale of the first model is relatively small.

[0045] With the development of deep learning and deep neural network technologies, more and more evidence shows that larger-scale training data can achieve higher accuracy results. Therefore, the accuracy and stability of the first model deployed on a smaller scale urgently need to be improved.

[0046] Furthermore, the diverse sources of the raw training data used in the first model—such as LiDAR, structured light, binocular stereo vision, and multi-view technology—lead to significant differences in the characteristics of the sample data collected from different sources, primarily in depth map range and texture features. Since models deployed on electronic devices like smartphones or tablets generally have fewer parameters and limited capabilities, they cannot overcome the unevenness of samples caused by different data sources. Therefore, this further deteriorates the accuracy and stability of the first model.

[0047] To address the above problems, this application proposes the following solutions:

[0048] The training data used by the first model is the training data obtained after being cleaned and processed by the second model.

[0049] The second model can be a model generated by a cloud server.

[0050] The second model cleans the original training data, ensuring that the amount of original training data is not reduced, while making the distribution of the cleaned training data more uniform. Therefore, when using the cleaned training data to build the first model, the accuracy and stability of the depth map output by the first model can be improved.

[0051] Specifically, the first model is obtained through the following methods:

[0052] The second model is obtained by training the original training data using a scale-invariant loss function as the loss function.

[0053] The original training data is input into the second model to obtain the predicted depth value. The predicted depth value is used as the new label in the original training data to obtain the training data with updated label.

[0054] The first model is obtained by training the training data with the updated labels.

[0055] In the above implementation, the original training data is the original image. Since the original image records the most original feature values ​​of the image itself, the accuracy of the second model can be effectively improved by using the original training data to train and build the second model.

[0056] Furthermore, since the original training data may come from different data sources, using a scale-invariant loss function as the loss function can better process the original training data from multiple data sources and complete the training operation of the second model.

[0057] In the above implementation, the second model is used to determine the predicted depth value of the original training data, and the predicted depth value is used as the new label in the original training data. This weakens the differences in sample distribution from different sources, making the overall performance of the training data after label update more consistent. Therefore, the accuracy and stability of the first model obtained by using the training data after label update can be improved.

[0058] In step 10, the depth map of the current frame image is obtained through the first model, including:

[0059] Step 11: Input the current frame image into the first model to perform a depth map generation operation to obtain an initial depth map;

[0060] The initial depth map is not the final depth map used in the current frame image. It needs to be fused with the depth map of the previous frame image before it can be used as the final depth map for the current frame image.

[0061] Step 12: Perform image fusion on the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image;

[0062] Specifically, to address the issue of high-frequency flickering between consecutive frames, fusing the depth map of each frame with the depth map of the previous frame effectively overcomes this problem. Furthermore, to address the issue of low-frequency flickering between dozens or even more frames, fusing the depth map of each frame with the depth map of the previous frame ensures that each frame is obtained through depth map fusion. This indicates that the depth map of the previous frame has incorporated the depth maps of several previous frames, thus effectively overcoming the low-frequency flickering problem across multiple frames.

[0063] In step 12, image fusion is performed on the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image, including:

[0064] For each pixel of the current frame image, the following operations are performed to obtain the depth value of each pixel in the depth map of the current frame image, including:

[0065] For the current pixel, calculate the product of the depth value in the initial depth map and the first weight to obtain a first result; and calculate the product of the depth value in the depth map of the previous frame and the second weight to obtain a second result; based on the first result and the second result, obtain the depth value of the current pixel.

[0066] In the above implementation, the depth values ​​of each pixel in the initial depth map of the previous frame and the depth map of the previous frame are weighted and summed to obtain the depth value of each pixel in the depth map of the current frame. This allows the depth values ​​of each pixel to be fused, which can effectively improve the stability of the depth map between frames.

[0067] Furthermore, before performing the image fusion operation of the depth map, pixel alignment is performed using the optical flow of the current frame image and the optical flow of the previous frame image. After completing the pixel alignment operation between the current frame image and the previous frame image, the image fusion operation is then performed to improve the accuracy of the image fusion result.

[0068] During video recording, especially with handheld cameras (such as mobile phones), motion blur is common when the camera moves, particularly in low-light conditions like indoors or at night. Therefore, overcoming the negative effects of motion blur is a problem that needs to be solved. This application proposes the following solutions:

[0069] Depending on whether the current frame image is blurry, different methods are used to determine the value of the first weight, including:

[0070] Determine whether the current frame image meets the criteria for image blurring;

[0071] Specifically, an image blur detection model can be preset to detect whether the input image meets the judgment conditions for image blur.

[0072] When the current frame image does not meet the judgment condition of image blur, the difference between the pixels between the current frame image and the previous frame image is calculated to obtain the current pixel difference information, and a value positively correlated with the current pixel difference information is determined as the first weight.

[0073] The current pixel difference information can be determined based on the difference between the optical flow of the current frame image and the optical flow of the previous frame image.

[0074] Specifically, a pre-defined correspondence between pixel difference information and a first weight can be established. Based on this correspondence, the value corresponding to the current pixel difference information is determined as the first weight. This correspondence can be the first weight corresponding to different ranges of pixel difference information values; or it can be a pre-defined calculation expression that uses the pixel difference information as the independent variable and the first weight as the dependent variable.

[0075] When the current frame image meets the judgment condition of image blur, the preset default value is set to the first weight;

[0076] The default value is usually less than 0.5, so that the second weight corresponding to the depth map of the previous frame image is greater than the first weight corresponding to the initial depth map of the current frame image, in order to suppress the negative impact of motion blur in the current frame image.

[0077] Based on whether the current frame image is blurred, an appropriate method is selected to determine the corresponding value for the first weight, which can provide support for depth map fusion operations and effectively suppress the negative impact of motion blur on depth map fusion operations.

[0078] Preferably, in the application scenario of the solution in this application embodiment, since motion blur is mainly manifested between consecutive frames, the change information of the gradient sum between consecutive frames can be used as the judgment condition for image blur.

[0079] Specifically, the criteria for determining if an image is blurry include:

[0080] The gradient sum of the current frame image is less than the gradient sum of the previous frame image, and the absolute value of the difference between the gradient sum of the current frame image and the gradient sum of the previous frame image is greater than a preset gradient sum threshold.

[0081] Since the content of images in consecutive frames of a video is highly similar, the sum of the gradients of consecutive frames is also similar. Therefore, if the sum of the gradients of a certain frame is much smaller than that of the previous frame, it indicates that motion blur exists. This method has low computational complexity and high accuracy.

[0082] During filming, the subject may move. For example, a person or pet's head may turn (e.g., tilting down, turning back, turning left, or turning right), or a person or pet's body parts may move spatially (e.g., limbs may move or bend), or electronic products with mobility functions may move spatially (e.g., a robot vacuum cleaner or a toy electric car may move). As the subject moves, the camera adapts to track it, resulting in a change in the focal plane.

[0083] Taking portrait scenes as an example, when using faces and heads for detection and tracking, if the head can be continuously tracked, it indicates that the differences in the scene and the person between consecutive frames are relatively small, and the difference in the depth value of the focus plane between those frames is also small, resulting in a stable image. When a face or head is obscured or disappears (e.g., moves out of the shooting range, or a frontal view turns to a back view), detection and tracking may fail, requiring a switch to human detection. Alternatively, the shooting scene may change due to the subject being replaced by another subject (e.g., another person's face frame is larger than the original subject's face frame). When the detection method or shooting scene changes, the focus plane needs to be switched to adapt to these changes.

[0084] When the focus plane changes, it is necessary to ensure a smooth transition between the focus planes of the preceding and following frames to maintain the continuity of the background blur image and make the background blur effect between the preceding and following frames transition naturally. Therefore, embodiments of this application provide the following solutions, including:

[0085] In step 20, the depth value of the focus plane is determined based on the depth value corresponding to the main body region in the depth map of the current frame image, including:

[0086] Step 21: Based on the main body region, determine the depth value of the focus plane of the current frame image in the depth map of the current frame image, and use it as the initial focus plane depth value.

[0087] The initial focus plane depth value is not the final depth value of the focus plane in the current frame image. It needs to be fused with the depth value of the focus plane in the previous frame image before it can be used as the final depth of the focus plane in the current frame image.

[0088] Step 22: Based on the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image, determine whether the focus plane of the current frame image has changed.

[0089] Specifically, when subject region tracking is successful, due to the continuity between consecutive frames, the initial focus plane depth value is relatively close to the focus plane depth value of the previous frame. Only when the focus changes will the difference between the initial focus plane depth value and the focus plane depth value of the previous frame increase. Therefore, based on the above characteristics, the magnitude of the difference between the initial focus plane depth value and the focus plane depth value of the previous frame can be used to determine whether the focus plane of the current frame has changed.

[0090] Step 23: If the focus plane of the current frame image changes, update the initial focus plane depth value and use the updated initial focus plane depth value as the depth value of the focus plane; otherwise, determine the initial focus plane depth value as the depth value of the focus plane.

[0091] Specifically, when the focus plane of the current frame changes, updating the initial focus plane depth value ensures a smooth transition between the focus planes of the preceding and following frames, guaranteeing the continuity of the background blur and resulting in a natural transition of the background blur effect between frames. When the focus plane of the current frame remains unchanged, there is no need to update the initial focus plane depth value; it can be directly used as the depth value of the focus plane.

[0092] By determining whether the focus plane of the current frame image has changed, and then selecting the corresponding method to determine the depth value of the focus plane of the current frame image, the stability of the focus plane between consecutive frames can be improved.

[0093] In step 21, based on the main body region, the depth value of the focus plane of the current frame image is determined in the depth map of the current frame image as the initial focus plane depth value, including:

[0094] The depth map region corresponding to the main body region is selected from the depth map of the current frame image to form the main body region depth map;

[0095] Specifically, by using existing methods for detecting the main body region, the pixel information of the main body region in the current frame image can be determined. By determining the region corresponding to the pixel information in the depth map, the main body depth map region can be obtained.

[0096] Determine the number of pixels corresponding to different depth values ​​in the depth map of the main body region;

[0097] Specifically, the range of depth values ​​in the depth map of the main body region is statistically analyzed. Taking the range of depth values ​​as including A values ​​as an example, the depth values ​​are classified according to their magnitude to determine the number of pixels corresponding to each depth value in the depth map of the main body region. That is, A depth values ​​correspond to A categories, and the number of pixels corresponding to each category is determined, where A is an integer greater than or equal to 2.

[0098] Starting from a preset depth value, the cumulative number of pixels corresponding to different depth ranges is counted according to a preset direction.

[0099] The preset depth value is less than or equal to the maximum depth value in the depth map of the main area, and greater than or equal to the minimum depth value in the depth map of the main area.

[0100] The preset direction is the direction of depth value change, which can be along the direction of decreasing preset depth value or along the direction of increasing preset depth value. In actual use, the appropriate direction can be selected according to the magnitude of the selected preset depth value.

[0101] Specifically, based on the direction of depth value change, starting from a preset depth value, the total number of pixels corresponding to the depth values ​​in the depth map of the main area is accumulated to obtain the cumulative number of pixels.

[0102] If the maximum value or the nearest maximum value in the depth map of the main area is used as the preset depth value, the number of pixels can be accumulated in the direction of decreasing preset depth value; if the minimum value or the nearest minimum value in the depth map of the main area is used as the preset depth value, the number of pixels can be accumulated in the direction of increasing preset depth value.

[0103] When the cumulative number of pixels accounts for a predetermined first ratio of the total number of pixels in the depth map of the main area, the depth value of the other endpoint of the depth range that is different from the starting point is taken as the depth value corresponding to the initial focus plane depth value.

[0104] The first ratio is used to measure whether the cumulative number of pixels can cover most of the pixels in the main area.

[0105] For example, if the maximum value of the depth in the depth map of the main area is taken as the preset depth value, the number of pixels can be accumulated in the direction of decreasing depth value. When the depth range is [max, a1], the ratio has not reached the first ratio, so the number of pixels continues to be accumulated in the direction of decreasing depth value. When the depth range is [max, a2], the ratio reaches the first ratio, so the value a2 is taken as the initial depth value of the focus plane, where a1 is greater than a2.

[0106] The above method can accurately determine the initial focus plane depth value of the current frame image.

[0107] by Figure 2 Taking the distribution map shown as an example, the horizontal axis represents the depth value, the vertical axis represents the number of pixels, and the curve represents the correspondence between the depth value and the number of pixels in the depth map of a certain main area.

[0108] like Figure 2 As shown, the shaded area is composed of a curve, a scale threshold line, an abscissa, and a ordinate. The position of the scale threshold line indicates that the cumulative number of pixels accounts for the proportion of the total number of pixels in the depth map of the main area. This proportion is a preset first ratio. In other words, the abscissa of the scale threshold line is used as the initial depth value of the focus plane.

[0109] In step 22, based on the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image, it is determined whether the focus plane of the current frame image has changed, including:

[0110] The difference between the initial focus plane depth value and the focus plane depth value of the previous frame image is calculated to obtain the focus depth difference.

[0111] If the absolute value of the focus depth difference is greater than the preset focus depth threshold, it indicates that the captured image is unstable, and it is determined that the focus plane of the current frame image has changed; otherwise, it indicates that the captured image is stable, and it is determined that the focus plane of the current frame image has not changed.

[0112] By utilizing the relationship between the absolute value of the focus depth difference and the focus depth threshold, it is possible to determine whether the focus plane of the current frame image has changed, which is simple and convenient.

[0113] In step 23, updating the initial focus plane depth value to obtain an updated initial focus plane depth value includes:

[0114] The initial focus plane depth value is smoothed using the depth value of the focus plane in the previous frame image to obtain an updated initial focus plane depth value.

[0115] Specifically, by using the depth value of the focal plane of the previous frame image to smooth the depth value of the initial focal plane, the change in the depth value of the focal plane between the previous and next frames can be kept within a reasonable range, thereby achieving a smooth transition of the focal plane between the previous and next frames, ensuring the stability of the image between the previous and next frames, and reducing the probability of image instability.

[0116] Preferably, the step of smoothing the initial focus plane depth value using the depth value of the focus plane from the previous frame image to obtain an updated initial focus plane depth value includes:

[0117] The difference between the initial focus plane depth value and the focus plane depth value of the previous frame image is calculated to obtain the focus depth difference value m; wherein, the focus depth difference value m is used to measure the degree of change in the depth of the focus plane.

[0118] Based on the focus depth difference m and the preset number of frames n, calculate the depth value adjustment o for each frame; where the depth value adjustment o for each frame can be the ratio of the focus depth value m to the number of frames n, where n is an integer greater than or equal to 2.

[0119] Specifically, the focus depth difference is compensated using n image frames, including the current frame and subsequent consecutive image frames, with each compensation being a depth value adjustment, thus achieving a smooth switching of the focus plane.

[0120] Compensation for the focus depth difference begins as soon as the current frame image where the focus plane change occurs, enabling rapid handling of the negative impact caused by focus plane switching. Furthermore, batch compensation across multiple image frames ensures smooth focus plane switching between consecutive frames.

[0121] The new depth value is obtained by adjusting the depth value of each frame and the depth value of the focal plane of the previous frame image.

[0122] In existing technologies, after a change in the focus plane, the initial focus plane depth value is directly used as the focus plane depth value of the current frame image. Since the focus plane depth value changes significantly, it can cause instability between consecutive frames. Therefore, this application proposes to set the focus plane depth value of the current frame image to the value of the focus plane depth value of the adjacent previous frame image to ensure that the change in focus plane depth value between consecutive frames is within a reasonable range, thus ensuring image stability. Then, the focus depth difference is compensated through n-1 consecutive image frames after the current frame image, achieving batch compensation and thus ensuring the stability of consecutive frames.

[0123] For example, if the depth value of the focus plane in frame N is z0, and the focus plane changes in frame N+1, where the initial depth value of the focus plane in frame N+1 is z0+m, then the depth value z of the focus plane in frame N+1 cannot be set to the initial depth value z1. Instead, the depth value z of the focus plane in the current frame is set to a value that is closer to z0.

[0124] The expression for calculating the depth value z of the focal plane in the current frame is as follows:

[0125]

[0126] In the process of compensating for the focus depth difference through a total of n image frames, including the current frame and subsequent consecutive image frames, if both the camera and the subject are relatively stationary, it takes n frames to complete the focus switching; if the camera or the subject has corresponding movement, the depth value of each frame can be adjusted during each compensation operation to control the compensation operation to be completed in more than or less than n image frames.

[0127] In one exemplary embodiment, n takes the value 10.

[0128] In step 30, a bokeh map is determined based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane, including:

[0129] Step 31: Filter all pixels that meet the first condition in the depth map of the current frame image, wherein all pixels form a non-subject area; wherein the first condition is that the absolute value of the difference between the depth value of a pixel in the depth map of the current frame image and the depth value of the focal plane is greater than or equal to a preset depth value threshold.

[0130] Determining non-subject areas using pixels as the unit can improve the accuracy of non-subject area identification.

[0131] Specifically, since the main subject is generally not a pure plane, it has a corresponding depth value range in the depth map. Therefore, by setting a depth value threshold, the main subject area composed of all pixels smaller than the depth value threshold can be effectively excluded, thereby ensuring the clarity of the main subject area.

[0132] Step 32: In the non-subject area, determine the blur intensity of the pixel based on the difference between the depth value of the pixel and the focal plane.

[0133] In this context, the blur intensity s of each pixel in the non-subject region is determined by the absolute value of the difference between the depth value d of the pixel and the depth value z of the focal plane of the current frame image. Furthermore, the blur intensity s is positively correlated with the absolute value of the difference; that is, the larger the value of |dz|, the larger the blur intensity s of the pixel; conversely, the smaller the value of |dz|, the smaller the blur intensity s of the pixel.

[0134] Step 33: Combine the blur intensity of all pixels in the non-main region to form the blur mapping map.

[0135] Specifically, the bokeh map composed of the bokeh intensity of all pixels in the current frame image can be directly used as the final bokeh map of the current frame image. Alternatively, the bokeh map composed of the bokeh intensity of all pixels in the current frame can be processed, and the processed image can be used as the final bokeh map of the current frame image.

[0136] Based on the above method, a blurred mapping map with clear boundaries from the main area can be obtained.

[0137] Preferably, when generating the bokeh map, the bokeh map composed of the bokeh intensity of all pixels in the current frame image can be used as the initial bokeh map of the current frame image, and the initial bokeh map of the current frame image and the bokeh map of the previous frame image can be fused to obtain the bokeh map of the current frame image.

[0138] Because the background content of consecutive frames in a video is highly similar, the background blur maps obtained after blurring the background of consecutive frames are also highly similar. Therefore, by using the blur map of the previous frame and the initial blur map of the current frame to perform image fusion, the stability of the blur map between consecutive frames can be guaranteed, making the background blur effect between consecutive frames transition naturally.

[0139] Figure 3 This is another schematic flowchart illustrating the image processing method provided in an embodiment of this application. For example... Figure 3 As shown, the method includes:

[0140] Step S1: Receive the current frame image, and then execute steps S2 and S3 respectively;

[0141] Step S2: Perform main region detection on the current frame image, and then execute step S5;

[0142] Step S3: Generate the initial depth map of the current frame image, and then execute step S4;

[0143] Step S4: Using the depth map of the previous frame image, perform depth map stabilization processing on the initial depth map to obtain the depth map of the current frame image, and then execute step S5.

[0144] Step S5: Determine the depth value of the focus plane based on the depth value corresponding to the main body region in the depth map of the current frame image;

[0145] Step S6: Determine whether the current frame image is the first frame image;

[0146] If it is the first frame image, then proceed to step S8; otherwise, proceed to step S7 first, and then proceed to step S8.

[0147] Step S7: Use the depth value of the focal plane of the previous frame image to determine whether the focal plane of the current frame image has changed, and perform smoothing processing on the depth value of the focal plane when the focal plane changes.

[0148] Step S8: Determine the initial blur map of the non-subject region in the current frame image;

[0149] Step S9: Using the bokeh map of the previous frame image, fuse the initial bokeh map to obtain the bokeh map of the current frame image;

[0150] Step S10: Use the blur mapping map to blur the current frame image.

[0151] The method provided in this application embodiment ensures the privacy requirement of background blurring in non-subject areas and the usage requirement of clear images in subject areas, taking into account both privacy and security and communication needs. In addition, steps S3 and S4 are used to achieve the stability of the depth map between consecutive frames, step S7 is used to achieve the stability of the focus plane between consecutive frames, and step S9 is used to achieve the stability of the blur mapping map between consecutive frames.

[0152] Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 4 As shown, the device includes:

[0153] Module 401 is configured to acquire a depth map using the first model.

[0154] The first determining module 402 is configured to determine the depth value of the focusing plane based on the depth value corresponding to the main body region in the depth map of the current frame image.

[0155] The second determining module 403 is configured to determine the bokeh map based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane.

[0156] The processing module 404 is configured to perform blurring processing on the current frame image using the blurring mapping map.

[0157] The acquisition module 401 includes:

[0158] The first generation unit is configured to input the current frame image into the first model to perform a depth map generation operation to obtain an initial depth map.

[0159] The first fusion unit is configured to perform image fusion on the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image.

[0160] The first fusion unit is configured to perform the following operations on each pixel of the current frame image to obtain the depth value of each pixel in the depth map of the current frame image, including:

[0161] For the current pixel, calculate the product of the depth value in the initial depth map and the first weight to obtain a first result; and calculate the product of the depth value in the depth map of the previous frame and the second weight to obtain a second result; based on the first result and the second result, obtain the depth value of the current pixel.

[0162] The first weight used by the first fusion unit is obtained in the following ways:

[0163] Determine whether the current frame image meets the criteria for image blurring;

[0164] When the current frame image does not meet the judgment condition of image blur, calculate the difference between each pixel point in the current frame image and the previous frame image to obtain the current pixel difference information, and determine the value that is positively correlated with the current pixel difference information as the first weight;

[0165] When the current frame image meets the judgment condition of image blur, the preset default value is set to the first weight.

[0166] The criteria for determining image blur include:

[0167] The gradient sum of the current frame image is less than the gradient sum of the previous frame image, and the absolute value of the difference between the gradient sum of the current frame image and the gradient sum of the previous frame image is greater than a preset gradient sum threshold.

[0168] The first model used by the acquisition module 401 is obtained through the following methods:

[0169] The second model is obtained by training the original training data using a scale-invariant loss function as the loss function.

[0170] The original training data is input into the second model to obtain the predicted depth value. The predicted depth value is used as the new label in the original training data to obtain the training data with updated label.

[0171] The first model is obtained by training the training data with the updated labels.

[0172] The first determining module 402 includes:

[0173] The first determining unit is configured to determine the depth value of the focus plane of the current frame image based on the main body region in the depth map of the current frame image, and use it as the initial focus plane depth value.

[0174] The first judgment unit is configured to determine whether the focus plane of the current frame image has changed based on the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image.

[0175] The first processing unit is configured to update the initial focus plane depth value if the focus plane of the current frame image changes, and use the updated initial focus plane depth value as the depth value of the focus plane; otherwise, determine the initial focus plane depth value as the depth value of the focus plane.

[0176] Preferably, the first determining unit is configured to: filter out the depth map region corresponding to the main body region from the depth map of the current frame image, to form the main body region depth map; determine the number of pixels corresponding to different depth values ​​in the main body region depth map; starting from a preset depth value, count the cumulative number of pixels corresponding to different depth ranges according to a preset direction; when the cumulative number of pixels accounts for a preset first ratio of the total number of pixels in the main body region depth map, take the depth value of the other endpoint of the depth range that is different from the starting point as the depth value corresponding to the initial focus plane depth value as the initial focus plane depth value.

[0177] Preferably, the first determining unit is configured to calculate the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image to obtain a focus depth difference; if the absolute value of the focus depth difference is greater than a preset focus depth threshold, then it is determined that the focus plane of the current frame image has changed; otherwise, it is determined that the focus plane of the current frame image has not changed.

[0178] Preferably, the first processing unit is configured to use the depth value of the focus plane of the previous frame image to smooth the initial focus plane depth value to obtain an updated initial focus plane depth value.

[0179] Specifically, the first processing unit is configured to calculate the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image to obtain a focus depth difference; calculate the depth value adjustment amount for each frame based on the focus depth difference and a preset number of frames; and obtain the new depth value based on the depth value adjustment amount for each frame and the focus plane depth value of the previous frame image.

[0180] The second determining module 403 includes:

[0181] The second processing unit is configured to filter all pixels that satisfy a first condition in the depth map of the current frame image, wherein all pixels constitute a non-subject region; wherein the first condition is that the absolute value of the difference between the depth value of a pixel in the depth map of the current frame image and the depth value of the focal plane is greater than or equal to a preset depth value threshold.

[0182] The second determining unit is configured to determine the blur intensity of a pixel in the non-subject area based on the difference between the depth value of the pixel and the focal plane.

[0183] The third processing unit is configured to compose the blurring map by combining the blurring intensities of all pixels in the non-subject region.

[0184] Preferably, the third processing unit is configured to use the bokeh map composed of the bokeh intensity of all pixels in the current frame as the initial bokeh map of the current frame image, and to perform image fusion on the initial bokeh map of the current frame image and the bokeh map of the previous frame image to obtain the bokeh map of the current frame image.

[0185] The apparatus provided in this application embodiment obtains the depth map of the current frame image through a first model, and determines the depth value of the focal plane based on the depth value corresponding to the main body region in the depth map of the current frame image. Based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focal plane, the region of the current frame image is clearly divided into the main body region and the non-main body region, and a blur map is determined to achieve the purpose of blurring only the non-main body region. Then, the blur map is used to blur the current frame image, ensuring the privacy requirement of background blurring in the non-main body region and ensuring the use requirement of clear image in the main body region, thus taking into account the dual needs of privacy and security and communication use.

[0186] This application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method described in any of the preceding descriptions when it runs.

[0187] This application provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the method described in any of the preceding descriptions.

[0188] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. An image processing method, characterized in that, include: The depth map of the current frame image is obtained using the first model; The depth value of the focus plane is determined based on the depth value corresponding to the main body region in the depth map of the current frame image. A bokeh map is determined based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane. The current frame image is blurred using the blur mapping map; The process of obtaining the depth map of the current frame image using the first model includes: The current frame image is input into the first model to perform a depth map generation operation, resulting in an initial depth map; Image fusion is performed on the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image; Before performing the image fusion operation of the depth map, pixel alignment is performed using the optical flow of the current frame image and the optical flow of the previous frame image.

2. The method according to claim 1, characterized in that, The step of fusing the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image includes: For each pixel of the current frame image, the following operations are performed to obtain the depth value of each pixel in the depth map of the current frame image, including: For the current pixel, calculate the product of the depth value in the initial depth map and the first weight to obtain a first result; and calculate the product of the depth value in the depth map of the previous frame and the second weight to obtain a second result; based on the first result and the second result, obtain the depth value of the current pixel.

3. The method according to claim 2, characterized in that, The first weight is obtained in the following ways: Determine whether the current frame image meets the criteria for image blurring; When the current frame image does not meet the judgment condition of image blur, calculate the difference between each pixel point in the current frame image and the previous frame image to obtain the current pixel difference information, and determine the value that is positively correlated with the current pixel difference information as the first weight; When the current frame image meets the judgment condition of image blur, the preset default value is set to the first weight.

4. The method according to claim 3, characterized in that, The criteria for determining image blur include: The gradient sum of the current frame image is less than the gradient sum of the previous frame image, and the absolute value of the difference between the gradient sum of the current frame image and the gradient sum of the previous frame image is greater than a preset gradient sum threshold.

5. The method according to claim 1, characterized in that, The first model was trained using training data cleaned from the second model, including: The second model is obtained by training the original training data using a scale-invariant loss function as the loss function. The original training data is input into the second model to obtain the predicted depth value. The predicted depth value is used as the new label in the original training data to obtain the training data with updated label. The first model is obtained by training the training data with the updated labels.

6. The method according to any one of claims 1 to 5, characterized in that, Based on the depth value corresponding to the main body region in the depth map of the current frame image, the depth value of the focus plane is determined, including: Based on the main body region, the depth value of the focus plane of the current frame image is determined in the depth map of the current frame image, and used as the initial focus plane depth value; Based on the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image, it is determined whether the focus plane of the current frame image has changed. If the focus plane of the current frame image changes, the initial focus plane depth value is updated and the updated initial focus plane depth value is used as the depth value of the focus plane; otherwise, the initial focus plane depth value is determined to be the depth value of the focus plane.

7. The method according to claim 6, characterized in that, Based on the main body region, the depth value of the focus plane of the current frame image is determined in the depth map of the current frame image, and used as the initial focus plane depth value, including: The depth map region corresponding to the main body region is selected from the depth map of the current frame image to form the main body region depth map; Determine the number of pixels corresponding to different depth values ​​in the depth map of the main body region; Starting from a preset depth value, the cumulative number of pixels corresponding to different depth ranges is counted according to a preset direction. When the cumulative number of pixels accounts for a predetermined first ratio of the total number of pixels in the depth map of the main area, the depth value of the other endpoint of the depth range that is different from the starting point is taken as the depth value corresponding to the initial focus plane depth value.

8. The method according to claim 6, characterized in that, The step of determining whether the focus plane of the current frame image has changed based on the initial focus plane depth value includes: The difference between the initial focus plane depth value and the focus plane depth value of the previous frame image is calculated to obtain the focus depth difference. If the absolute value of the focus depth difference is greater than the preset focus depth threshold, it is determined that the focus plane of the current frame image has changed; otherwise, it is determined that the focus plane of the current frame image has not changed.

9. The method according to claim 6, characterized in that, The step of updating the initial focus plane depth value to obtain the updated initial focus plane depth value includes: The initial focus plane depth value is smoothed using the depth value of the focus plane in the previous frame image to obtain an updated initial focus plane depth value.

10. The method according to claim 9, characterized in that, The step of smoothing the initial focus plane depth value using the depth value of the focus plane from the previous frame image to obtain an updated initial focus plane depth value includes: The difference between the initial focus plane depth value and the focus plane depth value of the previous frame image is calculated to obtain the focus depth difference value m. Based on the focus depth difference m and the preset number of frames n, calculate the depth value adjustment o for each frame, where n is an integer greater than or equal to 2. Based on the depth value adjustment amount o for each frame and the depth value of the focus plane of the previous frame, the focus depth difference is compensated using a total of n image frames, including the current frame and subsequent consecutive image frames, and the magnitude of each compensation is the depth value adjustment amount o, to obtain the new depth values ​​of the n image frames.

11. The method according to claim 1, characterized in that, Based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane, a bokeh map is determined, including: In the depth map of the current frame image, all pixels that meet the first condition are selected, and all pixels form a non-subject area; wherein, the first condition is that the absolute value of the difference between the depth value of a pixel in the depth map of the current frame image and the depth value of the focal plane is greater than or equal to a preset depth value threshold. In the non-subject area, the blur intensity of the pixel is determined based on the difference between the depth value of the pixel and the focal plane. The blurring intensity of all pixels in the non-main region is used to form the blurring map.

12. The method according to claim 11, characterized in that, The step of assembling the blur map by combining the blur intensity of all pixels in the non-subject region includes: The bokeh map composed of the bokeh intensity of all pixels in the non-subject region in the current frame is used as the initial bokeh map of the current frame image. Image fusion is performed on the initial bokeh map of the current frame image and the bokeh map of the previous frame image to obtain the bokeh map of the current frame image.

13. An image processing apparatus, characterized in that, include: The acquisition module is configured to acquire the depth map of the current frame image using the first model; The first determining module is configured to determine the depth value of the focusing plane based on the depth value corresponding to the main body region in the depth map of the current frame image. The second determining module is configured to determine the bokeh map based on the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane. The processing module is configured to perform blurring processing on the current frame image using the blurring mapping map; The acquisition module is used to input the current frame image into the first model to perform a depth map generation operation to obtain an initial depth map; Image fusion is performed on the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image; Before performing the image fusion operation of the depth map, pixel alignment is performed using the optical flow of the current frame image and the optical flow of the previous frame image.

14. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 12 when it is run.

15. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 12.

Images