Image processing method and apparatus, and electronic device and storage medium

Abstract

An image processing method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring a depth map of a current image frame by means of a first model; on the basis of a depth value corresponding to a subject area in the depth map of the current image frame, determining the depth value of a focusing plane; determining a blurred map on the basis of the depth value of each pixel point in the depth map of the current image frame and the depth value of the focusing plane; and using the blurred map to perform blurring processing on the current image frame. Whilst improving the accuracy and stability, the method provided in the embodiments of the present disclosure can simulate a more natural gradient blurring effect of single-lens reflex on the basis of the distance between a person and a scene object.

Description

Image processing method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed on September 1, 2023, with application number 202311125064.X and invention name “An image processing method, device, electronic device and storage medium”, the content of which should be understood as incorporated into this application by reference. Technical Field

[0002] The present disclosure relates to the field of image processing, and more particularly to an image processing method, apparatus, electronic device, and storage medium. Background Art

[0003] Background blurring refers to blurring the background of an image, thereby allowing the foreground to appear clearly. It effectively highlights the subject and creates a visually appealing effect, making it a widely used photography technique. Some professional cameras can directly capture images with a blurred background, but this requires high lens aperture and zoom ratios. The cameras in electronic devices like mobile phones and tablets, due to limitations in size and thickness, lack the performance of lenses comparable to professional cameras, making it impossible to directly capture images with a blurred background. Therefore, image processing software is required to achieve this.

[0004] Several methods for blurring digital images have been proposed in the prior art, such as object-space depth-of-field rendering methods and image-space depth-of-field rendering methods. While object-space depth-of-field rendering methods produce realistic results, they are computationally intensive and memory-intensive, making it difficult to obtain results quickly and directly. Image-space depth-of-field rendering methods offer fast computational speeds, but the resulting background blur can result in blurred boundaries between the subject and the background, and unnatural transitions.

[0005] Summary of the Invention

[0006] In order to solve any of the above technical problems, the embodiments of the present disclosure provide an image processing method, device, electronic device and storage medium.

[0007] To achieve the purpose of the embodiments of the present disclosure, the embodiments of the present disclosure provide an image processing method, including:

[0008] Obtaining a depth map of the current frame image through the first model;

[0009] Determining a depth value of a focus plane according to a depth value corresponding to a main body area in a depth map of the current frame image;

[0010] Determining a blur map according to the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane;

[0011] The current frame image is defocused using the defocusing map.

[0012] An image processing device, comprising:

[0013] An acquisition module, configured to acquire a depth map of a current frame image through a first model;

[0014] A first determining module is configured to determine a depth value of a focus plane according to a depth value corresponding to a subject area in a depth map of the current frame image;

[0015] A second determining module is configured to determine a blur map according to the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane;

[0016] The processing module is configured to perform blurring processing on the current frame image using the blurring map.

[0017] A storage medium stores a computer program, wherein the computer program is configured to execute the method described above when running.

[0018] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method described above.

[0019] One of the above technical solutions has the following advantages or beneficial effects:

[0020] A depth map of the current frame image is obtained through the first model, and the depth value of the focus plane is determined according to the depth value corresponding to the main area in the depth map of the current frame image. According to the depth value of each pixel point in the depth map of the current frame image and the depth value of the focus plane, the area of ​​the current frame image is clearly divided into the main area and the non-main area, and a blur mapping map is determined to achieve the purpose of blurring only the non-main area. The current frame image is then blurred using the blur mapping map to ensure the privacy requirement of background blur in the non-main area and the use requirement of clear image in the main area, taking into account the dual needs of privacy security and communication use.

[0021] Other features and advantages of the embodiments of the present disclosure will be described in the following description, and in part will become apparent from the description, or may be understood by practicing the embodiments of the present disclosure. The objectives and other advantages of the embodiments of the present disclosure may be achieved and obtained through the structures particularly pointed out in the description, claims, and drawings.

[0022] Summary of the Figures

[0023] The accompanying drawings are used to provide a further understanding of the technical solutions of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the embodiments of the present disclosure, they are used to explain the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions of the embodiments of the present disclosure.

[0024] FIG1 is a schematic diagram of a flow chart of an image processing method provided by an embodiment of the present disclosure;

[0025] FIG2 is a schematic diagram showing the distribution of the number of pixels corresponding to depth values ​​provided by an embodiment of the present disclosure;

[0026] FIG3 is another schematic flow chart of the image processing method provided by an embodiment of the present disclosure;

[0027] FIG4 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure.

[0028] Details

[0029] To make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other in any way.

[0030] FIG1 is a flow chart of an image processing method provided by an embodiment of the present disclosure. As shown in FIG1 , the method includes:

[0031] Step 10: Obtain a depth map of the current frame image through the first model;

[0032] The first model is a deep learning model that is used to output a corresponding depth map for an input image. Using the first model to generate a depth map for the current frame image can effectively improve the efficiency of depth map generation.

[0033] Step 20: Determine the depth value of the focus plane according to the depth value corresponding to the main body area in the depth map of the current frame image;

[0034] The subject area is a face area, a body area, or an exhibit (such as an animal or an object).

[0035] The subject area can be obtained by semantic segmentation or instance segmentation, wherein the semantic segmentation can be implemented based on face segmentation, portrait segmentation or pet segmentation;

[0036] Alternatively, the subject area can be obtained through target detection technology, such as face detection, head detection, human body detection, pet detection, and target detection of other objects, where the working principle of pet detection is similar to that of human detection, the difference being that the object is different.

[0037] Among them, the method for determining the depth value of the focus plane can be reasonably selected and determined according to the depth value of the main area. For example, after the mask of the main area is obtained by semantic segmentation or instance segmentation, a depth value is selected from the depth map area corresponding to the mask as the depth value of the focus plane; or, after the target detection technology determines the target detection frame corresponding to the main area, a depth value is selected from the depth map area corresponding to the target detection frame as the depth value of the focus plane.

[0038] Of course, in addition to determining the depth value of the focus plane in the above manner, a fixed value may also be set as the depth value of the focus plane of the current frame image.

[0039] Step 30: determining a blur map according to the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane;

[0040] Specifically, taking pixels as units, determine whether each pixel is in the subject area or the non-subject area, thereby determining all the pixels that need to be blurred background, obtaining the non-subject area, and generating a blur map corresponding to the non-subject area as the image required for background blur of the subject area.

[0041] By clearly dividing the area of ​​the current frame image into the main area and the non-subject area, a blur mapping map is determined to achieve the purpose of blurring only the non-subject area, and then using the blur mapping map to blur the current frame image, ensuring the privacy requirement of background blur in the non-subject area and the use requirement of clear image in the main area, taking into account the dual needs of privacy security and communication use, highlighting the main effect and having a better visual effect.

[0042] For example, taking the application scenario of video conferencing as an example, privacy protection in video conferencing is an important requirement for users. The privacy information includes other people in the background, locations in the background, and other information, which need to be hidden in the video conference. Therefore, the method provided in the embodiment of the present disclosure can be used to blur the background of the non-subject area to achieve the purpose of ensuring privacy. Moreover, since the method provided in the embodiment of the present disclosure does not blur the subject area, the image of the subject area in the video conference is guaranteed to be clear, and the normal progress of the video conference is guaranteed.

[0043] Furthermore, as mobile photography and video production have become increasingly common, technological advancements have made it possible to achieve many professional-level photography techniques with mobile phones. One of these techniques is portrait video blurring, also known as "movie mode." Movie mode utilizes software algorithms to simulate a blurred background, thereby focusing the viewer's attention on the main part of the video. For this scenario, the techniques provided by the embodiments of the present disclosure can highlight the subject, such as a person or object, while blurring the background, giving the video a more professional and cinematic feel and enhancing the visual quality.

[0044] Step 40: Using the blur map to blur the current frame image;

[0045] Specifically, the blur map may be fused with the current frame image to obtain a blur processing result.

[0046] The method provided by the embodiment of the present disclosure obtains the depth map of the current frame image through the first model, and determines the depth value of the focus plane according to the depth value corresponding to the main area in the depth map of the current frame image, and clearly divides the area of ​​the current frame image into the main area and the non-main area according to the depth value of each pixel point in the depth map of the current frame image and the depth value of the focus plane, and determines a blur mapping map to achieve the purpose of blurring only the non-main area, and then uses the blur mapping map to blur the current frame image, thereby ensuring the privacy requirement of background blur in the non-main area and the use requirement of clear image in the main area, taking into account the dual needs of privacy security and communication use.

[0047] The following describes the method provided by the embodiment of the present disclosure:

[0048] In step 10, the first model is usually deployed on an electronic device such as a mobile phone or a tablet computer, so the deployment scale of the first model is relatively small.

[0049] With the development of deep learning and deep neural network technology, more and more evidence shows that larger-scale training data can achieve more accurate results. Therefore, the accuracy and stability of the first model deployed on a smaller scale need to be improved urgently.

[0050] Furthermore, the original training data used by the first model comes from a variety of sources, such as LiDAR, structured light, binocular stereo vision, and multi-view technology. The characteristics of the sample data collected by these different data sources vary significantly, primarily in depth map range and texture features. Since models deployed on electronic devices like mobile phones and tablets generally have fewer parameters and limited capabilities, they are unable to overcome the sample unevenness caused by different data sources. Consequently, this further deteriorates the accuracy and stability of the first model.

[0051] To address the above issues, the present disclosure proposes the following solutions, including:

[0052] The training data used by the first model is the training data obtained after being cleaned by the second model.

[0053] The second model may be a model generated by a cloud server.

[0054] The original training data is cleaned by the second model to ensure that the distribution of the cleaned training data is more uniform while the amount of the original training data is not reduced. Therefore, when the first model is established using the cleaned training data, the accuracy and stability of the depth map output by the first model can be improved.

[0055] Specifically, the first model is obtained by:

[0056] Based on the scale-invariant loss function as the loss function, the second model is obtained by training with the original training data;

[0057] Inputting the original training data into the second model to obtain a predicted depth value, and using the predicted depth value as a new label in the original training data to obtain training data with an updated label;

[0058] The first model to be trained is trained using the training data after the label is updated to obtain the first model.

[0059] In the above implementation, the original training data is the original image. Since the original image records the most original feature value of the image itself, the accuracy of the second model can be effectively improved by training and establishing the second model using the original training data.

[0060] Furthermore, since the original training data may come from different data sources, using a scale-invariant loss function as the loss function can better process the original training data from multiple data sources and complete the training operation of the second model.

[0061] In the above implementation, the second model is used to determine the predicted depth value of the original training data, and the predicted depth value is used as the new label in the original training data, weakening the distribution difference of samples from different sources, so that the overall performance of the training data after the label is updated is more consistent. Therefore, the accuracy and stability of the first model obtained by using the training data after the label is updated can be improved.

[0062] In step 10, a depth map of the current frame image is obtained by using the first model, including:

[0063] Step 11: Input the current frame image into the first model to perform a depth map generation operation to obtain an initial depth map;

[0064] The initial depth map is not the depth map ultimately used by the current frame image, and needs to be fused with the depth map of the previous frame image before it can be used as the depth map ultimately used by the current frame image.

[0065] Step 12: performing image fusion on the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image;

[0066] Specifically, to address the problem of high-frequency flickering between previous and next frame images, the depth map of each frame image is fused with the depth map of its previous frame image, which can effectively overcome the problem of high-frequency flickering between previous and next frame images. In addition, to address the problem of low-frequency flickering between dozens of frames or even more frames, the depth map of each frame image is fused with the depth map of its previous frame image, so that each frame is obtained based on the depth map fusion processing, indicating that the depth map of the previous frame image has been fused with the depth maps of the previous frames, so the problem of low-frequency flickering in multiple frames can be effectively overcome.

[0067] In step 12, the initial depth map of the current frame image and the depth map of the previous frame image are fused to obtain the depth map of the current frame image, including:

[0068] The following operations are performed on each pixel of the current frame image to obtain a depth value of each pixel in the depth map of the current frame image, including:

[0069] For the current pixel point, calculate the product of the depth value in the initial depth map and the first weight to obtain a first result; and calculate the product of the depth value in the depth map of the previous frame image and the second weight to obtain a second result; based on the first result and the second result, obtain the depth value of the current pixel point.

[0070] In the above implementation method, the weighted sum of the depth values ​​corresponding to each pixel in the initial depth map of the previous frame image and the depth map of the previous frame image is calculated in units of pixels, and the depth value of each pixel in the depth map of the current frame image is obtained, so that the depth value of each pixel can be fused, which can effectively improve the stability of the depth map between the previous and next frames.

[0071] Furthermore, before performing the image fusion operation of the depth map, the optical flow of the current frame image and the optical flow of the previous frame image are used to perform pixel alignment operation. After completing the pixel alignment operation between the current frame image and the previous frame image, the image fusion operation is performed to improve the accuracy of the image fusion result.

[0072] When shooting video using a camera, especially a handheld camera (e.g., a mobile phone), motion blur can easily occur in the captured image when the camera moves. This is especially true in low-light environments such as indoors or at night, where the probability of motion blur is higher. Therefore, overcoming the negative effects of motion blur is a problem that needs to be addressed. The present disclosure proposes the following solutions, including:

[0073] Depending on whether the current frame image is blurred, different methods are used to determine the value of the first weight, including:

[0074] Determining whether the current frame image meets the image blur judgment condition;

[0075] Specifically, an image blur detection model may be preset to detect whether an input image satisfies a judgment condition of image blur.

[0076] When the current frame image does not meet the image blur judgment condition, calculating the difference between the pixels of the current frame image and the previous frame image to obtain current pixel difference information, and determining a value that is positively correlated with the current pixel difference information as the first weight;

[0077] The current pixel difference information may be determined based on the difference between the optical flow of the current frame image and the optical flow of the previous frame image.

[0078] Specifically, a correspondence between pixel difference information and a first weight may be pre-set, and based on this correspondence, a numerical value corresponding to the current pixel difference information is determined as the first weight. The correspondence may be the first weight corresponding to different value ranges of pixel difference information, or a pre-set calculation expression having the pixel difference information as an independent variable and the first weight as a dependent variable.

[0079] When the current frame image meets the judgment condition of image blur, setting a preset default value as the first weight;

[0080] The default value is usually less than 0.5, so that the second weight corresponding to the depth map of the previous frame image is greater than the first weight corresponding to the initial depth map of the current frame image, so as to suppress the negative impact of motion blur of the current frame image.

[0081] According to whether the current frame image is blurred, a suitable method is selected for the first weight to determine the corresponding value, which can provide support for the depth map fusion operation and effectively suppress the negative impact of motion blur on the depth map fusion operation.

[0082] Preferably, in the application scenario of the solution of the embodiment of the present disclosure, since motion blur is mainly reflected between the previous and next frames, the change information of the gradient sum of the previous and next frames can be used as the judgment condition for image blur.

[0083] Specifically, the conditions for determining whether the image is blurred include:

[0084] The gradient sum of the current frame image is smaller than the gradient sum of the previous frame image, and the absolute value of the difference between the gradient sum of the current frame image and the gradient sum of the previous frame image is greater than a preset gradient sum threshold.

[0085] Since the image content of the previous and next frames in the video is highly similar, the magnitude of the gradient sum of the previous and next frames is also similar. Therefore, if the gradient sum of a frame is much smaller than the gradient sum of the previous frame, it indicates that there is motion blur, with low computational complexity and high accuracy.

[0086] During the filming process, the subject may move, such as a person or pet's head turning (such as lowering their head, turning backward, turning left or right), a person or pet's body moving in space (such as moving or bending a limb), or an electronic product with mobile functionality moving in space (such as a robot vacuum or electric car toy). As the subject moves, the camera will adaptively track the subject, resulting in a change in the focus plane.

[0087] Taking portrait scenes as an example, when using faces and heads for detection and tracking, if the head can be continuously tracked, it means that the changes in the scene and the characters in the previous and next frames are relatively small, and the depth value difference of the focus plane of the previous and next frames is also small, so the captured image is stable. When the face or head is blocked or disappears (such as moving out of the shooting range, or the front face turns to the back), detection and tracking will fail, and it is necessary to switch to human body detection. Or, the shooting scene will change because the subject is replaced by another subject (for example, the face frame of another person is larger than the face frame of the original subject). When the detection method or shooting scene changes, the focus plane needs to be switched to adapt to the above changes.

[0088] When the focus plane changes, it is necessary to ensure a smooth transition between the focus planes of the previous and next frame images to ensure the continuity of the background blur image and make the background blur effect transition between the previous and next frames natural. Therefore, the embodiments of the present disclosure provide the following solutions, including:

[0089] In step 20, the depth value of the focus plane is determined according to the depth value corresponding to the main body area in the depth map of the current frame image, including:

[0090] Step 21: determining, according to the subject area, a depth value of a focus plane of the current frame image in the depth map of the current frame image as an initial focus plane depth value;

[0091] The initial focus plane depth value is not the final depth value of the focus plane of the current frame image, and needs to be fused with the depth value of the focus plane of the previous frame image before it can be used as the final depth value of the focus plane of the current frame image.

[0092] Step 22: determining whether the focus plane of the current frame image has changed based on the difference between the depth value of the initial focus plane and the depth value of the focus plane of the previous frame image;

[0093] Specifically, when the subject area is successfully tracked, the initial focus plane depth value is relatively close to the depth value of the focus plane of the previous frame due to the continuity of the previous and subsequent frames. Only when the focus on the subject switches does the difference between the initial focus plane depth value and the depth value of the focus plane of the previous frame increase. Therefore, based on this characteristic, the difference between the initial focus plane depth value and the depth value of the focus plane of the previous frame can be used to determine whether the focus plane of the current frame has changed.

[0094] Step 23: If the focus plane of the current frame image changes, the initial focus plane depth value is updated to obtain an updated initial focus plane depth value, and the updated initial focus plane depth value is used as the depth value of the focus plane; otherwise, the initial focus plane depth value is determined to be the depth value of the focus plane;

[0095] Specifically, when the focus plane of the current frame image changes, updating the initial focus plane depth value ensures a smooth transition between the focus planes of the previous and next frames, thereby ensuring the continuity of the background blur image and making the background blur effect transition between the previous and next frames natural. If the focus plane of the current frame image does not change, there is no need to update the initial focus plane depth value; the initial focus plane depth value can be directly used as the depth value of the focus plane.

[0096] By determining whether the focus plane of the current frame image has changed and selecting a corresponding method to determine the depth value of the focus plane of the current frame image, the stability of the focus plane between the previous and next frames can be improved.

[0097] In step 21, according to the subject area, a depth value of a focus plane of the current frame image is determined in the depth map of the current frame image as an initial focus plane depth value, including:

[0098] Filtering a depth map area corresponding to the main body area from the depth map of the current frame image to obtain a depth map of the main body area;

[0099] Specifically, the subject area detection method in the prior art can be used to determine the pixel information of the subject area in the current frame image, and the area corresponding to the pixel information can be determined in the depth map to obtain the subject area depth map.

[0100] Determine the number of pixels corresponding to different depth values ​​in the depth map of the subject area;

[0101] Specifically, the range of depth values ​​in the depth map of the main area is counted. Taking the range of depth values ​​including A values ​​as an example, the depth values ​​are classified according to their size, and the number of pixels corresponding to each depth value in the depth map of the main area is determined, that is, A depth values ​​correspond to A categories, and the number of pixels corresponding to each category is determined, where A is an integer greater than or equal to 2.

[0102] Starting from the preset depth value, the cumulative number of pixels corresponding to different depth ranges is counted along the preset direction;

[0103] The preset depth value is smaller than or equal to the maximum depth value in the main body area depth map, and larger than or equal to the minimum depth value in the main body area depth map.

[0104] The preset direction is the direction of depth value change, which may be a direction in which the preset depth value decreases or a direction in which the preset depth value increases. In actual use, a suitable direction may be selected according to the size of the selected preset depth value.

[0105] Specifically, according to the depth value change direction, starting from the preset depth value, the total number of pixels corresponding to the depth values ​​in the depth map of the main area is accumulated to obtain the cumulative number of pixels.

[0106] If the maximum value of the depth value in the depth map of the main area or the value adjacent to the maximum value is used as the preset depth value, the number of pixels can be accumulated in the direction of decreasing the preset depth value; if the minimum value of the depth value in the depth map of the main area or the value adjacent to the minimum value is used as the preset depth value, the number of pixels can be accumulated in the direction of increasing the preset depth value.

[0107] When the ratio of the accumulated number of pixels to the total number of pixels in the depth map of the main body area is a preset first ratio, a depth value of another endpoint of the depth range different from the starting point is used as the initial focus plane depth value;

[0108] The first ratio is used to measure whether the cumulative number of pixels can cover most of the pixels in the main area.

[0109] For example, taking the maximum value max of the depth value in the depth map of the main area as the preset depth value, the number of pixels can be accumulated in the direction of decreasing depth value. When the depth value range is [max, a1], the ratio has not reached the first ratio, and the number of pixels continues to be accumulated in the direction of decreasing depth value. When the depth value range is [max, a2], the ratio reaches the first ratio, and the value a2 is used as the initial focus plane depth value, where a1 is greater than a2.

[0110] The above method can accurately determine the initial focus plane depth value of the current frame image.

[0111] Taking the distribution diagram shown in FIG2 as an example, the horizontal axis is the depth value, the vertical axis is the number of pixels, and the curve represents the corresponding relationship between the depth value and the number of pixels in the depth map of a certain subject area.

[0112] As shown in Figure 2, the shaded portion is composed of a curve, a proportional threshold line, a horizontal axis, and a vertical axis. The position of the proportional threshold line indicates that the ratio of the cumulative number of pixel points to the total number of pixel points in the depth map of the main area is a preset first ratio, that is, the value of the horizontal axis of the proportional threshold line is used as the initial focus plane depth value.

[0113] In step 22, judging whether the focus plane of the current frame image has changed according to the difference between the depth value of the initial focus plane and the depth value of the focus plane of the previous frame image includes:

[0114] Calculating the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image to obtain a focus depth difference;

[0115] If the absolute value of the focus depth difference is greater than the preset focus depth threshold, it means that the captured image is unstable and it is determined that the focus plane of the current frame image has changed; otherwise, it means that the captured image is stable and it is determined that the focus plane of the current frame image has not changed.

[0116] The magnitude relationship between the absolute value of the focus depth difference and the focus depth threshold is used to determine whether the focus plane of the current frame image has changed, which is simple and convenient to implement.

[0117] In step 23, the initial focal plane depth value is updated to obtain an updated initial focal plane depth value, including:

[0118] The initial focus plane depth value is smoothed by using the depth value of the focus plane of the previous frame image to obtain the updated initial focus plane depth value.

[0119] Specifically, by using the depth value of the focus plane of the previous frame image to smooth the initial focus plane depth value, it can be ensured that the change in the depth value of the focus plane of the previous and next frame images is within a reasonable space, thereby achieving a smooth transition of the focus planes of the previous and next frame images, ensuring the stability of the picture between the previous and next frames, and reducing the probability of picture instability problems.

[0120] Preferably, the step of smoothing the initial focus plane depth value by using the depth value of the focus plane of the previous frame image to obtain an updated initial focus plane depth value comprises:

[0121] Step 231: Calculate the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image to obtain a focus depth difference m; wherein the focus depth difference m is used to measure the degree of change in the depth of the focus plane;

[0122] Step 232: Calculate a depth value adjustment amount o for each frame based on the focus depth difference m and a preset number of frames n; wherein the depth value adjustment amount o for each frame may be a ratio of the focus depth value m to the number of frames n, where n is an integer greater than or equal to 2;

[0123] Specifically, a total of n image frames including the current frame image and subsequent continuous image frames are used to complete the compensation of the focus depth difference, and the amplitude of each compensation is the depth value adjustment amount, thereby completing the smooth switching of the focus plane.

[0124] Compensating for the difference in focus depth starting with the current frame where the focus plane shift occurs can quickly address the negative impact of the focus plane switch. Furthermore, performing compensation across multiple frames ensures smooth focus plane switching between previous and next frames.

[0125] Step 233: Obtain the updated initial focus plane depth value according to the depth value adjustment amount of each frame and the depth value of the focus plane of the previous frame image;

[0126] After the focus plane changes, the existing technology directly uses the initial focus plane depth value as the focus plane depth value of the current frame image. Since the depth value of the focus plane changes greatly, the previous and next frame images may become unstable. Therefore, the embodiment of the present disclosure proposes setting the depth value of the focus plane of the current frame image to the depth value of the focus plane of the adjacent previous frame image to ensure that the depth value change of the focus plane of the previous and next frame images is within a reasonable range, ensuring the stability of the picture, and then completing the compensation of the focus depth difference through the consecutive n-1 image frames after the current frame image to achieve batch compensation, thereby ensuring the stability of the previous and next frame pictures.

[0127] For example, the depth value of the focus plane of the Nth frame is z0, and the focus plane of the N+1th frame changes, wherein the initial focus plane depth value z1 of the focus plane of the N+1th frame is z0+m. At this time, the depth value z of the focus plane of the N+1th frame cannot be set to the initial focus plane depth value z1. Instead, the depth value z of the focus plane of the current frame is set to a value relatively close to z0.

[0128] The calculation expression of the depth value z of the focus plane of the current frame is as follows:

[0129] In the process of completing the focus depth difference compensation through a total of n image frames including the current frame image and the subsequent continuous image frames, if the camera and the subject are in a relatively static state, n frames are required to complete the focus switching; if the camera or the subject has corresponding movement, when performing each compensation operation, the depth value adjustment amount of each frame can be corrected to control the compensation operation to be greater than or less than n image frames.

[0130] In an exemplary embodiment, the value of n is 10.

[0131] In step 30, a blur map is determined according to the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane, including:

[0132] Step 31: Filter all pixels that meet a first condition in the depth map of the current frame image, where all pixels constitute a non-subject area; wherein the first condition is that the absolute value of the difference between the depth value of the pixel in the depth map of the current frame image and the depth value of the focus plane is greater than or equal to a preset depth value threshold;

[0133] Among them, determining the non-subject area using pixels as units can improve the accuracy of the non-subject area.

[0134] Specifically, since the subject is generally not a pure plane, there is a corresponding depth value range in the depth map. Therefore, by setting a depth value threshold, the subject area composed of all pixels less than the depth value threshold can be effectively excluded, thereby ensuring the clarity of the subject area.

[0135] Step 32: In the non-subject area, determine the blur intensity of the pixel according to the difference between the depth value of the pixel and the focus plane;

[0136] Among them, the blur intensity s of each pixel point in the non-subject area is determined according to the absolute value of the difference between the depth value d of the pixel point and the depth value z of the focus plane of the current frame image, and the blur intensity s is positively correlated with the absolute value of the difference, that is, the larger the value of |dz|, the greater the blur intensity s of the pixel point; conversely, the smaller the value of |dz|, the smaller the blur intensity s of the pixel point.

[0137] Step 33: Combining the blur intensities of all pixels in the non-subject area into the blur map.

[0138] Specifically, the blur map composed of the blur intensities of all pixels in the current frame image can be directly used as the blur mapping map finally used by the current frame image, or, after processing the blur map composed of the blur intensities of all pixels in the current frame image, the processed image can be used as the blur mapping map finally used by the current frame image.

[0139] Based on the above method, a blurred mapping image with a clear boundary with the main area can be obtained.

[0140] Preferably, when generating the blur mapping map, the blur mapping map composed of the blur intensities of all pixels in the non-subject area of ​​the current frame image can be used as the initial blur mapping map of the current frame image, and the initial blur mapping map of the current frame image and the blur mapping map of the previous frame image are subjected to image fusion to obtain the blur mapping map of the current frame image.

[0141] Since the background image content of the previous and next frames in the video is highly similar, the blur maps obtained after the background blurring of the previous and next frames are also highly similar. Therefore, by using the blur map of the previous frame image and performing image fusion with the initial blur map of the current frame image, the blur map can be ensured to be stable between the previous and next frames, so that the background blur effect between the previous and next frames transitions naturally.

[0142] FIG3 is another flow chart of an image processing method provided by an embodiment of the present disclosure. As shown in FIG3 , the method includes:

[0143] Step S1, receiving the current frame image, and executing steps S2 and S3 respectively;

[0144] Step S2, performing subject area detection on the current frame image, and then executing step S5;

[0145] Step S3: Generate an initial depth map of the current frame image, and then execute step S4;

[0146] Step S4: Using the depth map of the previous frame image, perform depth map stabilization processing on the initial depth map to obtain the depth map of the current frame image, and then execute step S5;

[0147] Step S5: determining a depth value of the focus plane according to a depth value corresponding to the main body area in the depth map of the current frame image;

[0148] Step S6: determining whether the current frame image is the first frame image;

[0149] If it is the first frame image, execute step S8; otherwise, execute step S7 first, and then execute step S8;

[0150] Step S7: using the depth value of the focus plane of the previous frame image to determine whether the focus plane of the current frame image has changed, and performing a smoothing process on the depth value of the focus plane if the focus plane has changed;

[0151] Step S8, determining an initial blur image of the non-subject area in the current frame image;

[0152] Step S9: using the blur map of the previous frame image to fuse the initial blur map to obtain the blur map of the current frame image;

[0153] Step S10: Using the blur map to blur the current frame image.

[0154] The method provided by the embodiment of the present disclosure ensures the privacy requirement of background blur in non-subject areas and the use requirement of clear images in the subject area, taking into account the dual needs of privacy security and communication use. In addition, the stability of the depth map between the previous and next frames is achieved by using steps S3 and S4, the stability of the focus plane between the previous and next frames is achieved by using step S7, and the stability of the blur mapping map between the previous and next frames is achieved by using step S9.

[0155] FIG4 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. As shown in FIG4 , the device includes:

[0156] An acquisition module 401 is configured to acquire a depth map of a current frame image through a first model;

[0157] A first determining module 402 is configured to determine a depth value of a focus plane according to a depth value corresponding to a subject area in a depth map of the current frame image;

[0158] A second determining module 403 is configured to determine a blur map according to the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane;

[0159] The processing module 404 is configured to perform blurring processing on the current frame image using the blurring map.

[0160] The acquisition module 401 includes:

[0161] A first generating unit is configured to input the current frame image into the first model to perform a depth map generation operation to obtain an initial depth map;

[0162] The first fusion unit is configured to perform image fusion on the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image.

[0163] The first fusion unit is configured to perform the following operations on each pixel of the current frame image to obtain a depth value of each pixel in the depth map of the current frame image, including:

[0164] For the current pixel point, calculate the product of the depth value in the initial depth map and the first weight to obtain a first result; and calculate the product of the depth value in the depth map of the previous frame image and the second weight to obtain a second result; based on the first result and the second result, obtain the depth value of the current pixel point.

[0165] The first weight used by the first fusion unit is obtained by:

[0166] Determining whether the current frame image meets the image blur judgment condition;

[0167] When the current frame image does not meet the image blur judgment condition, calculating the difference between each pixel in the current frame image and the previous frame image to obtain current pixel difference information, and determining a value that is positively correlated with the current pixel difference information as the first weight;

[0168] When the current frame image meets the judgment condition of image blur, the preset default value is set as the first weight.

[0169] The conditions for determining whether the image is blurred include:

[0170] The gradient sum of the current frame image is smaller than the gradient sum of the previous frame image, and the absolute value of the difference between the gradient sum of the current frame image and the gradient sum of the previous frame image is greater than a preset gradient sum threshold.

[0171] The training data used by the first model used by the acquisition module 401 is the training data cleaned by the second model, wherein the first model is obtained by:

[0172] Based on the scale-invariant loss function as the loss function, the second model is obtained by training with the original training data;

[0173] Inputting the original training data into the second model to obtain a predicted depth value, and using the predicted depth value as a new label in the original training data to obtain training data with an updated label;

[0174] The first model to be trained is trained using the training data after the label is updated to obtain the first model.

[0175] The first determining module 402 includes:

[0176] A first determining unit is configured to determine, according to the subject area, a depth value of a focus plane of the current frame image in the depth map of the current frame image as an initial focus plane depth value;

[0177] a first judging unit configured to judge whether the focus plane of the current frame image has changed according to a difference between the depth value of the initial focus plane and the depth value of the focus plane of the previous frame image;

[0178] The first processing unit is configured to update the initial focal plane depth value if the focal plane of the current frame image changes, obtain an updated initial focal plane depth value, and use the updated initial focal plane depth value as the depth value of the focal plane; otherwise, determine the initial focal plane depth value as the depth value of the focal plane.

[0179] Preferably, the first determination unit is configured to filter out a depth map area corresponding to the main area from the depth map of the current frame image, which is the main area depth map; determine the number of pixels corresponding to different depth values ​​in the main area depth map; take the preset depth value as the starting point, and count the cumulative number of pixels corresponding to different depth ranges in a preset direction; when the ratio of the cumulative number of pixels to the number of all pixels in the main area depth map is a preset first ratio, the depth value of another endpoint of the depth range that is different from the starting point is used as the initial focus plane depth value.

[0180] Preferably, the first determination unit is configured to calculate the difference between the initial focus plane depth value and the depth value of the focus plane of the previous frame image to obtain a focus depth difference; if the absolute value of the focus depth difference is greater than a preset focus depth threshold, it is determined that the focus plane of the current frame image has changed; otherwise, it is determined that the focus plane of the current frame image has not changed.

[0181] Preferably, the first processing unit is configured to use the depth value of the focus plane of the previous frame image to perform smoothing processing on the initial focus plane depth value to obtain an updated initial focus plane depth value.

[0182] Specifically, the first processing unit is configured to calculate the difference between the initial focus plane depth value and the depth value of the focus plane of the previous frame image to obtain the focus depth difference; calculate the depth value adjustment amount of each frame based on the focus depth difference and a preset number of frames; and obtain the updated initial focus plane depth value based on the depth value adjustment amount of each frame and the depth value of the focus plane of the previous frame image.

[0183] The second determining module 403 includes:

[0184] a second processing unit configured to screen all pixels in the depth map of the current frame image that meet a first condition, wherein all pixels constitute a non-subject area; wherein the first condition is that an absolute value of a difference between a depth value of a pixel in the depth map of the current frame image and a depth value of a focus plane is greater than or equal to a preset depth value threshold;

[0185] a second determining unit configured to determine, in the non-subject area, a blur intensity of a pixel point according to a difference between a depth value of the pixel point value and a focus plane;

[0186] The third processing unit is configured to combine the blur intensities of all pixels in the non-subject area into the blur map.

[0187] Preferably, the third processing unit is configured to use the blur map composed of the blur intensities of all pixel points in the non-subject area of ​​the current frame image as the initial blur mapping map of the current frame image, and perform image fusion on the initial blur mapping map of the current frame image and the blur mapping map of the previous frame image to obtain the blur mapping map of the current frame image.

[0188] The device provided by the embodiment of the present disclosure obtains the depth map of the current frame image through the first model, and determines the depth value of the focus plane according to the depth value corresponding to the main area in the depth map of the current frame image, and clearly divides the area of ​​the current frame image into the main area and the non-main area according to the depth value of each pixel point in the depth map of the current frame image and the depth value of the focus plane, and determines a blur mapping map to achieve the purpose of blurring only the non-main area, and then uses the blur mapping map to blur the current frame image, thereby ensuring the privacy requirement of background blur in the non-main area and the use requirement of clear image in the main area, taking into account the dual needs of privacy security and communication use.

[0189] An embodiment of the present disclosure provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to execute any one of the methods described above when running.

[0190] An embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform any of the methods described above.

[0191] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. An image processing method, comprising: Acquire a depth map of the current frame image through the first model; Determine the depth value of the focus plane according to the depth value corresponding to the main body area in the depth map of the current frame image; Determine a blur map according to the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane; The current frame image is blurred using the blur map.

2. The method according to claim 1, wherein: The step of obtaining a depth map of the current frame image by using the first model includes: Inputting the current frame image into the first model to perform a depth map generation operation to obtain an initial depth map; The initial depth map of the current frame image and the depth map of the previous frame image are fused to obtain the depth map of the current frame image.

3. The method according to claim 2, wherein: The performing image fusion on the initial depth map of the current frame image and the depth map of the previous frame image to obtain the depth map of the current frame image includes: The following operations are performed on each pixel of the current frame image to obtain a depth value of each pixel in the depth map of the current frame image, including: For the current pixel, the product of the depth value in the initial depth map and the first weight is calculated to obtain a first result; and the product of the depth value in the depth map of the previous frame image and the second weight is calculated to obtain a second result; based on the first result and the second result, the depth value of the current pixel is obtained.

4. The method according to claim 3, wherein: The first weight is obtained in the following manner, including: Determine whether the current frame image meets the judgment condition of image blur; When the current frame image does not meet the judgment condition of image blur, the difference between the current frame image and the previous frame image of each pixel is calculated to obtain current pixel difference information, and a value positively correlated with the current pixel difference information is determined as the first weight; When the current frame image meets the judgment condition of image blur, the preset default value is set as the first weight.

5. The method according to claim 4, wherein: The judgment conditions for image blur include: The gradient sum of the current frame image is smaller than the gradient sum of the previous frame image, and the absolute value of the difference between the gradient sum of the current frame image and the gradient sum of the previous frame image is greater than a preset gradient sum threshold.

6. The method according to claim 1, wherein: The training data used by the first model is the training data cleaned by the second model; The first model is obtained by: Based on the scale-invariant loss function as the loss function, the second model is obtained by training with the original training data; Inputting the original training data into the second model to obtain a predicted depth value, using the predicted depth value as a new label in the original training data, and obtaining training data with an updated label; The first model to be trained is trained using the training data after the label is updated to obtain the first model.

7. The method according to any one of claims 1 to 6, wherein: The step of determining the depth value of the focus plane according to the depth value corresponding to the main body area in the depth map of the current frame image includes: According to the subject area, determining a depth value of a focus plane of the current frame image in the depth map of the current frame image as an initial focus plane depth value; Determining whether the focus plane of the current frame image has changed according to the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image; If the focus plane of the current frame image changes, the initial focus plane depth value is updated to obtain an updated initial focus plane depth value, and the updated initial focus plane depth value is used as the depth value of the focus plane; otherwise, the initial focus plane depth value is determined to be the depth value of the focus plane.

8. The method according to claim 7, wherein: The determining, according to the subject area, a depth value of a focus plane of the current frame image in the depth map of the current frame image as an initial focus plane depth value comprises: Filtering out a depth map area corresponding to the main body area from the depth map of the current frame image as the main body area depth map; Determine the number of pixels corresponding to different depth values ​​in the depth map of the main area; Taking the preset depth value as the starting point, the cumulative number of pixels corresponding to different depth ranges is counted in the preset direction; When the ratio of the accumulated number of pixels to the number of all pixels in the main area depth map is a preset first ratio, a depth value of another end point of the depth range different from the starting point is used as the initial focus plane depth value.

9. The method according to claim 7, wherein: The determining, according to the difference between the initial focal plane depth value and the focal plane depth value of the previous frame image, whether the focal plane of the current frame image has changed includes: Calculating the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image to obtain a focus depth difference; If the absolute value of the focus depth difference is greater than a preset focus depth threshold, it is determined that the focus plane of the current frame image has changed; otherwise, it is determined that the focus plane of the current frame image has not changed.

10. The method according to claim 7, wherein: The updating of the initial focal plane depth value to obtain an updated initial focal plane depth value includes: The initial focus plane depth value is smoothed by using the depth value of the focus plane of the previous frame image to obtain the updated initial focus plane depth value.

11. The method according to claim 10, wherein: The step of smoothing the initial focal plane depth value by using the depth value of the focal plane of the previous frame image to obtain the updated initial focal plane depth value comprises: Calculating the difference between the initial focus plane depth value and the focus plane depth value of the previous frame image to obtain a focus depth difference; Calculating a depth value adjustment amount for each frame according to the focus depth difference and a preset number of frames; The updated initial focal plane depth value is obtained according to the depth value adjustment amount of each frame and the depth value of the focal plane of the previous frame image.

12. The method according to claim 1, wherein: The step of determining the blur map according to the depth value of each pixel in the depth map of the current frame image and the depth value of the focus plane comprises: Screening all pixels that meet a first condition in the depth map of the current frame image, wherein all pixels constitute a non-subject area; wherein the first condition is that an absolute value of a difference between a depth value of a pixel in the depth map of the current frame image and a depth value of a focus plane is greater than or equal to a preset depth value threshold; In the non-subject area, the blur intensity of the pixel is determined according to the difference between the depth value of the pixel value and the focus plane; The blurring intensities of all pixels in the non-subject area are combined into the blurring map.

13. The method according to claim 12, wherein: The forming the blur map from blur intensities of all pixels in the non-subject area comprises: Using a blur map composed of blur intensities of all pixels in the non-subject area of ​​the current frame image as an initial blur mapping map of the current frame image; The initial blur map of the current frame image and the blur map of the previous frame image are image-fused to obtain the blur map of the current frame image.

14. An image processing device, comprising: An acquisition module, configured to acquire a depth map of a current frame image through a first model; A first determination module is configured to determine a depth value of a focus plane according to a depth value corresponding to a main body area in a depth map of the current frame image; A second determination module is configured to determine a blur map according to a depth value of each pixel in the depth map of the current frame image and a depth value of the focus plane; The processing module is configured to use the blur map to perform blur processing on the current frame image.

15. A storage medium storing a computer program, wherein: The computer program is arranged to perform the method as claimed in any one of claims 1 to 13 when executed.

16. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 13.

Images