Image blurring processing method and device, equipment, storage medium and program product

Abstract

The invention relates to an image blurring processing method and device, equipment, a storage medium and a program product. According to one embodiment of the invention, the method comprises the steps of determining focusing information of a current image in response to an obtained to-be-processed current image; determining depth information of the current image based on the focusing information; and performing background blurring processing on the current image based on the depth information. According to the invention, the cost of equipment can be reduced, and images acquired by double cameras do not need to be processed, so that the image processing process can be simplified, the processing burden of the system is reduced, the image processing speed can be improved, and the power consumption of the equipment is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of terminal technology, and in particular to an image blurring processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of terminal technology, taking photos with mobile phones and other devices is becoming more and more common, and people's requirements for photo quality and shooting experience are also getting higher and higher.

[0003] To adapt to complex and ever-changing shooting scenarios, highlight the main subject of the shot, and enhance the depth of the image, it is usually necessary to blur the background of the captured image. This means keeping the main subject in the image sharp while blurring other background areas.

[0004] In related technologies, bokeh can be achieved by adjusting the aperture or by calculating depth information from the main and secondary camera images. However, adjusting the aperture requires a camera with a variable aperture, which increases the hardware cost. Calculating depth information from both main and secondary camera images requires both cameras, and performing calculations on both images slows down the shooting speed, increases the system's processing load, and raises the device's power consumption, thus impacting the user experience. Summary of the Invention

[0005] To overcome the problems existing in the related technologies, the present disclosure provides an image blurring processing method, apparatus, device, storage medium, and program product to solve the defects in the related technologies.

[0006] According to a first aspect of the present disclosure, an image blurring processing method is provided, the method comprising:

[0007] In response to acquiring the current image to be processed, the focus information of the current image is determined;

[0008] The depth information of the current image is determined based on the focus information;

[0009] The current image is blurred based on the depth information.

[0010] In some embodiments, the focus information includes the phase difference of each of a preset number of image blocks in the current image, and the depth information includes the depth information of each of the preset number of image blocks.

[0011] In some embodiments, determining the focus information of the current image includes:

[0012] The first phase detection image and the second phase detection image are respectively divided into the preset number of image blocks. The first phase detection image and the second phase detection image include the phase detection image obtained by the phase detection pixels in the image sensor during the process of the image sensor acquiring the current image.

[0013] Based on the similarity between each image block in the first phase detection image and each image block in the second phase detection image, the phase difference between each image block in the first phase detection image is determined.

[0014] Based on the phase difference of each image block in the first phase detection image, the phase difference of image blocks at the same position in the current image is determined.

[0015] In some embodiments, determining the phase difference of each image patch in the first phase detection image based on the similarity between each image patch in the first phase detection image and each image patch in the second phase detection image includes:

[0016] For each first image block in the first phase detection image, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second phase detection image;

[0017] The similarity between each of the plurality of third image blocks and the first image block is determined respectively;

[0018] Based on the movement amount of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block, the phase difference of the first image block is determined, and the movement amount is the offset of the target image block relative to the second image block.

[0019] In some embodiments, determining the depth information of the current image based on the focus information includes:

[0020] The target position of each image block is determined based on the phase difference of each image block in the current image and the current motor position. The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image. The target position is the focus position corresponding to each image block.

[0021] The image distance offset of each image block is determined based on the target position of each image block and a first relationship between the predetermined different target positions and the corresponding image distance offsets.

[0022] The object distance of each image block is determined based on the image distance offset of each image block and a second relationship between different image distance offsets and corresponding object distances, and is used as the depth information of each image block.

[0023] In some embodiments, the method further includes determining the first relationship based on the following:

[0024] The object-image relationship of the optical system is transformed to obtain an expression for the image distance offset. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0025] Substituting the first object distance and the second object distance into the expression respectively, we obtain the first image distance offset corresponding to the first object distance and the second image distance offset corresponding to the second object distance. The first object distance and the second object distance are known, and the second object distance is the motor position at the hyperfocal distance where the imaging sharpness meets the requirements.

[0026] The slope of the first relationship is determined based on the ratio of the object distance difference between the second object distance and the first object distance to the image distance offset difference between the second image distance offset and the first image distance offset.

[0027] The first relationship is determined based on the slope, the second object distance, and the second image distance offset.

[0028] In some embodiments, the method further includes determining the second relationship based on the following:

[0029] The object-image relationship of the optical system is transformed to obtain the second relationship, which includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane, wherein the difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0030] In some embodiments, the background blurring process applied to the current image based on the depth information includes:

[0031] Identify the subject in the current image;

[0032] The area in the current image where the subject is located is defined as the subject area, and the area in the current image other than the subject area is defined as the non-subject area;

[0033] Based on the depth information of the non-subject region, the non-subject region is blurred.

[0034] In some embodiments, blurring the non-subject region based on its depth information includes:

[0035] Determine the depth difference between the depth information of each pixel in the non-subject region and the depth information of the focus of the current image;

[0036] The blur intensity of the corresponding pixel is determined based on the depth difference, and there is a positive correlation between the depth difference and the blur intensity of the same pixel;

[0037] The corresponding pixel is blurred based on the blur intensity.

[0038] According to a second aspect of the present disclosure, an image blurring processing apparatus is provided, the apparatus comprising:

[0039] The focus information determination module is used to determine the focus information of the current image in response to acquiring the current image to be processed;

[0040] A depth information determination module is used to determine the depth information of the current image based on the focus information;

[0041] The background blurring module is used to blur the background of the current image based on the depth information.

[0042] In some embodiments, the focus information includes the phase difference of each of a preset number of image blocks in the current image, and the depth information includes the depth information of each of the preset number of image blocks.

[0043] In some embodiments, the focus information determination module includes:

[0044] The detection image segmentation unit is used to divide the first phase detection image and the second phase detection image into the preset number of image blocks, wherein the first phase detection image and the second phase detection image include the phase detection image acquired by the phase detection pixels in the image sensor during the process of the image sensor acquiring the current image;

[0045] The detection phase determination unit is used to determine the phase difference of each image block in the first phase detection image based on the similarity between each image block in the first phase detection image and each image block in the second phase detection image;

[0046] The focus information determination unit is used to determine the phase difference of image blocks at the same position in the current image based on the phase difference of each image block in the first phase detection image.

[0047] In some embodiments, the detection phase determination unit is further configured to:

[0048] For each first image block in the first phase detection image, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second phase detection image;

[0049] The similarity between each of the plurality of third image blocks and the first image block is determined respectively;

[0050] Based on the movement amount of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block, the phase difference of the first image block is determined, and the movement amount is the offset of the target image block relative to the second image block.

[0051] In some embodiments, the depth information determination module includes:

[0052] The target position determination unit is used to determine the target position of each image block based on the phase difference of each image block in the current image and the current motor position. The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image. The target position is the focus position corresponding to each image block.

[0053] The image distance offset determination unit is used to determine the image distance offset of each image block based on the target position of each image block and a first relationship between the predetermined different target positions and the corresponding image distance offsets.

[0054] The depth information determination unit is used to determine the object distance of each image block based on the image distance offset of each image block and a second relationship between different image distance offsets and corresponding object distances, and to use this as the depth information of each image block.

[0055] In some embodiments, the apparatus further includes a first relationship determination module;

[0056] The first relationship determination module includes:

[0057] The expression determination unit is used to transform the form of the object-image relationship of the optical system to obtain an expression for the image distance offset. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0058] The offset determination unit is used to substitute the first object distance and the second object distance into the expression respectively to obtain the first image distance offset corresponding to the first object distance and the second image distance offset corresponding to the second object distance. The first object distance and the second object distance are known, and the second object distance is the motor position at the hyperfocal distance where the imaging sharpness meets the requirements.

[0059] The slope determination unit is used to determine the slope of the first relationship based on the ratio of the object distance difference between the second object distance and the first object distance to the image distance offset difference between the second image distance offset and the first image distance offset.

[0060] The relationship determination unit is used to determine the first relationship based on the slope, the second object distance, and the second image distance offset.

[0061] In some embodiments, the apparatus further includes:

[0062] The second relationship determination module is used to transform the form of the object-image relationship of the optical system to obtain the second relationship. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0063] In some embodiments, the background blurring processing module includes:

[0064] Subject recognition unit, used to identify the subject being photographed in the current image;

[0065] A region division unit is used to determine the region in the current image where the subject is located as the subject region, and to determine the region in the current image other than the subject region as the non-subject region;

[0066] The fuzzing unit is used to fuzz the non-subject region based on the depth information of the non-subject region.

[0067] In some embodiments, the blurring unit is further configured to:

[0068] Determine the depth difference between the depth information of each pixel in the non-subject region and the depth information of the focus of the current image;

[0069] The blur intensity of the corresponding pixel is determined based on the depth difference, and there is a positive correlation between the depth difference and the blur intensity of the same pixel;

[0070] The corresponding pixel is blurred based on the blur intensity.

[0071] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising:

[0072] Processor and memory used to store computer programs;

[0073] The processor is configured to implement the image blurring processing method described above when executing the computer program.

[0074] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the image blurring processing method described in any of the preceding claims.

[0075] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image blurring processing method described in any of the preceding claims.

[0076] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0077] This disclosure, in response to acquiring a current image to be processed, determines the focus information of the current image, and determines the depth information of the current image based on the focus information, and then performs background blurring processing on the current image based on the depth information. Since there is no need to use a variable aperture camera or dual cameras, the cost of the device can be reduced. Furthermore, there is no need to process images acquired by dual cameras, thus simplifying the image processing process, reducing the processing burden of the system, thereby improving image processing speed and reducing device power consumption.

[0078] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0079] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0080] Figure 1 This is a flowchart illustrating an image blurring process according to an exemplary embodiment of the present disclosure;

[0081] Figure 2A This is a flowchart illustrating how to determine the focus information of the current image according to an exemplary embodiment of the present disclosure;

[0082] Figure 2B This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to an exemplary embodiment of the present disclosure;

[0083] Figure 2C This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to yet another exemplary embodiment of the present disclosure;

[0084] Figure 2D This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to yet another exemplary embodiment of the present disclosure;

[0085] Figure 3A This is a flowchart illustrating how to determine the phase difference of each image block in the first phase detection image according to an exemplary embodiment of the present disclosure;

[0086] Figure 3B This is a schematic diagram of a cost-k curve according to yet another exemplary embodiment of the present disclosure;

[0087] Figure 4A This is a flowchart illustrating how to determine the depth information of the current image based on the focus information, according to an exemplary embodiment of this disclosure;

[0088] Figure 4B This is a schematic diagram of the focus position in an image according to an exemplary embodiment of the present disclosure;

[0089] Figure 5 This is a flowchart illustrating how the first relationship is determined according to an exemplary embodiment of this disclosure;

[0090] Figure 6A This is a flowchart illustrating how to perform background blurring processing on the current image based on the depth information, according to an exemplary embodiment of this disclosure;

[0091] Figure 6B This is a schematic diagram illustrating the background blurring effect of a current image according to an exemplary embodiment of the present disclosure;

[0092] Figure 7A This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to blur the non-subject region based on the depth information of the non-subject region;

[0093] Figure 7B This is a schematic diagram illustrating the relationship between the depth difference of pixels and the blur intensity according to an exemplary embodiment of the present disclosure;

[0094] Figure 8 This is a block diagram illustrating an image blurring processing apparatus according to an exemplary embodiment of the present disclosure;

[0095] Figure 9 This is a block diagram illustrating yet another image blurring processing apparatus according to an exemplary embodiment of the present disclosure;

[0096] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0097] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0098] Figure 1 This is a flowchart illustrating an image blurring processing method according to an exemplary embodiment. The method of this embodiment can be executed by an image blurring processing device, which can be configured in an electronic device, such as a server, workstation, personal computer, mobile terminal (e.g., mobile phone, tablet computer), wearable device (e.g., glasses, watch), etc. Figure 1 As shown, the method includes the following steps S101-S103:

[0099] In step S101, in response to acquiring the current image to be processed, the focus information of the current image is determined.

[0100] In this embodiment, when the electronic device acquires the current image to be processed (e.g., an image acquired by the camera of the electronic device), the focus information of the current image can be determined.

[0101] The focus information of the current image may include the focus information used by the camera of the electronic device when capturing the image.

[0102] For example, the electronic device can focus based on phase detection autofocus (PDAF) during the process of capturing the image; furthermore, the aforementioned focus information can include the phase detection image (PD Raw) acquired by the phase detection pixels in the image sensor during the process of the camera's image sensor acquiring the current image.

[0103] Alternatively, the focus information of the current image can be obtained by processing the phase detection image described above, such as the phase difference (PD Value) of a preset number of image blocks in the current image. The following explanation uses this phase difference as the focus information. The current image is divided into blocks in the same way as the phase detection image, for example, divided into M*N blocks, where M and N are positive integers, and their values ​​can be set according to the image size.

[0104] In step S102, the depth information of the current image is determined based on the focus information.

[0105] In this embodiment, after determining the focus information of the current image, the depth information of the current image can be determined based on the focus information.

[0106] In computer vision, focus information and depth information are closely related. Focus information is typically used to indicate which parts of an image are in focus, while depth information indicates the distance of objects in a scene from the camera. When an object is accurately focused, it is usually within the so-called "depth of field." Depth of field refers to the range of distances in front of and behind the focal point that remain sufficiently sharp. In this embodiment, techniques from computational photography and computer vision are used to derive the relationship between focus information and depth information for the same image. Therefore, once the focus information of the current image is determined, this relationship and the focus information can be used to determine the depth information of the current image.

[0107] Since this solution does not rely on a variable aperture camera or use two images (main and secondary cameras) to calculate image depth information as in related technologies, but instead uses the focus information of the main camera image to obtain image depth information, the hardware cost of implementing this solution can be reduced. Furthermore, during image processing, there is no need to process two images (main and secondary camera images). Instead, the focus information used in the current image acquisition process is converted into depth information, which can greatly reduce the computational overhead of the system, thereby improving the image output speed and reducing the power consumption of the device.

[0108] In other embodiments, the method for determining the depth information of the current image based on the focus information described above can be found in the following: Figure 2A The embodiments shown will not be described in detail here.

[0109] In step S103, the current image is subjected to background blurring based on the depth information.

[0110] In this embodiment, after determining the depth information of the current image based on the focus information, the current image can be blurred based on the depth information.

[0111] After determining the depth information of the current image, the background of the current image can be blurred according to the background blurring processing method in the relevant technology. This embodiment does not limit the background blurring processing method described above.

[0112] In other embodiments, the above-described method of blurring the background of the current image can also be found in the following: Figure 6A The embodiments shown will not be described in detail here.

[0113] As described above, the method of this embodiment determines the focus information of the current image in response to acquiring the current image to be processed, and determines the depth information of the current image based on the focus information. Then, it performs background blurring processing on the current image based on the depth information. Since there is no need to use a variable aperture camera or dual cameras, the cost of the device can be reduced. Moreover, there is no need to process the images acquired by dual cameras, which simplifies the image processing process, reduces the processing burden of the system, thereby improving the image processing speed and reducing the power consumption of the device.

[0114] Figure 2A This is a flowchart illustrating how to determine the focus information of the current image according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to determine the focus information of the current image as an example for illustrative explanation.

[0115] In this embodiment, the focus information of the current image includes the phase difference of each of a preset number of image blocks in the current image, and the depth information includes the depth information of each of the preset number of image blocks.

[0116] like Figure 2A As shown, determining the focus information of the current image in step S101 above may include the following steps S201-S203:

[0117] In step S201, the first phase detection image and the second phase detection image are respectively divided into the preset number of image blocks.

[0118] In this embodiment, after the first phase detection image and the second phase detection image are obtained during the acquisition of the current image, the first phase detection image and the second phase detection image can be divided into the preset number of image blocks respectively.

[0119] The first phase detection image and the second phase detection image mentioned above can be phase detection images acquired by phase detection pixels (such as PD pixels) in the image sensor during the process of the image sensor of the electronic device acquiring the current image.

[0120] For example, Figure 2B This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to an exemplary embodiment of the present disclosure.

[0121] Figure 2C This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to yet another exemplary embodiment of this disclosure. Figure 2D This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to another exemplary embodiment of the present disclosure.

[0122] like Figures 2B to 2DAs shown, the image sensor's photosensitive element has pre-set left and right PD pixels (i.e., Figure 2B (The pixels are indicated by "L" and "R" in the diagram). The phase detection image output from all L pixels is called the L-image. Based on the density of the PD pixel distribution, photosensitive elements can be divided into two types: dense PD sensors and sparse PD sensors. For example, Figure 2B The image shows the pixel distribution of a dense PD sensor. Figure 2C and Figure 2D The diagram illustrates two different PD pixel distributions in a sparse PD sensor. It's understood that a dense PD sensor uses more PD points to calculate the phase difference, resulting in higher accuracy of the calculated phase value, but with a larger computational load; while a sparse PD sensor uses fewer PD points to calculate the phase difference, resulting in lower accuracy of the calculated phase value, but with a smaller computational load. Therefore, either a dense PD sensor or a sparse PD sensor can be selected based on the application requirements; this embodiment does not impose any limitations on this.

[0123] In some embodiments, the first phase detection image (e.g., "L image") and the second phase detection image (e.g., "R image") can be divided into the predetermined number of image blocks, for example, into N*M blocks (i.e., the image has N rows and M columns of image blocks), where M and N are positive integers, and their values ​​can be set according to the size of the image. For example, assuming the image size is 4000*3000, the image can typically be divided into 30*30 image blocks.

[0124] In step S202, the phase difference of each image block in the first phase detection image is determined based on the similarity between each image block in the first phase detection image and each image block in the second phase detection image.

[0125] In this embodiment, after the first phase detection image and the second phase detection image are respectively divided into the preset number of image blocks, the phase difference of each image block in the first phase detection image can be determined based on the similarity between each image block in the first phase detection image and each image block in the second phase detection image.

[0126] In some embodiments, the similarity between two different image patches can be measured based on the SAD (Sum of Absolute Differences) algorithm. For example, the similarity C between two image patches can be set to the negative of the sum of absolute differences.

[0127] In other embodiments, the method for determining the phase difference of each image block in the first phase detection image may also be described below. Figure 3A The embodiments shown will not be described in detail here.

[0128] In step S203, the phase difference of image blocks at the same position in the current image is determined based on the phase difference of each image block in the first phase detection image.

[0129] In this embodiment, after determining the phase difference of each image block in the first phase detection image, the phase difference of the image blocks at the same position in the current image can be determined based on the phase difference of each image block in the first phase detection image.

[0130] For example, after determining the phase difference of each image block in the first phase detection image, the phase difference of each image block in the first phase detection image can be determined as the phase difference of image blocks at the same position in the current image. For instance, when the phase difference of the image block in the i-th row and j-th column of the first phase image is determined to be d, the phase difference of the image block in the i-th row and j-th column of the current image can be set to d. Similarly, the phase difference of each image block in the current image can be determined.

[0131] As described above, this embodiment divides the first phase detection image and the second phase detection image into a preset number of image blocks, and determines the phase difference of each image block in the first phase detection image based on the similarity between each image block in the first phase detection image and each image block in the second phase detection image. Then, based on the phase difference of each image block in the first phase detection image, the phase difference of image blocks at the same position in the current image is determined. This can accurately determine the focus information of the current image, and subsequently determine the depth information of the current image based on the focus information, and perform background blurring processing on the current image based on the depth information. This can improve the accuracy of determining the depth information of the current image, reduce the cost of the device, simplify the image processing process, reduce the processing burden of the system, thereby improving the image processing speed and reducing the power consumption of the device.

[0132] Figure 3A This is a flowchart illustrating how to determine the phase difference of each image block in the first phase detection image according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to determine the phase difference of each image block in the first phase detection image as an example for illustrative explanation.

[0133] like Figure 3AAs shown, the step S202 above, which determines the phase difference of each image patch in the first phase detection image based on the similarity between each image patch in the first phase detection image and each image patch in the second phase detection image, may include the following steps S301-S303:

[0134] In step S301, for each first image block in the first phase detection image, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second phase detection image.

[0135] In this embodiment, the first phase detection image is still taken as the L-image and the second phase detection image is taken as the R-image for illustration.

[0136] The aforementioned preset direction can be the row direction of the second phase detection image (e.g., the horizontal direction of the second phase detection image). The aforementioned preset range can be the range [-h, +h], where the value of h depends on the motor formation.

[0137] For example, when the first image block of the first phase detection image is represented as I L When (x+i, y+j), the second image patch at the same position in the second phase detection image can be represented as I. R (x+i,y+j), and thus the third image block in the row direction of the second image block within the range [-h,+h] can be represented as I. R (x+i+k,y+j). Where I represents the image, x+i and y+j are the x and y coordinates of the corresponding image patch in the image, respectively, and k is the offset, which is in the range [-h,+h].

[0138] In step S302, the similarity between each of the plurality of third image blocks and the first image block is determined.

[0139] In this embodiment, after acquiring multiple third image blocks, the similarity between each of the multiple third image blocks and the first image block can be determined. Taking the similarity as the inverse of the absolute difference and CAD as an example, the similarity Cost between each of the multiple third image blocks and the first image block can be determined based on the following formula (3-1):

[0140]

[0141] In the above formula, a = w / M, where w is the width of the image and M is the number of columns in the image block; b = h / N, where h is the height of the image and N is the number of rows in the image block.

[0142] In other words, the image patch in the R image moves left and right relative to the image patch in the L image in the image row direction, and the corresponding similarity Cost is calculated.

[0143] In step S303, the phase difference of the first image block is determined based on the amount of movement of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block.

[0144] In this embodiment, after determining the similarity between each of the plurality of third image blocks and the first image block, the movement amount of the target image block whose similarity with the first image block meets a preset condition (e.g., highest similarity) can be determined, and the phase difference of the first image block can be determined based on this movement amount. The movement amount is the offset of the target image block relative to the second image block.

[0145] For example, after determining the similarity between each of the plurality of third image patches and the first image patch, a Cost-k curve (i.e., a curve showing the cost as a function of k) can be plotted, such as... Figure 3B As shown. Based on this, the image block to which the peak point P1 of the curve (i.e., the point with the highest similarity) belongs can be determined as the target image block that meets the above preset conditions (i.e., the most similar matching block between the R image and the L image). Then, the amount of movement of the target image block, that is, the horizontal coordinate k1 of the peak point P1, is used as the phase difference PD value (unit: pixel) corresponding to the first image block in the R image.

[0146] As described above, this embodiment obtains multiple third image blocks within a preset direction and preset range for each first image block in the first phase detection image and a second image block at the same position in the second phase detection image. It then determines the similarity between each of the multiple third image blocks and the first image block, and based on the movement of the target image block whose similarity to the first image block satisfies a preset condition, determines the phase difference of the first image block. This allows for accurate determination of the phase difference of each image block in the first phase detection image, and subsequently, based on the phase difference of each image block in the first phase detection image, accurately determining the phase difference of image blocks at the same position in the current image.

[0147] Figure 4A This is a flowchart illustrating how to determine the depth information of the current image based on the focus information, according to an exemplary embodiment of the present disclosure. This embodiment is based on the above embodiment and takes how to determine the depth information of the current image based on the focus information as an example for illustrative explanation.

[0148] like Figure 4A As shown, the step S102 above, which involves determining the depth information of the current image based on the focus information, may include the following steps S401-S403:

[0149] In step S401, the target position of each image block is determined based on the phase difference of each image block in the current image and the current motor position.

[0150] In this embodiment, when determining the depth information of the current image based on the focus information, the target position of each image block can be determined first based on the phase difference of each image block in the current image and the current motor position.

[0151] The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image, and the target position is the focus position corresponding to each image block.

[0152] For example, Figure 4B This is a schematic diagram of the focus position in an image according to an exemplary embodiment of the present disclosure; as shown below. Figure 4B As shown, when the object in the frame is in focus, such as Figure 4B When the image is at the position corresponding to the gray solid line of the clearest image on the left (the third image from the top on the left), the L image block and the R image block overlap, with a phase difference of 0, and this current position is the target position. However, when the object in the frame is not in focus, the L image block and the R image block will have a left-right deviation; this deviation is the phase difference. Furthermore, the farther the object in the frame is from the focus position, the greater the phase difference between the corresponding L image block and the R image block. Therefore, after obtaining the current motor position (i.e., the motor position at the time of obtaining the PD raw image) and the phase difference of each image block in the current image, the target position of each image block can be determined. For example, the target position of each image block can be determined based on the following formula (4-1):

[0153] Targetpos=Current_pos+PD_value*DCC

[0154] (4-1)

[0155] In the above formula, Target pos The target position of the current image patch is defined by Current_pos, the focus position of the current image patch is defined by Current_pos, the phase difference of the current image patch is defined by PD_value, and DCC is the unit conversion (unit: dac / pixel, DCC is a calibrable parameter).

[0156] In step S402, the image distance offset of each image block is determined according to the target position of each image block and the first relationship between the predetermined different target positions and the corresponding image distance offset.

[0157] In this embodiment, after determining the target position of each image block, the image distance offset of each image block can be determined according to the target position of each image block and a first relationship between the predetermined different target positions and the corresponding image distance offsets. For example, the first relationship can be in the form of the following equation (4-2):

[0158] Δtarget=n+λ*(Target pos -m)

[0159] (4-2)

[0160] In the above formula, Δtarget is the image distance offset of the current image block, and m, n and λ are pre-calibrated constants.

[0161] It is worth noting that the calibration methods for m, n, and λ mentioned above (i.e., the methods for determining the first relationship) can be derived from the object-image relationship of the optical system. The derivation process can be found in relevant technical records or in the following text. Figure 5 The embodiments shown will not be described in detail here.

[0162] In step S403, the object distance of each image block is determined based on the image distance offset of each image block and the second relationship between different image distance offsets and the corresponding object distance, and is used as the depth information of each image block.

[0163] In this embodiment, after determining the image distance offset Δtarget of each image block, the object distance of each image block can be determined based on the image distance offset of each image block and the second relationship between different image distance offsets and corresponding object distances, and used as the depth information of each image block.

[0164] In some embodiments, the second relationship can be obtained by transforming the form of the object-image relationship of the optical system.

[0165] The object-image relationship includes the relationship between the lens focal length f (known), the object distance u, and the distance v from the lens to the image plane. For example, this object-image relationship can be represented by the following equation (4-3):

[0166]

[0167] Based on this, we can transform equation (4-3) to obtain equation (4-4):

[0168]

[0169] In equation (4-4) above, Δ=vf represents the image distance offset.

[0170] Furthermore, the object distance u (i.e., the depth information distance) can be derived from equation (4-4) above. target The expression for ) is shown in equation (4-5) below:

[0171]

[0172] Therefore, once the image distance offset Δtarget corresponding to the current image patch is determined, it can be substituted into equation (4-5) above to obtain the depth information distance of the current image patch. target .

[0173] As described above, this embodiment determines the target position of each image block based on the phase difference of each image block in the current image and the current motor position. It then determines the image distance offset of each image block based on the target position of each image block and a first relationship between different predetermined target positions and corresponding image distance offsets. Finally, it determines the object distance of each image block based on the image distance offset of each image block and a second relationship between different predetermined image distance offsets and corresponding object distances, using this object distance as the depth information of each image block. This allows for accurate determination of the depth information of the current image based on the focus information, enabling subsequent background blurring processing of the current image based on the depth information. Since a variable aperture camera and dual cameras are not required, the cost of the device can be reduced. Furthermore, the image processing process can be simplified by eliminating the need to process images acquired by dual cameras, reducing the system's processing burden and thus improving image processing speed and reducing device power consumption.

[0174] Figure 5 This is a flowchart illustrating how to determine the first relationship according to an exemplary embodiment of the present disclosure; this embodiment is an exemplary description based on the above embodiment, taking how to determine the first relationship as an example.

[0175] like Figure 5 As shown, the image blurring processing method of this embodiment may further include determining the first relationship based on the following steps S501-S504:

[0176] In step S501, the form of the object-image relationship of the optical system is transformed to obtain an expression for the image distance offset.

[0177] In this embodiment, when determining the first relationship between different target positions and the corresponding image distance offset, the form of the object-image relationship of the optical system can be transformed to obtain the expression for the image distance offset.

[0178] The above-mentioned object-image relationship includes the relationship between the lens focal length, object distance, and the distance from the lens to the image plane, wherein the difference between the distance from the lens to the image plane and the focal length is the image distance offset. For example, this object-image relationship can be represented by the following equation (5-1):

[0179]

[0180] Based on this, equation (5-1) can be transformed to obtain the expression for the image distance offset, as shown in equation (5-2) below:

[0181]

[0182] In equation (5-2) above, Δ=vf represents the image distance offset.

[0183] In step S502, the first object distance and the second object distance are substituted into the expression respectively to obtain the first image distance offset corresponding to the first object distance and the second image distance offset corresponding to the second object distance.

[0184] Once the expression for the image distance offset is obtained, the first object distance and the second object distance can be substituted into the expression to obtain the first image distance offset corresponding to the first object distance and the second image distance offset corresponding to the second object distance. Here, the first object distance and the second object distance are known, and the second object distance is the motor position at the hyperfocal distance where the imaging sharpness meets the requirements.

[0185] For example, the first object distance, distance1, can be the motor position, position1 (unit: dac, which can be obtained through calibration), at 20cm where the image is clearest. The corresponding image distance offset, Δ1, can be represented by the following formula (5-3):

[0186]

[0187] The second object distance, distance2, can be the motor position, position2 (unit: dac, which can be obtained through calibration), at the hyperfocal distance where the image is clearest. The corresponding image distance offset, Δ2, can be represented by the following formula (5-4):

[0188]

[0189] It is understandable that "hyperfocal distance" means focusing on a point at a distance so that the other extreme of the depth of field is "infinity". Then, the distance from "infinity (∞)" to the nearest photographic distance within the depth of field is the distance from "infinity (∞)".

[0190] And distance2 can be calculated from the optical coefficients shown in (5-5):

[0191]

[0192] Among them, F n Let F be the aperture value, and p be the pixel size (known). The F value can be calculated as shown in equation (5-6):

[0193] F-number = Lens focal length (mm) / Aperture diameter (mm)

[0194] (5-6)

[0195] In step S503, the slope of the first relationship is determined based on the ratio of the object distance difference between the second object distance and the first object distance to the image distance offset difference between the second image distance offset and the first image distance offset.

[0196] In this embodiment, since there is a linear relationship between Δ1 and Δ2 and position1 and position2, the slope of the first relationship can be determined based on the following equation (5-7):

[0197]

[0198] In step S504, the first relationship is determined based on the slope, the second object distance, and the second image distance offset.

[0199] Once the slope is determined, if the target position Targetpos of the object is known, the corresponding offset Δtarget can be calculated. The determined first relationship is shown in equation (5-8) below:

[0200] Δtarget = Δ2 + Slope * (Target) pos -position2)

[0201] (5-8)

[0202] As described above, this embodiment transforms the form of the object-image relationship of the optical system to obtain an expression for the image distance offset. Then, the first object distance and the second object distance are substituted into the expression to obtain the first image distance offset corresponding to the first object distance and the second image distance offset corresponding to the second object distance. Based on the ratio of the object distance difference between the second object distance and the first object distance to the image distance offset difference between the second image distance offset and the first image distance offset, the slope of the first relationship is determined. Then, based on the slope, the second object distance, and the second image distance offset, the first relationship is determined. This can accurately determine the first relationship between different target positions and the corresponding image distance offsets, and subsequently, based on the target position of each image block and the first relationship, the image distance offset of each image block can be determined.

[0203] Figure 6A This is a flowchart illustrating how to perform background blurring processing on the current image based on the depth information, according to an exemplary embodiment of this disclosure. This embodiment is an exemplary description based on the above embodiment, using the example of how to perform background blurring processing on the current image based on the depth information.

[0204] like Figure 6A As shown, the background blurring process of the current image based on the depth information described in step S103 above may include the following steps S601-S603:

[0205] In step S601, the subject being photographed is identified in the current image.

[0206] In this embodiment, when the current image is blurred based on the depth information, the subject being photographed can be identified in the current image first.

[0207] Taking a portrait as the subject, Figure 6B This is a schematic diagram illustrating the background blurring effect of a current image according to an exemplary embodiment of this disclosure. When the camera module of an electronic device is in "portrait mode," i.e., when portrait background blurring is required, the portrait being photographed can first be identified in the current image based on a portrait recognition algorithm in related technologies, such as... Figure 6B As shown.

[0208] It is understood that the subject of the above-mentioned shooting can be not only human figures, but also buildings, animals, flowers and trees, etc., and then the subject of the shooting can be identified from the current image by the matching object recognition algorithm in related technologies. This embodiment does not limit this.

[0209] In step S602, the area in the current image where the subject is located is determined as the subject area, and the area in the current image other than the subject area is determined as the non-subject area.

[0210] In this embodiment, after identifying the subject in the current image, the area in the current image where the subject is located can be determined as the subject area, and the remaining areas, that is, the areas in the current image other than the subject area, can be determined as non-subject areas.

[0211] In step S603, the non-subject region is blurred based on its depth information.

[0212] In this embodiment, after the current image is divided into a subject area and a non-subject area, the non-subject area can be regarded as the background of the subject being photographed, and the depth information of the non-subject area can be combined to blur (blur) it.

[0213] Depend on Figure 6B As can be seen, the background area outside the portrait area of ​​the image is blurred after processing. Pixels closer to the subject in the background area are clearer, while pixels farther away from the subject are more blurred.

[0214] It is worth noting that the method of blurring a region (non-subject region) based on the depth information of a certain region in the image can be found in the blurring processing schemes in related technologies, and this embodiment does not limit it.

[0215] In other embodiments, the method of blurring the non-subject region based on its depth information can also be described below. Figure 7A The embodiments shown will not be described in detail here.

[0216] As described above, this embodiment identifies the subject in the current image, defines the area where the subject is located as the subject area, and defines the areas in the current image other than the subject area as non-subject areas. Then, based on the depth information of the non-subject areas, the non-subject areas are blurred. This allows for background blurring of the current image based on the depth information. Since the depth information used is determined based on the focus information of the current image, there is no need to use a variable aperture camera or dual cameras, thus reducing the cost of the device. Furthermore, there is no need to process images captured by dual cameras, which simplifies the image processing process, reduces the processing burden of the system, and thereby improves image processing speed and reduces device power consumption.

[0217] Figure 7AThis is a flowchart illustrating how to blur the non-subject region based on the depth information of the non-subject region according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to blur the non-subject region based on the depth information of the non-subject region as an example for illustrative explanation.

[0218] like Figure 7A As shown, the blurring process of the non-subject region based on the depth information of the non-subject region in step S603 above may include the following steps S701-S703:

[0219] In step S701, the depth difference between the depth information of each pixel in the non-subject region and the depth information of the focus of the current image is determined.

[0220] In this embodiment, after determining the non-subject region in the current image and the depth information of that region, the depth difference between the depth information of each pixel in the non-subject region and the depth information of the focal point of the current image can be determined.

[0221] The focus of the current image can refer to the position of the focal plane where the subject (such as a human figure) is located.

[0222] In step S702, the blur intensity of the corresponding pixel is determined based on the depth difference.

[0223] In this embodiment, after determining the depth difference between the depth information of each pixel in the non-subject area and the depth information of the focus of the current image, the blur intensity of the corresponding pixel can be determined based on the depth difference.

[0224] Among these, the depth difference of the same pixel is positively correlated with the blur intensity. For example, Figure 7B This is a schematic diagram illustrating the relationship between the depth difference of pixels and the blur intensity according to an exemplary embodiment of this disclosure; as shown below. Figure 7B As shown, the greater the difference between the depth information of a pixel and the depth information of the focal point, the greater the corresponding blur intensity; conversely, the smaller the difference between the depth information of a pixel and the depth information of the focal point, the smaller the corresponding blur intensity. It is worth noting that... Figure 7B The non-linear curves shown are for illustrative purposes only and are not intended to limit the scope of this embodiment.

[0225] In some embodiments, the methods for blurring non-subject regions include, but are not limited to, Gaussian blurring, and the blur intensity can refer to the blur radius of the Gaussian blur.

[0226] In step S703, the corresponding pixel is blurred based on the blur intensity.

[0227] In this embodiment, after determining the blur intensity of the pixel based on the depth difference, the pixel can be blurred based on the blur intensity.

[0228] Taking Gaussian blur as an example, a larger blur radius means that the pixel's value is influenced by pixels at greater distances. In other words, the larger the blur radius of a pixel, the more blurred it becomes, because the new value obtained after blurring is based on the average value calculated from pixels in a larger neighborhood.

[0229] As described above, this embodiment determines the depth difference between the depth information of each pixel in the non-subject area and the depth information of the focus of the current image, and determines the blur intensity of the corresponding pixel based on the depth difference. Then, it performs blurring processing on the corresponding pixel based on the blur intensity. This can achieve accurate blurring processing of the non-subject area based on the depth information of the non-subject area. Since the depth information used is determined based on the focus information of the current image, there is no need to use a variable aperture camera or dual cameras, thus reducing the cost of the device. Furthermore, there is no need to process images captured by dual cameras, thus simplifying the image processing process, reducing the processing burden of the system, thereby improving the image processing speed and reducing the power consumption of the device.

[0230] Figure 8 This is a block diagram illustrating an image blurring processing apparatus according to an exemplary embodiment of the present disclosure; the apparatus of this embodiment can be configured in electronic devices, such as servers, workstations, personal computers, mobile terminals (such as mobile phones, tablets, etc.), wearable devices (such as glasses, watches, etc.). Figure 8 As shown, the device may include: a focus information determination module 110, a depth information determination module 120, and a background blur processing module 130, wherein:

[0231] The focus information determination module 110 is used to determine the focus information of the current image in response to acquiring the current image to be processed;

[0232] The depth information determination module 120 is used to determine the depth information of the current image based on the focus information;

[0233] Background blurring processing module 130 is used to perform background blurring processing on the current image based on the depth information.

[0234] As described above, the device in this embodiment determines the focus information of the current image in response to acquiring the current image to be processed, and determines the depth information of the current image based on the focus information. Then, it performs background blurring processing on the current image based on the depth information. Since there is no need to use a variable aperture camera or dual cameras, the cost of the device can be reduced. Furthermore, there is no need to process images acquired by dual cameras, which simplifies the image processing process, reduces the processing burden of the system, thereby improving the image processing speed and reducing the power consumption of the device.

[0235] Figure 9 This is a block diagram illustrating another image blurring processing apparatus according to an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be configured in electronic devices, such as servers, workstations, personal computers, mobile terminals (e.g., mobile phones, tablets), wearable devices (e.g., glasses, watches), etc. The focus information determination module 210, depth information determination module 220, and background blurring processing module 230 are as described above. Figure 8 The focus information determination module 110, depth information determination module 120, and background blur processing module 130 in the illustrated embodiment have the same functions, and will not be described in detail here.

[0236] In this embodiment, the focus information may include the phase difference of each of a preset number of image blocks in the current image, and the depth information may include the depth information of each of the preset number of image blocks.

[0237] In some embodiments, the focus information determination module 210 described above may include:

[0238] The detection image segmentation unit 211 is used to divide the first phase detection image and the second phase detection image into the preset number of image blocks respectively. The first phase detection image and the second phase detection image include the phase detection image obtained by the phase detection pixels in the image sensor during the process of the image sensor acquiring the current image.

[0239] The detection phase determination unit 212 is used to determine the phase difference of each image block in the first phase detection image based on the similarity between each image block in the first phase detection image and each image block in the second phase detection image;

[0240] The focus information determination unit 213 is used to determine the phase difference of image blocks at the same position in the current image based on the phase difference of each image block in the first phase detection image.

[0241] In some embodiments, the detection phase determination unit 212 described above can also be used for:

[0242] For each first image block in the first phase detection image, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second phase detection image;

[0243] The similarity between each of the plurality of third image blocks and the first image block is determined respectively;

[0244] Based on the movement amount of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block, the phase difference of the first image block is determined, and the movement amount is the offset of the target image block relative to the second image block.

[0245] In some embodiments, the depth information determination module 220 described above may include:

[0246] The target position determination unit 221 is used to determine the target position of each image block based on the phase difference of each image block in the current image and the current motor position. The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image. The target position is the focus position corresponding to each image block.

[0247] Image distance offset determination unit 222 is used to determine the image distance offset of each image block according to the target position of each image block and a first relationship between different target positions and corresponding image distance offsets.

[0248] The depth information determination unit 223 is used to determine the object distance of each image block based on the image distance offset of each image block and a second relationship between different image distance offsets and corresponding object distances, and use it as the depth information of each image block.

[0249] In some embodiments, the above-described apparatus may further include a first relationship determination module 240;

[0250] The first relationship determination module 240 may include:

[0251] The expression determination unit 241 is used to transform the form of the object-image relationship of the optical system to obtain an expression for the image distance offset. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0252] The offset determination unit 242 is used to substitute the first object distance and the second object distance into the expression respectively to obtain the first image distance offset corresponding to the first object distance and the second image distance offset corresponding to the second object distance. The first object distance and the second object distance are known, and the second object distance is the motor position at the hyperfocal distance where the imaging sharpness meets the requirements.

[0253] The slope determination unit 243 is used to determine the slope of the first relationship based on the ratio of the object distance difference between the second object distance and the first object distance to the image distance offset difference between the second image distance offset and the first image distance offset.

[0254] The relationship determination unit 244 is used to determine the first relationship based on the slope, the second object distance, and the second image distance offset.

[0255] In some embodiments, the above-described apparatus may further include:

[0256] The second relationship determination module 250 is used to transform the form of the object-image relationship of the optical system to obtain the second relationship. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane, wherein the difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0257] In some embodiments, the background blurring processing module 230 described above may include:

[0258] Subject recognition unit 231 is used to identify the subject being photographed in the current image;

[0259] The region division unit 232 is used to determine the region in the current image where the subject is located as the subject region, and to determine the region in the current image other than the subject region as the non-subject region.

[0260] The fuzzing unit 233 is used to fuzz the non-subject region based on the depth information of the non-subject region.

[0261] In some embodiments, the fuzzing unit 233 described above can also be used for:

[0262] Determine the depth difference between the depth information of each pixel in the non-subject region and the depth information of the focus of the current image;

[0263] The blur intensity of the corresponding pixel is determined based on the depth difference, and there is a positive correlation between the depth difference and the blur intensity of the same pixel;

[0264] The corresponding pixel is blurred based on the blur intensity.

[0265] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0266] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0267] Reference Figure 10 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0268] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the image blurring processing method described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0269] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0270] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0271] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0272] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0273] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0274] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 can detect the on / off state of device 900, the relative positioning of components such as the display panel and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0275] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0276] In an exemplary embodiment, device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image blurring processing method described above.

[0277] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of the device 900 to complete the image blurring processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0278] In an exemplary embodiment, a computer program product including instructions is also provided, which can be executed by the processor 920 of the device 900 to perform the image blurring processing method described above.

[0279] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0280] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image blurring processing method, characterized in that, The method includes: In response to acquiring the current image to be processed, the focus information of the current image is determined; The depth information of the current image is determined based on the focus information; The current image is blurred based on the depth information.

2. The method according to claim 1, characterized in that, The focus information includes the phase difference of a preset number of image blocks in the current image, and the depth information includes the depth information of the preset number of image blocks.

3. The method according to claim 2, characterized in that, Determining the focus information of the current image includes: The first phase detection image and the second phase detection image are respectively divided into the preset number of image blocks. The first phase detection image and the second phase detection image include the phase detection image obtained by the phase detection pixels in the image sensor during the process of the image sensor acquiring the current image. Based on the similarity between each image block in the first phase detection image and each image block in the second phase detection image, the phase difference between each image block in the first phase detection image is determined. Based on the phase difference of each image block in the first phase detection image, the phase difference of image blocks at the same position in the current image is determined.

4. The method according to claim 3, characterized in that, The step of determining the phase difference of each image patch in the first phase detection image based on the similarity between each image patch in the first phase detection image and each image patch in the second phase detection image includes: For each first image block in the first phase detection image, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second phase detection image; The similarity between each of the plurality of third image blocks and the first image block is determined respectively; Based on the movement amount of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block, the phase difference of the first image block is determined, and the movement amount is the offset of the target image block relative to the second image block.

5. The method according to claim 3, characterized in that, Determining the depth information of the current image based on the focus information includes: The target position of each image block is determined based on the phase difference of each image block in the current image and the current motor position. The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image. The target position is the focus position corresponding to each image block. The image distance offset of each image block is determined based on the target position of each image block and a first relationship between the predetermined different target positions and the corresponding image distance offsets. The object distance of each image block is determined based on the image distance offset of each image block and a second relationship between different image distance offsets and corresponding object distances, and is used as the depth information of each image block.

6. The method according to claim 5, characterized in that, The method further includes determining the first relationship based on the following: The object-image relationship of the optical system is transformed to obtain an expression for the image distance offset. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset. Substituting the first object distance and the second object distance into the expression respectively, we obtain the first image distance offset corresponding to the first object distance and the second image distance offset corresponding to the second object distance. The first object distance and the second object distance are known, and the second object distance is the motor position at the hyperfocal distance where the imaging sharpness meets the requirements. The slope of the first relationship is determined based on the ratio of the object distance difference between the second object distance and the first object distance to the image distance offset difference between the second image distance offset and the first image distance offset. The first relationship is determined based on the slope, the second object distance, and the second image distance offset.

7. The method according to claim 5, characterized in that, The method further includes determining the second relationship based on the following: The object-image relationship of the optical system is transformed to obtain the second relationship, which includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane, wherein the difference between the distance from the lens to the image plane and the focal length is the image distance offset.

8. The method according to claim 1, characterized in that, The background blurring process of the current image based on the depth information includes: Identify the subject in the current image; The area in the current image where the subject is located is defined as the subject area, and the area in the current image other than the subject area is defined as the non-subject area; Based on the depth information of the non-subject region, the non-subject region is blurred.

9. The method according to claim 8, characterized in that, The process of blurring the non-subject region based on its depth information includes: Determine the depth difference between the depth information of each pixel in the non-subject region and the depth information of the focus of the current image; The blur intensity of the corresponding pixel is determined based on the depth difference, and there is a positive correlation between the depth difference and the blur intensity of the same pixel; The corresponding pixel is blurred based on the blur intensity.

10. An image blurring processing device, characterized in that, The device includes: The focus information determination module is used to determine the focus information of the current image in response to acquiring the current image to be processed; A depth information determination module is used to determine the depth information of the current image based on the focus information; The background blurring module is used to blur the background of the current image based on the depth information.

11. An electronic device, characterized in that, The device includes: Processor and memory used to store computer programs; The processor is configured to implement the image blurring processing method according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image blurring processing method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image blurring processing method according to any one of claims 1 to 9.

Images