Face focusing method, device, equipment, storage medium and program product

Abstract

The present disclosure relates to a face focusing method, device, equipment, storage medium and program product. According to an example of the present disclosure, the method comprises: in response to determining a face frame of a face in a current framing image, dividing an image region within the face frame into a plurality of image blocks; eliminating image blocks containing a background from the plurality of image blocks to obtain remaining image blocks; determining a focusing position based on depth information of the remaining image blocks; and performing face focusing in the current framing image based on the focusing position. The present disclosure can avoid the problem of face out of focus due to the determination of the face frame containing the background in the related art, can improve the accuracy of face focusing, improve the quality of the photographed portrait, and thus can improve the user's shooting experience.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a face focusing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of terminal technology, more and more users are taking photos using smartphones and other electronic devices. When shooting portraits, it is usually necessary to focus on the face to ensure that the image is clear. Related technologies for face focusing can first perform face detection to obtain a face bounding box, and then combine this with phase detection autofocus (PDAF) technology to automatically focus on the face.

[0003] However, the above-mentioned face detection and phase detection autofocus technology for automatic face focusing sometimes results in faces being out of focus, leading to poor portrait quality and affecting the user's shooting experience. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, the present disclosure provides a face focusing method, apparatus, device, storage medium, and program product to solve the defects in the related technologies.

[0005] According to a first aspect of the present disclosure, a face focusing method is provided, the method comprising:

[0006] In response to determining the face bounding box of a face in the current framed image, the image region within the face bounding box is divided into multiple image blocks;

[0007] Remove the image blocks containing the background from the plurality of image blocks to obtain the remaining image blocks;

[0008] The focus position is determined based on the depth information of the remaining image blocks;

[0009] Based on the focus position, face focusing is performed in the current viewfinder image.

[0010] In some embodiments, removing image blocks containing background elements from the plurality of image blocks to obtain remaining image blocks includes:

[0011] Determine the depth information of each of the plurality of image blocks;

[0012] The remaining image blocks are obtained by removing image blocks whose depth information is greater than a depth threshold from the plurality of image blocks.

[0013] In some embodiments, determining the depth information of each of the plurality of image patches includes:

[0014] The phase difference of each of the plurality of image blocks is determined based on feature matching between the first phase detection image and the second phase detection image. The first phase detection image and the second phase detection image include phase detection images acquired by the left phase detection pixel and the right phase detection pixel in the image sensor, respectively, during the process of the image sensor acquiring the current view image.

[0015] The depth information of each of the multiple image blocks is determined based on their respective phase differences and a pre-determined relationship model between phase differences and depth information.

[0016] In some embodiments, determining the phase difference of each of the plurality of image blocks based on feature matching between the first phase detection image and the second phase detection image includes:

[0017] Based on the face bounding box of the current view image, the first face bounding box of the first phase detection image and the second face bounding box of the second phase detection image are determined respectively;

[0018] According to the division method of the face frame in the current view image, the image areas within the first face frame and the second face frame are respectively divided into multiple image blocks;

[0019] Based on the similarity between each image block in the first face frame and each image block in the second face frame, the phase difference between each image block in the first face frame is determined;

[0020] Based on the phase difference of each image block in the first face frame, the phase difference of image blocks at the same position in the face frame of the face in the current view image is determined.

[0021] In some embodiments, determining the phase difference of each image patch in the first face frame based on the similarity between each image patch in the first face frame and each image patch in the second face frame includes:

[0022] For each first image block in the first face frame, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second face frame;

[0023] The similarity between each of the plurality of third image blocks and the first image block is determined respectively;

[0024] Based on the movement amount of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block, the phase difference of the first image block is determined, and the movement amount is the offset of the target image block relative to the second image block.

[0025] In some embodiments, determining the depth information of each of the plurality of image patches based on their respective phase differences and a pre-determined relationship model between phase differences and depth information includes:

[0026] The target position of each of the multiple image blocks is determined based on the phase difference of each of the multiple image blocks and the current motor position. The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image. The target position is the focus position of each of the multiple image blocks.

[0027] The image distance offset of each of the multiple image blocks is determined based on the target position of each of the multiple image blocks and the first relationship between the different target positions and the corresponding image distance offsets as determined in advance.

[0028] Based on the image distance offset of each of the multiple image blocks and the second relationship between the different image distance offsets and the corresponding object distances, the object distances of the multiple image blocks are determined and used as the depth information of the multiple image blocks.

[0029] In some embodiments, the method further includes determining the first relationship based on the following:

[0030] The object-image relationship of the optical system is transformed to obtain an expression for the image distance offset. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0031] Substituting the first motor position and the second motor position into the expression respectively, we obtain the first image distance offset corresponding to the first motor position and the second image distance offset corresponding to the second motor position. The first motor position is the motor position where the imaging sharpness meets the requirements at a specified object distance, and the second motor position is the motor position where the imaging sharpness meets the requirements at an object distance of equivalent hyperfocal distance.

[0032] The slope of the first relationship is determined based on the ratio of the difference in motor position between the second motor position and the first motor position to the difference in image distance offset between the second image distance offset and the first image distance offset.

[0033] The first relationship is determined based on the slope, the position of the second motor, and the second image distance offset.

[0034] In some embodiments, the method further includes determining the second relationship based on the following:

[0035] The object-image relationship of the optical system is transformed to obtain the second relationship, which includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane, wherein the difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0036] In some embodiments, the depth threshold is determined based on dual-camera images of the face and the principle of binocular 3D reconstruction, wherein the dual-camera images include a first image acquired by a first camera and a second image acquired by a second camera.

[0037] In some embodiments, the depth threshold is determined based on the following:

[0038] Determine the coordinates of the matching point pairs between the first image and the second image;

[0039] Based on the coordinates of the matching point pair and the pre-calibrated projection matrix of the first camera and the second camera, the spatial coordinates of the target point are determined by combining the principle of binocular 3D reconstruction. The target point is the point on the face that corresponds to the matching point.

[0040] The depth threshold is determined based on the spatial coordinates and the distance formula in three-dimensional space.

[0041] In some embodiments, determining the focus position based on the depth information of the remaining image patch includes:

[0042] The average depth information of the remaining image blocks is determined and used as the focal point position.

[0043] According to a second aspect of the present disclosure, a face focusing device is provided, the device comprising:

[0044] The region segmentation module is used to divide the image region within the face frame into multiple image blocks in response to determining the face frame of a face in the current viewfinder image;

[0045] An image acquisition module is used to remove image blocks containing background from the plurality of image blocks to obtain the remaining image blocks;

[0046] The position determination module is used to determine the focus position based on the depth information of the remaining image blocks;

[0047] A face focusing module is used to focus on a face in the current viewfinder image based on the focusing position.

[0048] In some embodiments, the image acquisition module includes:

[0049] A depth information determination unit is used to determine the depth information of each of the plurality of image blocks;

[0050] The remaining image patch acquisition unit is used to remove image patches whose depth information is greater than a depth threshold from the plurality of image patches to obtain the remaining image patches.

[0051] In some embodiments, the depth information determining unit is further configured to:

[0052] The phase difference of each of the plurality of image blocks is determined based on feature matching between the first phase detection image and the second phase detection image. The first phase detection image and the second phase detection image include phase detection images acquired by the left phase detection pixel and the right phase detection pixel in the image sensor, respectively, during the process of the image sensor acquiring the current view image.

[0053] The depth information of each of the multiple image blocks is determined based on their respective phase differences and a pre-determined relationship model between phase differences and depth information.

[0054] In some embodiments, the depth information determining unit is further configured to:

[0055] Based on the face bounding box of the current view image, the first face bounding box of the first phase detection image and the second face bounding box of the second phase detection image are determined respectively;

[0056] According to the division method of the face frame in the current view image, the image areas within the first face frame and the second face frame are respectively divided into multiple image blocks;

[0057] Based on the similarity between each image block in the first face frame and each image block in the second face frame, the phase difference between each image block in the first face frame is determined;

[0058] Based on the phase difference of each image block in the first face frame, the phase difference of image blocks at the same position in the face frame of the face in the current view image is determined.

[0059] In some embodiments, the depth information determining unit is further configured to:

[0060] For each first image block in the first face frame, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second face frame;

[0061] The similarity between each of the plurality of third image blocks and the first image block is determined respectively;

[0062] Based on the movement amount of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block, the phase difference of the first image block is determined, and the movement amount is the offset of the target image block relative to the second image block.

[0063] In some embodiments, the depth information determining unit is further configured to:

[0064] The target position of each of the multiple image blocks is determined based on the phase difference of each of the multiple image blocks and the current motor position. The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image. The target position is the focus position of each of the multiple image blocks.

[0065] The image distance offset of each of the multiple image blocks is determined based on the target position of each of the multiple image blocks and the first relationship between the different target positions and the corresponding image distance offsets as determined in advance.

[0066] Based on the image distance offset of each of the multiple image blocks and the second relationship between the different image distance offsets and the corresponding object distances, the object distances of the multiple image blocks are determined and used as the depth information of the multiple image blocks.

[0067] In some embodiments, the depth information determining unit is further configured to determine the first relationship based on the following:

[0068] The object-image relationship of the optical system is transformed to obtain an expression for the image distance offset. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0069] Substituting the first motor position and the second motor position into the expression respectively, we obtain the first image distance offset corresponding to the first motor position and the second image distance offset corresponding to the second motor position. The first motor position is the motor position where the imaging sharpness meets the requirements at a specified object distance, and the second motor position is the motor position where the imaging sharpness meets the requirements at an object distance of equivalent hyperfocal distance.

[0070] The slope of the first relationship is determined based on the ratio of the difference in motor position between the second motor position and the first motor position to the difference in image distance offset between the second image distance offset and the first image distance offset.

[0071] The first relationship is determined based on the slope, the position of the second motor, and the second image distance offset.

[0072] In some embodiments, the depth information determining unit is further configured to determine the second relationship based on the following:

[0073] The object-image relationship of the optical system is transformed to obtain the second relationship, which includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane, wherein the difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0074] In some embodiments, the depth threshold is determined based on dual-camera images of the face and the principle of binocular 3D reconstruction, wherein the dual-camera images include a first image acquired by a first camera and a second image acquired by a second camera.

[0075] In some embodiments, the image acquisition module further includes a depth threshold determination unit;

[0076] The depth threshold determination unit is used to determine the depth threshold based on the following method:

[0077] Determine the coordinates of the matching point pairs between the first image and the second image;

[0078] Based on the coordinates of the matching point pair and the pre-calibrated projection matrix of the first camera and the second camera, the spatial coordinates of the target point are determined by combining the principle of binocular 3D reconstruction. The target point is the point on the face that corresponds to the matching point.

[0079] The depth threshold is determined based on the spatial coordinates and the distance formula in three-dimensional space.

[0080] In some embodiments, the position determination module is further configured to determine the average value of the depth information of the remaining image patch and use it as the focus position.

[0081] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising:

[0082] Processor and memory used to store computer programs;

[0083] The processor is configured to implement the face focusing method described above when executing the computer program.

[0084] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the face focusing method described in any of the preceding claims.

[0085] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the face focusing method described in any of the preceding claims.

[0086] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0087] This disclosure, in response to determining a face bounding box in the current viewfinder image, divides the image region within the face bounding box into multiple image blocks, removes image blocks containing background from the multiple image blocks to obtain remaining image blocks, and then determines the focus position based on the depth information of the remaining image blocks. Furthermore, it performs face focusing in the current viewfinder image based on the focus position. This avoids the problem of face being out of focus due to background inclusion in the determined face bounding box, which is present in related technologies. It improves the accuracy of face focusing, enhances the quality of portrait photography, and thus improves the user's shooting experience.

[0088] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0089] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0090] Figure 1A This is a schematic diagram illustrating the result of face frame adjustment according to an exemplary embodiment of the present disclosure;

[0091] Figure 1B This is a schematic diagram illustrating the face frame adjustment result according to yet another exemplary embodiment of this disclosure;

[0092] Figure 1C This is a flowchart illustrating a face focusing method according to an exemplary embodiment of the present disclosure;

[0093] Figure 2 This is a flowchart illustrating how to remove image blocks containing background from the plurality of image blocks according to an exemplary embodiment of the present disclosure;

[0094] Figure 3A This is a flowchart illustrating how to determine the depth information of each of the plurality of image blocks according to an exemplary embodiment of the present disclosure;

[0095] Figure 3B This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to an exemplary embodiment of the present disclosure;

[0096] Figure 3C This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to yet another exemplary embodiment of the present disclosure;

[0097] Figure 3D This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to yet another exemplary embodiment of the present disclosure;

[0098] Figure 4This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to determine the phase difference of each of the plurality of image blocks based on feature matching between a first phase detection image and a second phase detection image;

[0099] Figure 5A This is a flowchart illustrating how to determine the phase difference between each image block in the first face frame according to an exemplary embodiment of the present disclosure;

[0100] Figure 5B This is a schematic diagram of a cost-k curve according to yet another exemplary embodiment of the present disclosure;

[0101] Figure 6A This is a flowchart illustrating how to determine the depth information of each of the plurality of image blocks according to an exemplary embodiment of the present disclosure;

[0102] Figure 6B This is a schematic diagram of the focus position in an image according to an exemplary embodiment of the present disclosure;

[0103] Figure 7 This is a flowchart illustrating how the first relationship is determined according to an exemplary embodiment of this disclosure;

[0104] Figure 8A This is a flowchart illustrating how to determine the depth threshold according to an exemplary embodiment of the present disclosure;

[0105] Figure 8B This is a schematic diagram of a binocular 3D reconstruction principle according to an exemplary embodiment of the present disclosure;

[0106] Figure 9 This is a block diagram illustrating a face focusing device according to an exemplary embodiment of the present disclosure;

[0107] Figure 10 This is a block diagram illustrating yet another face focusing device according to an exemplary embodiment of the present disclosure;

[0108] Figure 11 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0109] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0110] With the development of terminal technology, more and more users are taking photos using electronic devices such as smartphones. When shooting portraits, it is usually necessary to focus on the face to ensure that the portrait is sharp.

[0111] When performing face focusing in related technologies, it is usually necessary to first perform face detection to obtain the face bounding box (i.e., the Face Detection box, or "FD box" for short), and then count the phase difference (PD) value in the face bounding box. Then, based on the phase detection autofocus (PDAF) technology and the counted PD value, the focusing position is calculated, and then the position of the motor is moved according to the focusing position to complete the automatic focusing of the face.

[0112] PDAF refers to separating the pixels in the pixel array of an image sensor into left and right sides, so that the pixels in the pixel array include left phase detection pixels and right phase detection pixels, thereby obtaining the PD value between the image information acquired by the left phase detection pixels and the image information acquired by the right phase detection pixels, and then performing portrait focusing based on the PD value.

[0113] In implementing the embodiments of this disclosure, the inventors discovered that in related technologies, in order to ensure sufficient detail for calculating the PD value, the face frame is usually set to be large, which can easily lead to the face frame being mistakenly placed in the background (i.e., the face frame is not accurately placed in the actual position of the face, but is mistakenly extended to the background behind the face). This will result in inaccurate calculated phase difference (PD) values, which in turn will lead to blurred faces, affecting the quality of portrait shooting, and thus affecting the user's shooting experience.

[0114] Specifically, the autofocus (AF) algorithm in related technologies can adjust the face bounding box output by the FD algorithm. For example, Figure 1A This is a schematic diagram illustrating the face frame adjustment result according to an exemplary embodiment of this disclosure; as shown below. Figure 1A As shown, the yellow box K1 is the FD bounding box output by the face detection FD algorithm, and the blue box K2 is the face bounding box obtained after adjusting K1. To ensure that the face bounding box has sufficient detail for calculating the PD value, the AF algorithm typically sets parameters to indent and expand the width and height of the FD bounding box, respectively. For example, the width of the face bounding box can be appropriately indented, while the height of the face bounding box can be expanded.

[0115] However, adjusting the FD box using only fixed parameters (i.e., width and height) can easily lead to the following two situations:

[0116] Case 1: The FD algorithm returns a FD bounding box with a deviation, causing the face bounding box to include the background.

[0117] Case 2: The fixed parameters of the AF algorithm for adjusting the FD bounding box are not applicable to various scenarios (such as side face scenes or backlit face scenes), causing the face bounding box to include the background.

[0118] For example, Figure 1B This is a schematic diagram illustrating the result of face frame adjustment according to yet another exemplary embodiment of this disclosure. For example... Figure 1B As shown, the yellow box K3 is the FD bounding box output by the face detection FD algorithm, and the blue box K4 is the face bounding box obtained after adjusting K3. However, because the upper right corner of the yellow box K3 encompasses the details of the checkerboard pattern in the background, it interferes with the subsequent PD value calculation results. This causes the calculated focus position to be located between the foreground face and the background checkerboard pattern, resulting in the face being out of focus.

[0119] The blue K4 frame is a face frame adjusted based on fixed parameters, but its lower right corner mistakenly frames the rectangular support behind it. This will also interfere with the subsequent PD value calculation results, causing the calculated focus position to be between the foreground face and the background support, which will lead to the face being out of focus.

[0120] In view of this, the present disclosure provides the following face focusing methods, apparatus, devices, storage media and program products to solve the above-mentioned drawbacks in the related technologies.

[0121] Figure 1C This is a flowchart illustrating a face focusing method according to an exemplary embodiment. The method of this embodiment can be executed by a face focusing device, which can be configured in an electronic device, including but not limited to smartphones, tablets, smart cameras, smartwatches, smart bracelets, smart glasses, and other electronic devices with shooting functions. Figure 1C As shown, the method includes the following steps S101-S104:

[0122] In step S101, in response to determining the face bounding box of a face in the current viewfinder image, the image region within the face bounding box is divided into multiple image blocks.

[0123] In this embodiment, the electronic device can divide the image region within the face frame into multiple image blocks in response to determining the face frame of a person in the current viewfinder image.

[0124] For example, when a user uses an electronic device to take a picture or video, the device's display screen can show the currently captured image. For instance, in response to a user opening a camera app on the device's main interface (e.g., clicking), the device's display screen can show the current view, i.e., the current frame of view.

[0125] Furthermore, electronic devices can determine the bounding box of a face in the currently captured image based on face detection (FD) algorithms in related technologies, and divide the image region within the bounding box into multiple image blocks. For example, it can be divided into N*M blocks (i.e., the image region within the bounding box has N rows and M columns of image blocks), where M and N are positive integers, and their values ​​can be set according to the size of the image region within the bounding box. For example, assuming the image region within the bounding box is 4000*3000 pixels, the image can typically be divided into 30*30 image blocks.

[0126] In step S102, image blocks containing background elements are removed from the plurality of image blocks to obtain the remaining image blocks.

[0127] In this embodiment, after the image area within the face frame is divided into multiple image blocks, the image blocks containing the background can be removed from the multiple image blocks to obtain the remaining image blocks.

[0128] It is worth noting that the method for removing image blocks containing background elements from the multiple image blocks described above can be selected from relevant technologies according to the needs of the scenario and business requirements. For example, it can be a method based on deep learning semantic segmentation, a method combining image processing and machine learning, or a method based on template matching and similarity calculation, etc. This embodiment does not limit this method. Wherein:

[0129] Deep learning-based semantic segmentation utilizes deep learning models to perform semantic segmentation on images, classifying each pixel as either foreground or background. For face images, a specialized semantic segmentation model can be trained to accurately identify the face region as foreground and other regions as background.

[0130] The approach combines traditional image processing techniques with machine learning. First, traditional image processing techniques are used to extract image features, such as edges, textures, and colors. Then, these features are used as input to train a binary classifier using machine learning algorithms to determine whether an image patch contains foreground (face) or background.

[0131] The template-matching and similarity calculation method involves pre-defining a facial template, which can be an image of a face or a mathematical model representing facial features. Then, the similarity between each image patch and the template is calculated; image patches with low similarity are considered to contain background and are discarded.

[0132] In other embodiments, the method of removing image blocks containing background from the plurality of image blocks described above can also be found in the following: Figure 2 The embodiments shown will not be described in detail here.

[0133] It is worth noting that in related technologies, the face is out of focus because the background is included in the face frame determined by the FD algorithm. Therefore, in this embodiment, removing the image block containing the background in the face frame is beneficial to improving the accuracy of determining the focus position based on the remaining image block, thereby improving the face focusing accuracy and the quality of the portrait.

[0134] In step S103, the focus position is determined based on the depth information of the remaining image blocks.

[0135] In this embodiment, after removing the image blocks containing the background from the plurality of image blocks to obtain the remaining image blocks, the focus position is determined based on the depth information of the remaining image blocks.

[0136] For example, once the remaining image blocks are obtained, the depth information of each of the remaining image blocks can be determined, and then the average value of the depth information of these remaining image blocks can be determined and used as the focusing position (e.g., the position where the lens is focused).

[0137] It is understandable that the depth information of each of the remaining image patches can refer to the distance information between the image patch and the camera.

[0138] In some embodiments, the depth information may be acquired by means of additional hardware devices (such as depth cameras) or by using software-based depth estimation algorithms, and this embodiment does not limit this.

[0139] In other embodiments, the method for obtaining the depth information described above may also be as follows. Figure 3A The embodiments shown will not be described in detail here.

[0140] In step S104, face focusing is performed on the current viewfinder image based on the focusing position.

[0141] In this embodiment, after determining the focus position based on the depth information of the remaining image blocks, face focusing can be performed on the current viewfinder image based on the focus position.

[0142] For example, once the focus position is determined, the actual distance from the face to the lens can be determined. Based on this distance, the specific position or number of steps that the camera motor needs to move can be calculated. Then, the motor can be controlled to move to the phase position so that the imaging plane of the face coincides with the focal plane of the image sensor, so as to obtain a clear image of the face.

[0143] As described above, the method of this embodiment, in response to determining the face bounding box of a face in the current viewfinder image, divides the image area within the face bounding box into multiple image blocks, removes image blocks containing background from the multiple image blocks to obtain remaining image blocks, then determines the focus position based on the depth information of the remaining image blocks, and then performs face focusing in the current viewfinder image based on the focus position. This can avoid the problem of face out-of-focus due to the presence of background in the determined face bounding box in related technologies, improve the accuracy of face focusing, improve the quality of portrait shooting, and thus enhance the user's shooting experience.

[0144] Figure 2 This is a flowchart illustrating how to remove image blocks containing background from a plurality of image blocks according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to remove image blocks containing background from a plurality of image blocks as an example for illustrative explanation.

[0145] like Figure 2 As shown, the step S102 above, which involves removing image blocks containing background elements from the plurality of image blocks to obtain the remaining image blocks, may include the following steps S201-S202:

[0146] In step S201, the depth information of each of the plurality of image blocks is determined.

[0147] In this embodiment, in order to remove image blocks containing background from the plurality of image blocks, the depth information of each of the plurality of image blocks can be determined first.

[0148] It is worth noting that depth information can directly reflect the distance between the image block and the camera. By determining the depth information of each of the multiple image blocks of the face frame, it is possible to accurately determine which parts of the multiple image blocks are the foreground (i.e., the face) and which are the background.

[0149] In some embodiments, the method for determining the depth information of each of the plurality of image blocks can be selected from relevant technologies according to business needs, and this embodiment does not limit this method.

[0150] In other embodiments, the method for determining the depth information of each of the plurality of image patches can also be found in the following: Figure 3A The embodiments shown will not be described in detail here.

[0151] In step S202, image blocks with depth information greater than a depth threshold are removed from the plurality of image blocks to obtain the remaining image blocks.

[0152] In this embodiment, after determining the depth information of each of the plurality of image blocks, the image blocks whose depth information is greater than the depth threshold can be removed from the plurality of image blocks to obtain the remaining image blocks.

[0153] It's worth noting that faces within the face bounding box are closer to the camera than background elements, thus having smaller depth values. Based on this principle, a suitable depth threshold can be determined to distinguish between foreground and background. Image blocks with depth information greater than the depth threshold are considered to contain background elements, effectively eliminating such blocks.

[0154] In some embodiments, the depth threshold can be determined based on dual-camera images of the face and binocular 3D reconstruction principles (e.g., calculating the 3D coordinates of objects in the image based on binocular vision technology), wherein the dual-camera images include a first image acquired by a first camera and a second image acquired by a second camera. For example, the method for determining the depth threshold can be found below. Figure 8A The embodiments shown will not be described in detail here.

[0155] As described above, this embodiment determines the depth information of each of the plurality of image blocks and removes image blocks whose depth information is greater than a depth threshold from the plurality of image blocks to obtain the remaining image blocks. This can accurately remove image blocks containing background from the plurality of image blocks, thereby enabling subsequent determination of the focus position based on the depth information of the remaining image blocks and face focusing in the current viewfinder image based on the focus position. This avoids the problem of face out-of-focus due to the presence of background in the determined face frame in related technologies, improves the accuracy of face focusing, enhances the quality of portrait photography, and thus improves the user's shooting experience.

[0156] Figure 3A This is a flowchart illustrating how to determine the depth information of each of the plurality of image blocks according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes the determination of the depth information of each of the plurality of image blocks as an example for illustrative explanation.

[0157] like Figure 3A As shown, determining the depth information of each of the plurality of image blocks in step S201 above may include the following steps S301-S302:

[0158] In step S301, the phase difference of each of the plurality of image blocks is determined based on feature matching between the first phase detection image and the second phase detection image.

[0159] In this embodiment, when determining the depth information of each of the plurality of image blocks, the phase difference of each of the plurality of image blocks can be determined first based on feature matching between the first phase detection image and the second phase detection image.

[0160] It is worth noting that the aforementioned first phase detection image and second phase detection image may include phase detection images acquired by the left phase detection pixel and the right phase detection pixel in the image sensor, respectively, during the process of the image sensor acquiring the current view image.

[0161] For example, Figure 3B This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to an exemplary embodiment of the present disclosure. Figure 3C This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to yet another exemplary embodiment of this disclosure. Figure 3D This is a schematic diagram illustrating the distribution of PD pixels on a photosensitive element according to another exemplary embodiment of the present disclosure.

[0162] like Figures 3B to 3D As shown, the photosensitive element of an image sensor has pre-defined left and right PD pixels (i.e., pixels indicated by "L" and "R"). The phase detection image output from all L pixels is the L image (i.e., the first phase detection image), and the phase detection image output from all R pixels is the R image (i.e., the second phase detection image). Based on the density of the PD pixel distribution, photosensitive elements can be classified into two types: dense PD sensors and sparse PD sensors. For example, Figure 3B The image shows the pixel distribution of a dense PD sensor. Figure 3C and Figure 3D The diagram illustrates two different PD pixel distributions in a sparse PD sensor. It's understood that a dense PD sensor uses more PD points to calculate the phase difference, resulting in higher accuracy of the calculated phase value, but with a larger computational load; while a sparse PD sensor uses fewer PD points to calculate the phase difference, resulting in lower accuracy of the calculated phase value, but with a smaller computational load. Therefore, either a dense PD sensor or a sparse PD sensor can be selected based on the application requirements; this embodiment does not impose any limitations on this.

[0163] In some embodiments, the method of determining the phase difference of each of the plurality of image blocks based on feature matching between the first phase detection image and the second phase detection image can be referred to the following: Figure 4 The embodiments shown will not be described in detail here.

[0164] In step S302, the depth information of each of the multiple image blocks is determined based on the phase difference of each of the multiple image blocks and a pre-determined relationship model between the phase difference and the depth information.

[0165] In this embodiment, after determining the phase difference of each of the plurality of image blocks based on feature matching between the first phase detection image and the second phase detection image, the depth information of each of the plurality of image blocks can be determined based on the phase difference of each of the plurality of image blocks and a pre-determined relationship model between the phase difference and the depth information.

[0166] The relationship model between the phase difference and depth information can include a model derived and analyzed based on the measurement principle of the phase difference and information such as the object-image relationship of the optical system. For example, see the following: Figure 6A The embodiments shown will not be described in detail here.

[0167] As described above, this embodiment determines the phase difference of each of the multiple image blocks based on feature matching between the first phase detection image and the second phase detection image, and determines the depth information of each of the multiple image blocks based on the phase difference of each of the multiple image blocks and a pre-determined relationship model between the phase difference and depth information. This can accurately determine the depth information of each of the multiple image blocks, and then enable the subsequent removal of image blocks whose depth information is greater than a depth threshold from the multiple image blocks. Based on the depth information of the remaining image blocks, the focus position is determined, and face focusing is performed in the current viewfinder image based on the focus position. This avoids the problem of face out-of-focus due to the inclusion of background in the determined face frame in related technologies, improves the accuracy of face focusing, improves the quality of portrait shooting, and thus improves the user's shooting experience.

[0168] Figure 4 This is a flowchart illustrating how to determine the phase difference of each of the plurality of image blocks based on feature matching between a first phase detection image and a second phase detection image, according to an exemplary embodiment of the present disclosure. This embodiment is based on the above embodiment and takes the determination of the phase difference of each of the plurality of image blocks based on feature matching between a first phase detection image and a second phase detection image as an example for illustrative explanation.

[0169] like Figure 4 As shown, the step S301 above, which determines the phase difference of each of the plurality of image blocks based on feature matching between the first phase detection image and the second phase detection image, may include the following steps S401-S404:

[0170] In step S401, based on the face bounding box of the current view image, the first face bounding box of the first phase detection image and the second face bounding box of the second phase detection image are determined respectively.

[0171] In this embodiment, when determining the phase difference of each of the plurality of image blocks based on feature matching between the first phase detection image and the second phase detection image, the first face bounding box of the first phase detection image and the second face bounding box of the second phase detection image can be determined based on the face bounding box of the current view image. For example, the positions of the first face bounding box of the first phase detection image and the second face bounding box of the second phase detection image can be determined according to the position of the face bounding box in the current view image.

[0172] In step S402, the image regions within the first face frame and the second face frame are divided into multiple image blocks according to the division method of the face frame in the current view image.

[0173] In this embodiment, after determining the first face frame of the first phase detection image and the second face frame of the second phase detection image, the image regions within the first face frame and the second face frame can be divided into multiple image blocks according to the division method of the face frame of the current view image.

[0174] For example, when the face bounding box of a person in the current viewfinder image is divided into N*M blocks (that is, the image area within the face bounding box has N rows and M columns of image blocks, where M and N are positive integers), the first phase detection image (e.g., "L image") and the second phase detection image (e.g., "R image") can also be divided into N*M blocks in the same way (that is, the L image and the R image each have N rows and M columns of image blocks).

[0175] In step S403, the phase difference of each image block in the first face frame is determined based on the similarity between each image block in the first face frame and each image block in the second face frame.

[0176] In this embodiment, after dividing the image regions within the first face frame and the second face frame into multiple image blocks, the phase difference between each image block in the first face frame can be determined based on the similarity between each image block in the first face frame and each image block in the second face frame.

[0177] In some embodiments, the similarity between two different image patches can be measured based on the SAD (Sum of Absolute Differences) algorithm. For example, the similarity C between two image patches can be set to the negative of the sum of absolute differences.

[0178] In other embodiments, the method for determining the phase difference of each image block in the first face frame can also be found in the following: Figure 5A The embodiments shown will not be described in detail here.

[0179] In step S404, based on the phase difference of each image block in the first face frame, the phase difference of the image blocks at the same position in the face frame of the face in the current view image is determined.

[0180] In this embodiment, after determining the phase difference of each image block in the first face frame, the phase difference of the image blocks at the same position in the face frame of the face in the current view image can be determined based on the phase difference of each image block in the first face frame.

[0181] For example, after determining the phase difference of each image block in the first face frame, the phase difference of each image block in the first face frame can be determined as the phase difference of image blocks at the same position in the face frame of the face in the current viewfinder image. For example, when the phase difference of the image block in the i-th row and j-th column of the first face frame is determined to be d, the phase difference of the image block in the i-th row and j-th column of the face frame of the face in the current viewfinder image can be set to d. Similarly, the phase difference of each image block in the face frame of the face in the current viewfinder image can be determined.

[0182] As described above, this embodiment determines the first face frame of the first phase detection image and the second face frame of the second phase detection image based on the face frame of the current viewfinder image. Following the division method of the face frame in the current viewfinder image, the image regions within the first and second face frames are divided into multiple image blocks. Then, based on the similarity between each image block in the first and second face frames, the phase difference between each image block in the first face frame is determined. Furthermore, based on the phase difference between each image block in the first face frame, the phase difference between image blocks at the same position within the face frame of the face in the current viewfinder image is determined. This allows for accurate determination of the phase difference between multiple image blocks within the face frame of the face in the current viewfinder image. Consequently, it enables accurate determination of the depth information of each of the multiple image blocks and the removal of image blocks containing background elements within the face frame. This avoids the problem of face defocusing caused by background elements within the determined face frame in related technologies, improving the accuracy of face focusing, enhancing the quality of portrait photography, and ultimately improving the user's shooting experience.

[0183] Figure 5AThis is a flowchart illustrating how to determine the phase difference of each image block in the first face frame according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to determine the phase difference of each image block in the first face frame as an example for illustrative explanation.

[0184] like Figure 5A As shown, the step S403 above, which determines the phase difference of each image block in the first face frame based on the similarity between each image block in the first face frame and each image block in the second face frame, may include the following steps S501-S503: In step S501, for each first image block in the first face frame, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second face frame.

[0185] In this embodiment, the first phase detection image is still taken as L image and the second phase detection image is taken as R image, that is, the first face frame is the face frame in L image and the second face frame is the face frame in R image.

[0186] The preset direction can be the row direction of the second face frame (e.g., the horizontal direction of the second face frame). The preset range can be [-h, +h], where the value of h depends on the motor stroke.

[0187] For example, when the first image block of the first face bounding box is represented as I L When (x+i, y+j), the second image block at the same position in the second face frame can be represented as I. R (x+i,y+j), and thus the third image block in the row direction of the second image block within the range [-h,+h] can be represented as I. R (x+i+k,y+j). Where I represents the image, x+i and y+j are the x and y coordinates of the corresponding image patch in the image, respectively, and k is the offset, which is in the range [-h,+h].

[0188] In step S502, the similarity between each of the plurality of third image blocks and the first image block is determined.

[0189] In this embodiment, after acquiring multiple third image blocks, the similarity between each of the multiple third image blocks and the first image block can be determined. Taking the similarity as the inverse of the absolute difference and CAD as an example, the similarity Cost between each of the multiple third image blocks and the first image block can be determined based on the following formula (5-1):

[0190]

[0191] In the above formula, a = w / M, where w is the width of the image region within the face frame and M is the number of columns in the image region within the face frame; b = h / N, where h is the height of the image region within the face frame and N is the number of rows in the image block.

[0192] In other words, the image patch in the face bounding box of image R moves left and right relative to the image patch in the face bounding box of image L in the image row direction, and the corresponding similarity Cost is calculated.

[0193] In step S503, the phase difference of the first image block is determined based on the amount of movement of the target image block among the plurality of third image blocks that has a similarity to the first image block that meets a preset condition.

[0194] In this embodiment, after determining the similarity between each of the plurality of third image blocks and the first image block, the movement amount of the target image block whose similarity with the first image block meets a preset condition (e.g., highest similarity) can be determined, and the phase difference of the first image block can be determined based on this movement amount. The movement amount is the offset of the target image block relative to the second image block.

[0195] For example, after determining the similarity between each of the plurality of third image patches and the first image patch, a Cost-k curve (i.e., a curve showing the cost as a function of k) can be plotted, such as... Figure 5B As shown. Based on this, the image block to which the peak point P1 of the curve (i.e., the point with the highest similarity) belongs can be determined as the target image block that meets the above preset conditions (i.e., the most similar matching block between the face box in the R image and the face box in the L image). Then, the amount of movement of the target image block, that is, the horizontal coordinate k1 of the peak point P1, is used as the phase difference PD value (unit: pixel) corresponding to the first image block in the face box of the L image.

[0196] As described above, this embodiment obtains multiple third image blocks within a preset direction and preset range for each first image block in the first face frame, and determines the similarity between each of the multiple third image blocks and the first image block. Then, based on the movement amount of the target image block whose similarity with the first image block meets the preset condition, the phase difference of the first image block is determined. This can accurately determine the phase difference of each image block in the first face frame, and subsequently, based on the phase difference of each image block in the first face frame, accurately determine the phase difference of the image blocks at the same position in the face frame of the face in the current viewfinder image.

[0197] Figure 6AThis is a flowchart illustrating how to determine the depth information of each of the plurality of image blocks according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes the determination of the depth information of each of the plurality of image blocks as an example for illustrative explanation.

[0198] like Figure 6A As shown, the step S302 above, which determines the depth information of each of the multiple image patches based on their respective phase differences and a pre-determined relationship model between phase differences and depth information, may include the following steps S601-S603:

[0199] In step S601, the target position of each of the multiple image blocks is determined based on the phase difference of each of the multiple image blocks and the current motor position.

[0200] In this embodiment, when determining the depth information of the current image based on the focus information, the target position of each of the multiple image blocks can be determined first based on the phase difference of each of the multiple image blocks and the current motor position.

[0201] The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image, and the target position is the focus position corresponding to each of the plurality of image blocks.

[0202] For example, Figure 6B This is a schematic diagram of the focus position in an image according to an exemplary embodiment of the present disclosure; as shown below. Figure 6B As shown, when the object in the frame is in focus, such as Figure 6B When the image in the frame is at the position corresponding to the gray solid line of the clearest image on the left (the third image from the top on the left), the L image block and the R image block overlap, meaning the phase difference between them is 0. The position of the object at this point is the target position. When the object in the frame is not in focus, the L image block and the R image block will have a left-right deviation; this deviation is the phase difference (PD_value). Furthermore, the farther the object in the frame is from the focus position, the greater the phase difference between the corresponding L image block and the R image block. Therefore, after obtaining the current motor position (i.e., the motor position at the time of obtaining PD raw) and the phase difference of each of the multiple image blocks, the target position of each of the multiple image blocks can be determined. For example, the target position of each image block can be determined based on the following formula (6-1):

[0203] Target pos =Current_pos + PD_value * DCC

[0204] (6-1)

[0205] In the above formula, Targetpos The target position of the current image patch is defined by Current_pos, the current motor position is defined by Current_pos, the phase difference of the current image patch is defined by PD_value, and DCC is the unit conversion (unit: dac / pixel, DCC is a pre-calibrated parameter).

[0206] In step S602, the image distance offset of each of the plurality of image blocks is determined according to the target position of each of the plurality of image blocks and the first relationship between the predetermined different target positions and the corresponding image distance offset.

[0207] In this embodiment, after determining the target positions of the plurality of image blocks, the image distance offset of each of the plurality of image blocks can be determined according to the target positions of the plurality of image blocks and a first relationship between the pre-determined different target positions and the corresponding image distance offsets. For example, the first relationship can be in the form of the following equation (6-2):

[0208] Δtarget=n+λ*(Target pos -m)

[0209] (6-2)

[0210] In the above formula, Δtarget is the image distance offset of the current image block, and m, n and λ are pre-calibrated constants.

[0211] It is worth noting that the calibration methods for m, n, and λ mentioned above (i.e., the methods for determining the first relationship) can be derived from the object-image relationship of the optical system. The derivation process can be found in relevant technical records or in the following text. Figure 7 The embodiments shown will not be described in detail here.

[0212] In step S603, the object distance of each of the multiple image blocks is determined based on the image distance offset of each of the multiple image blocks and the second relationship between the different image distance offsets and the corresponding object distances, and is used as the depth information of each of the multiple image blocks.

[0213] In this embodiment, after determining the image distance offset Δtarget of each of the plurality of image blocks, the object distance of each of the plurality of image blocks can be determined according to the image distance offset of each of the plurality of image blocks and the second relationship between the pre-determined different image distance offsets and the corresponding object distances, and used as the depth information of each of the plurality of image blocks.

[0214] In some embodiments, the second relationship can be obtained by transforming the form of the object-image relationship of the optical system.

[0215] The object-image relationship includes the relationship between the lens focal length f (known), the object distance u, and the distance v from the lens to the image plane. For example, this object-image relationship can be represented by the following equation (6-3):

[0216]

[0217] Based on this, we can transform equation (6-3) to obtain equation (6-4):

[0218]

[0219] In equation (6-4) above, Δ=vf represents the image distance offset.

[0220] Furthermore, the object distance u (i.e., the depth information distance) can be derived from the above equation (6-4). target The expression for ) is shown in equation (6-5) below:

[0221]

[0222] Therefore, once the image distance offset Δtarget corresponding to the current image patch is determined, it can be substituted into equation (6-5) to obtain the depth information distance of the current image patch. target .

[0223] As described above, this embodiment determines the target position of each of the multiple image blocks based on their respective phase differences and the current motor position. It then determines the image distance offset of each of the multiple image blocks based on their respective target positions and a first relationship between predetermined target positions and corresponding image distance offsets. Finally, it determines the object distance of each of the multiple image blocks based on their respective image distance offsets and a second relationship between predetermined image distance offsets and corresponding object distances, using this object distance as the depth information of each of the multiple image blocks. This allows for accurate determination of the depth information of the current image based on the focus information. Consequently, it enables the subsequent removal of image blocks containing background elements within the face frame based on the depth information. This avoids the problem of face defocusing caused by background elements within the determined face frame in related technologies, improving the accuracy of face focusing, enhancing the quality of portrait photography, and ultimately improving the user's shooting experience.

[0224] Figure 7 This is a flowchart illustrating how to determine the first relationship according to an exemplary embodiment of the present disclosure; this embodiment is an exemplary description based on the above embodiment, taking how to determine the first relationship as an example.

[0225] like Figure 7As shown, the face focusing method in this embodiment may further include determining the first relationship based on the following steps S701-S704:

[0226] In step S701, the form of the object-image relationship of the optical system is transformed to obtain an expression for the image distance offset.

[0227] In this embodiment, when the first relationship between different target positions and the corresponding image distance offset is determined, the form of the object-image relationship of the optical system can be transformed to obtain the expression for the image distance offset.

[0228] The aforementioned object-image relationship may include the relationship between the lens focal length f (known), the object distance u, and the distance v from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0229] For example, the object-image relationship can be represented by the following equation (7-1):

[0230]

[0231] Based on this, equation (7-1) can be transformed to obtain the expression for the image distance offset, as shown in equation (7-2) below:

[0232]

[0233] In equation (7-2) above, Δ=vf represents the image distance offset.

[0234] In step S702, the positions of the first motor and the second motor are substituted into the expression respectively to obtain the first image distance offset corresponding to the first motor position and the second image distance offset corresponding to the second motor position.

[0235] In this embodiment, after obtaining the expression for the image distance offset shown in the above formula (7-2), the positions of the first motor and the second motor can be substituted into the expression to obtain the first image distance offset corresponding to the first motor position and the second image distance offset corresponding to the second motor position.

[0236] The positions of the first motor and the second motor are known. The first motor position can be a motor position where the image sharpness at a specified object distance meets the requirements, while the second motor position can be a motor position where the image sharpness at an object distance equal to the equivalent hyperfocal distance meets the requirements.

[0237] For example, the first motor position distance1 can be the motor position position1 (unit dac, which can be obtained through calibration) at a specified object distance (the effective focusing distance of the camera module) of 20cm where the image is clearest. The corresponding image distance offset Δ1 can be shown in the following formula (7-3):

[0238]

[0239] The second motor position, distance2, can be the motor position position2 (unit: dac, which can be obtained through calibration) at the equivalent hyperfocal distance where the image is clearest. The corresponding image distance offset Δ2 can be expressed as shown in equation (7-4):

[0240]

[0241] It is understandable that "equivalent hyperfocal distance" means focusing on a point at a distance so that the other extreme of the depth of field is "infinity". Then, the distance from "infinity (∞)" to the nearest photographic distance within the depth of field is the distance from "infinity (∞)".

[0242] Distance 2 can be calculated from the optical coefficients shown in (7-5):

[0243]

[0244] Among them, F n Let F be the aperture value, and p be the pixel size (known). The F value can be calculated as shown in equation (7-6):

[0245] F-number = Lens focal length (mm) / Aperture diameter (mm)

[0246] (7-6)

[0247] In step S703, the slope of the first relationship is determined based on the ratio of the difference between the motor positions between the second motor position and the first motor position to the difference between the image distance offset between the second image distance offset and the first image distance offset.

[0248] In this embodiment, since there is a linear relationship between Δ1 and Δ2 and position1 and position2, the slope of the first relationship can be determined based on the following equation (7-7):

[0249]

[0250] In step S704, the first relationship is determined based on the slope, the position of the second motor, and the second image distance offset.

[0251] Once the slope is determined, and the target position corresponding to the object is known... pos Then, the corresponding phase offset Δtarget can be calculated, and the determined first relationship is shown in equation (7-8):

[0252] Δtarget = Δ2 + Slope * (Target) pos -position2)

[0253] (7-8)

[0254] As described above, this embodiment transforms the object-image relationship of the optical system to obtain an expression for the image distance offset. Then, the positions of the first and second motors are substituted into the expression to obtain the first image distance offset corresponding to the first motor position and the second image distance offset corresponding to the second motor position. Based on the ratio of the difference between the motor positions of the second and first motor positions to the difference between the second and first image distance offsets, the slope of the first relationship is determined. Then, based on the slope, the second motor position, and the second image distance offset, the first relationship is determined. This allows for the accurate determination of the first relationship between different target positions and their corresponding image distance offsets. Consequently, the image distance offsets of the multiple image blocks can be determined based on their respective target positions and the first relationship.

[0255] Figure 8A This is a flowchart illustrating how to determine the depth threshold according to an exemplary embodiment of the present disclosure; this embodiment is an illustrative example of how to determine the depth threshold based on the above embodiment.

[0256] like Figure 8A As shown, the face focusing method in this embodiment may further include determining the depth threshold based on the following steps S801-S803:

[0257] In step S801, the coordinates of the matching point pair between the first image and the second image are determined.

[0258] In this embodiment, after the dual cameras of the electronic device acquire dual images of the face, namely the first image and the second image, the coordinates of matching point pairs (e.g., a pair of matching points in the first and second images) can be determined. It is understood that the first and second images are acquired by the dual cameras from different viewpoints.

[0259] For example, when the first camera of an electronic device acquires the current view image and detects a face in the current view image, the second camera can be activated, and a first image and a second image of the face can be acquired based on the first and second cameras, respectively. Based on this, the coordinates of the matching point pair between the first and second images can be determined using a preset image registration algorithm.

[0260] In step S802, the spatial coordinates of the target point are determined based on the coordinates of the matching point pair and the pre-calibrated projection matrices of the first and second cameras, combined with the principle of binocular 3D reconstruction.

[0261] In this embodiment, after determining the coordinates of the matching point pair between the first image and the second image, the spatial coordinates of the target point can be determined based on the coordinates of the matching point pair and the pre-calibrated projection matrix of the first camera and the second camera, combined with the principle of binocular 3D reconstruction.

[0262] The target point mentioned above can be a point on the face that corresponds to the matching point.

[0263] For example, Figure 8B This is a schematic diagram of a model illustrating the principle of binocular 3D reconstruction according to an exemplary embodiment of this disclosure. Figure 8B As shown, for a target point P on a person's face, if observed using the first camera, it appears to be located at p1 in the first camera's image. However, the three-dimensional position of point P cannot be determined from p1, meaning that the image coordinates of any point p' on the line connecting O1P (O1 being the optical center of the first camera) are the same as those of p1. However, from... Figure 8B As can be seen, point P is located at the intersection of the two lines O1p1 and O2p2. Therefore, the three-dimensional position of point P is uniquely determined, and the three-dimensional coordinates of point P can be obtained by using stereo vision.

[0264] For example, if the target point P on the face is p1(u1,v1) on the first image acquired by the first camera and p2(u2,v2) on the second image acquired by the second camera, and the calibrated projection matrices of the first camera and the second camera are M1 and M2 respectively, then:

[0265]

[0266] Where (u1,v1,1) and (u2,v2,1) are the homogeneous coordinates of p1(u1,v1) and p2(u2,v2) in their respective images, respectively; (X w ,Y w Z w ,1) are the homogeneous coordinates of point P in the world coordinate system; For matrix M kThe element in the i-th row and j-th column. Equations (8-1) and (8-2) respectively eliminate Z c1 and Z c2 Get information about X w ,Y w Z w The four linear equations (8-3) to (8-6):

[0267]

[0268] Since point P is the intersection of O1p1 and O2p2 (the optical center of the second camera O2), it must simultaneously satisfy the above four linear equations. Therefore, these four equations can be solved simultaneously, and the three-dimensional coordinates (X, Y, F, Z) of point P can be calculated based on a preset algorithm (such as the least squares method). w ,Y w Z w For specific solutions, please refer to relevant technologies, which will not be described in detail in this embodiment.

[0269] In step S803, the depth threshold is determined based on the spatial coordinates and the distance formula in three-dimensional space.

[0270] In this embodiment, when the spatial coordinates (X, X) of the target point P are determined... w ,Y w Z w After that, the depth threshold can be determined based on the spatial coordinates and the distance formula in three-dimensional space.

[0271] For example, the depth threshold Distance_target can be determined based on the following equation (8-2):

[0272]

[0273] As described above, this embodiment determines the coordinates of matching point pairs between the first and second images, and based on the coordinates of the matching point pairs and the pre-calibrated projection matrices of the first and second cameras, combines the principle of binocular 3D reconstruction to determine the spatial coordinates of the target point, where the target point is the point on the face corresponding to the matching point. Then, based on the spatial coordinates and the distance formula in 3D space, the depth threshold is determined. This allows for accurate determination of the depth threshold, enabling the subsequent removal of image blocks with depth information greater than the depth threshold from the multiple image blocks. This improves the accuracy of determining the focus position, thereby enhancing the accuracy of face focusing, improving the quality of portrait photography, and ultimately enhancing the user's shooting experience.

[0274] Figure 9This is a block diagram illustrating a face focusing device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be configured in an electronic device, including but not limited to smartphones, tablets, smart cameras, smartwatches, smart bracelets, smart glasses, and other electronic devices with shooting functions. Figure 9 As shown, the device may include: a region segmentation module 110, an image acquisition module 120, a position determination module 130, and a face focusing module 140, wherein:

[0275] The region division module 110 is used to divide the image region within the face frame into multiple image blocks in response to determining the face frame of a face in the current view image;

[0276] Image acquisition module 120 is used to remove image blocks containing background from the plurality of image blocks to obtain the remaining image blocks;

[0277] The position determination module 130 is used to determine the focus position based on the depth information of the remaining image block;

[0278] The face focusing module 140 is used to focus on a face in the current viewfinder image based on the focusing position.

[0279] As described above, the device in this embodiment, in response to determining the face frame of a face in the current viewfinder image, divides the image area within the face frame into multiple image blocks, removes image blocks containing background from the multiple image blocks to obtain remaining image blocks, then determines the focus position based on the depth information of the remaining image blocks, and then performs face focusing in the current viewfinder image based on the focus position. This avoids the problem of face out-of-focus due to the presence of background in the determined face frame in related technologies, improves the accuracy of face focusing, enhances the quality of portrait photography, and thus improves the user's shooting experience.

[0280] Figure 10 This is a block diagram illustrating another face focusing device according to an exemplary embodiment of the present disclosure. The device of this embodiment can be configured in an electronic device, including but not limited to smartphones, tablets, smart cameras, smartwatches, smart bracelets, smart glasses, and other electronic devices with shooting functions. The region division module 210, image acquisition module 220, position determination module 230, and face focusing module 240 are as described above. Figure 9 The functions of the region division module 110, image acquisition module 120, position determination module 130, and face focusing module 140 in the illustrated embodiment are the same, and will not be described in detail here.

[0281] In some embodiments, the image acquisition module 220 may include:

[0282] The depth information determination unit 221 is used to determine the depth information of each of the plurality of image blocks;

[0283] The remaining image patch acquisition unit 222 is used to remove image patches whose depth information is greater than a depth threshold from the plurality of image patches to obtain the remaining image patches.

[0284] In some embodiments, the depth information determination unit 221 can also be used for:

[0285] The phase difference of each of the plurality of image blocks is determined based on feature matching between the first phase detection image and the second phase detection image. The first phase detection image and the second phase detection image include phase detection images acquired by the left phase detection pixel and the right phase detection pixel in the image sensor, respectively, during the process of the image sensor acquiring the current view image.

[0286] The depth information of each of the multiple image blocks is determined based on their respective phase differences and a pre-determined relationship model between phase differences and depth information.

[0287] In some embodiments, the depth information determination unit 221 can also be used for:

[0288] Based on the face bounding box of the current view image, the first face bounding box of the first phase detection image and the second face bounding box of the second phase detection image are determined respectively;

[0289] According to the division method of the face frame in the current view image, the image areas within the first face frame and the second face frame are respectively divided into multiple image blocks;

[0290] Based on the similarity between each image block in the first face frame and each image block in the second face frame, the phase difference between each image block in the first face frame is determined;

[0291] Based on the phase difference of each image block in the first face frame, the phase difference of image blocks at the same position in the face frame of the face in the current view image is determined.

[0292] In some embodiments, the depth information determination unit 221 can also be used for:

[0293] For each first image block in the first face frame, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second face frame;

[0294] The similarity between each of the plurality of third image blocks and the first image block is determined respectively;

[0295] Based on the movement amount of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block, the phase difference of the first image block is determined, and the movement amount is the offset of the target image block relative to the second image block.

[0296] In some embodiments, the depth information determination unit 221 can also be used for:

[0297] The target position of each of the multiple image blocks is determined based on the phase difference of each of the multiple image blocks and the current motor position. The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image. The target position is the focus position of each of the multiple image blocks.

[0298] The image distance offset of each of the multiple image blocks is determined based on the target position of each of the multiple image blocks and the first relationship between the different target positions and the corresponding image distance offsets as determined in advance.

[0299] Based on the image distance offset of each of the multiple image blocks and the second relationship between the different image distance offsets and the corresponding object distances, the object distances of the multiple image blocks are determined and used as the depth information of the multiple image blocks.

[0300] In some embodiments, the depth information determination unit 221 can also be used to determine the first relationship based on the following:

[0301] The object-image relationship of the optical system is transformed to obtain an expression for the image distance offset. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0302] Substituting the first motor position and the second motor position into the expression respectively, we obtain the first image distance offset corresponding to the first motor position and the second image distance offset corresponding to the second motor position. The first motor position is the motor position where the imaging sharpness meets the requirements at a specified object distance, and the second motor position is the motor position where the imaging sharpness meets the requirements at an object distance of equivalent hyperfocal distance.

[0303] The slope of the first relationship is determined based on the ratio of the difference in motor position between the second motor position and the first motor position to the difference in image distance offset between the second image distance offset and the first image distance offset.

[0304] The first relationship is determined based on the slope, the position of the second motor, and the second image distance offset.

[0305] In some embodiments, the depth information determination unit 221 can also be used to determine the second relationship based on the following:

[0306] The object-image relationship of the optical system is transformed to obtain the second relationship, which includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane, wherein the difference between the distance from the lens to the image plane and the focal length is the image distance offset.

[0307] In some embodiments, the depth threshold is determined based on dual-camera images of the face and the principle of binocular 3D reconstruction, wherein the dual-camera images include a first image acquired by a first camera and a second image acquired by a second camera.

[0308] In some embodiments, the image acquisition module 220 may further include a depth threshold determination unit 223;

[0309] The depth threshold determination unit 223 is used to determine the depth threshold based on the following method:

[0310] Determine the coordinates of the matching point pairs between the first image and the second image;

[0311] Based on the coordinates of the matching point pair and the pre-calibrated projection matrix of the first camera and the second camera, the spatial coordinates of the target point are determined by combining the principle of binocular 3D reconstruction. The target point is the point on the face that corresponds to the matching point.

[0312] The depth threshold is determined based on the spatial coordinates and the distance formula in three-dimensional space.

[0313] In some embodiments, the position determination module 230 may also be used to determine the average value of the depth information of the remaining image blocks and use it as the focus position.

[0314] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0315] In an exemplary embodiment, this disclosure also provides an electronic device including a processor and a memory for storing a computer program; wherein the processor is configured to implement the above-described face focusing method when executing the computer program. For example, Figure 11 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, device 1100 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0316] Reference Figure 11The device 1100 may include one or more of the following components: a processing component 1102, a memory 1104, a power supply component 1106, a multimedia component 1108, an audio component 1110, an input / output (I / O) interface 1112, a sensor component 1114, and a communication component 1116.

[0317] Processing component 1102 typically controls the overall operation of device 1100, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 1102 may include one or more processors 1120 to execute instructions to complete all or part of the steps of the face focusing method described above. Furthermore, processing component 1102 may include one or more modules to facilitate interaction between processing component 1102 and other components. For example, processing component 1102 may include a multimedia module to facilitate interaction between multimedia component 1108 and processing component 1102.

[0318] Memory 1104 is configured to store various types of data to support the operation of device 1100. Examples of this data include instructions for any application or method operating on device 1100, contact data, phonebook data, messages, pictures, videos, etc., where the application implements the aforementioned face focusing method when executed by the processor. Memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0319] Power supply component 1106 provides power to various components of device 1100. Power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1100.

[0320] Multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1108 includes a front-facing camera and / or a rear-facing camera. When the device 1100 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. For example, after the device 1100 captures real-time image data of the current scene based on the front-facing camera and / or the rear-facing camera, it may use this image data as the current viewfinder image and implement the face focusing method described above. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0321] Audio component 1110 is configured to output and / or input audio signals. For example, audio component 1110 includes a microphone (MIC) configured to receive external audio signals when device 1100 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1104 or transmitted via communication component 1116. In some embodiments, audio component 1110 also includes a speaker for outputting audio signals.

[0322] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0323] Sensor assembly 1114 includes one or more sensors for providing status assessments of various aspects of device 1100. For example, sensor assembly 1114 can detect the on / off state of device 1100, the relative positioning of components such as the display panel and keypad of device 1100, changes in the position of device 1100 or a component of device 1100, the presence or absence of user contact with device 1100, the orientation or acceleration / deceleration of device 1100, and temperature changes of device 1100. Sensor assembly 1114 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1114 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1114 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0324] Communication component 1116 is configured to facilitate wired or wireless communication between device 1100 and other devices. Device 1100 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 1116 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1116 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0325] In an exemplary embodiment, device 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the face focusing method described above.

[0326] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, which can be executed by the processor 1120 of the device 1100 to complete the face focusing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0327] In an exemplary embodiment, a computer program product including instructions is also provided, which can be executed by the processor 1120 of the device 1100 to perform the face focusing method described above.

[0328] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0329] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A face focusing method, characterized in that, The method includes: In response to determining the face bounding box of a face in the current framed image, the image region within the face bounding box is divided into multiple image blocks; Remove the image blocks containing the background from the plurality of image blocks to obtain the remaining image blocks; The focus position is determined based on the depth information of the remaining image blocks; Based on the focus position, face focusing is performed in the current viewfinder image.

2. The method according to claim 1, characterized in that, The step of removing image blocks containing background elements from the plurality of image blocks to obtain the remaining image blocks includes: Determine the depth information of each of the plurality of image blocks; The remaining image blocks are obtained by removing image blocks whose depth information is greater than a depth threshold from the plurality of image blocks.

3. The method according to claim 2, characterized in that, Determining the depth information of each of the plurality of image patches includes: The phase difference of each of the plurality of image blocks is determined based on feature matching between the first phase detection image and the second phase detection image. The first phase detection image and the second phase detection image include phase detection images acquired by the left phase detection pixel and the right phase detection pixel in the image sensor, respectively, during the process of the image sensor acquiring the current view image. The depth information of each of the multiple image blocks is determined based on their respective phase differences and a pre-determined relationship model between phase differences and depth information.

4. The method according to claim 3, characterized in that, The step of determining the phase difference of each of the plurality of image blocks based on feature matching between the first phase detection image and the second phase detection image includes: Based on the face bounding box of the current view image, the first face bounding box of the first phase detection image and the second face bounding box of the second phase detection image are determined respectively; According to the division method of the face frame in the current view image, the image areas within the first face frame and the second face frame are respectively divided into multiple image blocks; Based on the similarity between each image block in the first face frame and each image block in the second face frame, the phase difference between each image block in the first face frame is determined; Based on the phase difference of each image block in the first face frame, the phase difference of image blocks at the same position in the face frame of the face in the current view image is determined.

5. The method according to claim 4, characterized in that, The step of determining the phase difference of each image patch in the first face frame based on the similarity between each image patch in the first face frame and each image patch in the second face frame includes: For each first image block in the first face frame, multiple third image blocks are obtained in a preset direction and within a preset range of the second image block at the same position in the second face frame; The similarity between each of the plurality of third image blocks and the first image block is determined respectively; Based on the movement amount of the target image block among the plurality of third image blocks that meets the preset condition of similarity with the first image block, the phase difference of the first image block is determined, and the movement amount is the offset of the target image block relative to the second image block.

6. The method according to claim 3, characterized in that, The determination of the depth information of each of the multiple image patches based on their respective phase differences and a pre-determined relationship model between phase differences and depth information includes: The target position of each of the multiple image blocks is determined based on the phase difference of each of the multiple image blocks and the current motor position. The current motor position includes the motor position at the time of acquiring the first phase detection image and the second phase detection image. The target position is the focus position of each of the multiple image blocks. The image distance offset of each of the multiple image blocks is determined based on the target position of each of the multiple image blocks and the first relationship between the different target positions and the corresponding image distance offsets as determined in advance. Based on the image distance offset of each of the multiple image blocks and the second relationship between the different image distance offsets and the corresponding object distances, the object distances of the multiple image blocks are determined and used as the depth information of the multiple image blocks.

7. The method according to claim 6, characterized in that, The method further includes determining the first relationship based on the following: The object-image relationship of the optical system is transformed to obtain an expression for the image distance offset. The object-image relationship includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane. The difference between the distance from the lens to the image plane and the focal length is the image distance offset. Substituting the first motor position and the second motor position into the expression respectively, we obtain the first image distance offset corresponding to the first motor position and the second image distance offset corresponding to the second motor position. The first motor position is the motor position where the imaging sharpness meets the requirements at a specified object distance, and the second motor position is the motor position where the imaging sharpness meets the requirements at an object distance of equivalent hyperfocal distance. The slope of the first relationship is determined based on the ratio of the difference in motor position between the second motor position and the first motor position to the difference in image distance offset between the second image distance offset and the first image distance offset. The first relationship is determined based on the slope, the position of the second motor, and the second image distance offset.

8. The method according to claim 6, characterized in that, The method further includes determining the second relationship based on the following: The object-image relationship of the optical system is transformed to obtain the second relationship, which includes the relationship between the focal length of the lens, the object distance, and the distance from the lens to the image plane, wherein the difference between the distance from the lens to the image plane and the focal length is the image distance offset.

9. The method according to claim 2, characterized in that, The depth threshold is determined based on the dual-camera images of the face and the principle of binocular 3D reconstruction. The dual-camera images include a first image acquired by the first camera and a second image acquired by the second camera.

10. The method according to claim 9, characterized in that, The depth threshold is determined based on the following method: Determine the coordinates of the matching point pairs between the first image and the second image; Based on the coordinates of the matching point pair and the pre-calibrated projection matrix of the first camera and the second camera, the spatial coordinates of the target point are determined by combining the principle of binocular 3D reconstruction. The target point is the point on the face that corresponds to the matching point. The depth threshold is determined based on the spatial coordinates and the distance formula in three-dimensional space.

11. The method according to claim 1, characterized in that, Determining the focus position based on the depth information of the remaining image patch includes: The average depth information of the remaining image blocks is determined and used as the focal point position.

12. A face focusing device, characterized in that, The device includes: The region segmentation module is used to divide the image region within the face frame into multiple image blocks in response to determining the face frame of a face in the current viewfinder image; An image acquisition module is used to remove image blocks containing background from the plurality of image blocks to obtain the remaining image blocks; The position determination module is used to determine the focus position based on the depth information of the remaining image blocks; A face focusing module is used to focus on a face in the current viewfinder image based on the focusing position.

13. An electronic device, characterized in that, The device includes: Processor and memory used to store computer programs; The processor is configured to implement the face focusing method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the face focusing method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the face focusing method according to any one of claims 1 to 11.

Images