Depth information determination method and electronic equipment

Abstract

The invention discloses a depth information determination method and electronic equipment, and the method comprises the steps: determining the actual offset of different image regions in an image collected by an image sensor, the actual offset being used for representing the lens movement amount required for achieving the focusing of a collected object in the image region by the image sensor; determining the actual object distance of the collected object in the image area based on the conversion relation between the configured offset and the object distance and the actual offset of the image area; and determining depth information of the image based on the actual object distance of the collected object in the image area.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for determining depth information and an electronic device. Background Technology

[0002] After electronic devices capture images, it is often necessary to blur the background. A prerequisite for background blurring is determining the depth information of the image. Currently, two cameras are required to determine this depth information, but using two cameras results in significant power consumption. Summary of the Invention

[0003] On the one hand, this application provides a method for determining depth information, including:

[0004] The actual offset of different image regions in the image acquired by the image sensor is determined. The actual offset is used to characterize the amount of lens movement required to achieve focus on the acquired object in the image region.

[0005] Based on the conversion relationship between the configured offset and the object distance, and the actual offset of the image region, the actual object distance of the captured object in the image region is determined.

[0006] The depth information of the image is determined based on the actual object distance of the captured object in the image region.

[0007] In one possible implementation, determining the actual object distance of the captured object in the image region based on the configured conversion relationship between offset and object distance and the actual offset of the image region includes:

[0008] Based on the conversion relationship between the configured offset and the focal distance, the actual focal distance corresponding to the actual offset of the image region is determined. The actual focal distance is the distance between the captured object and the focal plane of the captured object in the image region.

[0009] The actual focal distance is determined as the actual object distance of the object being captured in the image region.

[0010] In another possible implementation, determining the actual object distance of the captured object in the image region based on the conversion relationship between the configured offset and the object distance, and the actual offset of the image region, includes:

[0011] Based on the conversion relationship between the configured offset and the objective lens distance, the actual objective lens distance corresponding to the actual offset of the image region is determined. The actual objective lens distance is the distance between the object being acquired in the image region and the lens of the image acquisition module.

[0012] The actual objective lens distance is determined as the actual object distance of the object being captured in the image region.

[0013] In another possible implementation, determining the depth information of the image based on the actual object distance of the captured object in the image region includes:

[0014] Determine the minimum actual object distance corresponding to each image region in the image;

[0015] Based on the minimum value, determine the depth compensation value;

[0016] Based on the depth compensation value, the actual object distance of the acquired object in the image region is converted into the depth value of the image region to obtain the depth information of the image, which includes the depth value of each image region in the image.

[0017] In yet another possible implementation, determining the depth compensation value based on the minimum value includes: determining the difference between zero and the minimum value as the depth compensation value;

[0018] The step of converting the actual object distance of the captured object in the image region into the depth value of the image region based on the depth compensation value includes:

[0019] The depth value corresponding to the image region is obtained by adding the actual object distance of the captured object in the image region to the depth compensation value.

[0020] Another possible implementation includes:

[0021] Based on the distribution of the phase focusing units in the image sensor, the image acquired by the image sensor is divided into at least one image region.

[0022] In another possible implementation, dividing the image acquired by the image sensor into at least one image region based on the distribution state of the phase focusing units in the image sensor includes:

[0023] In response to the fact that each physical pixel in the image sensor has a phase focusing unit, the image acquired by the image sensor is divided into at least one image region based on the arrangement of the physical pixels in the image sensor, and each image region includes at least one pixel.

[0024] In response to the fact that multiple physical pixels in the image sensor correspond to one phase focusing unit, the image acquired by the image sensor is divided into at least one image region according to the correspondence between each physical pixel in the image sensor and the phase focusing unit. Each image region includes a target number of pixels, where the target number is the number of physical pixels corresponding to a single phase focusing unit.

[0025] Another possible implementation includes:

[0026] Identify the target object in the image;

[0027] Based on the depth information of the image, background blurring is applied to areas outside the region where the target object is located in the image.

[0028] In another possible implementation, the conversion relationship between the offset and the focal distance is: a conversion function determined based on multiple measured offsets when there are multiple different focal distances between the reference object and the focal plane corresponding to the reference object;

[0029] Alternatively, the conversion relationship between the offset and the objective lens distance is a conversion function determined based on multiple measured offsets when there are multiple different objective lens distances between the reference object and the lens of the image acquisition module.

[0030] The offset is used to characterize the amount of lens movement required to achieve focusing of the image sensor on the reference object.

[0031] In another aspect, this application also provides an electronic device, including: an image acquisition module and a processor;

[0032] The image acquisition module includes an image sensor and a lens, wherein the image sensor is provided with at least one phase focusing unit;

[0033] The image sensor is used to determine the actual offset of different image regions in the image acquired by the image sensor based on the phase focusing unit. The actual offset is used to characterize the amount of lens movement required to achieve focusing on the acquired object in the image region.

[0034] The processor is configured to determine the actual object distance of the object being captured in the image region based on the configured conversion relationship between offset and object distance and the actual offset of the image region; and to determine the depth information of the image based on the actual object distance of the object being captured in the image region. Attached Figure Description

[0035] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0036] Figure 1 A flowchart illustrating the depth information determination method provided in this application;

[0037] Figure 2 This is an example diagram illustrating the division of an image into multiple image regions in this application;

[0038] Figure 3 Another flowchart illustrating the depth information determination method provided in this application;

[0039] Figure 4 This is a schematic diagram illustrating one implementation process for determining the conversion relationship between offset and focal distance in this application;

[0040] Figure 5 This is an example diagram showing the distances of multiple objective lenses and focal distances corresponding to the reference object during the testing process of this application.

[0041] Figure 6 Another flowchart illustrating the depth information determination method provided in this application;

[0042] Figure 7 An example diagram illustrates the correspondence between the actual offset of an image region and the corresponding depth value of the image region;

[0043] Figure 8 It shows Figure 2 An example image showing the depth values ​​corresponding to a portion of the image region.

[0044] Figure 9 Another flowchart illustrating the depth information determination method provided in this application;

[0045] Figure 10 A schematic diagram of the composition structure of an electronic device provided in this application. Detailed Implementation

[0046] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0047] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0048] like Figure 1 This illustration shows a flowchart of a depth information determination method provided in this application. The method of this embodiment can be applied to an electronic device, which includes an image acquisition module, and the image acquisition module includes an image sensor and a lens. For example, the electronic device can be a mobile phone, tablet computer, or laptop computer, etc., without limitation.

[0049] The method in this embodiment may include:

[0050] S101, determine the actual offset of different image regions in the image acquired by the image sensor.

[0051] In this application, each image region includes at least one pixel. The size of the image region (i.e., the number of pixels) can be determined according to actual needs, as long as it ensures that the image sensor can determine the actual offset of the image region.

[0052] The offset of the image region characterizes the amount of lens movement required to focus on the object captured in that image region. Focusing refers to the process by which the image acquisition module (to which the image sensor belongs) adjusts the forward and backward movement of the lens to change its optical characteristics (such as the position of the focal plane), so that the light reflected from the object captured by the image acquisition module, after being refracted by the lens, can accurately converge on the plane where the image sensor is located, forming a clear image of the object. The offset of the image region reflects the distance between the imaging plane of the image sensor and the focal plane corresponding to the object in that image region. The focal plane corresponding to the object is the plane on which a clear image of the object can be formed after being captured and focused by the lens. In this application, to distinguish it from the offset of a pre-tested image or a local image region, the actual offset of this image region is referred to as the actual offset of the image region.

[0053] An image sensor has at least one phase-detection autofocus element capable of determining the actual offset of each image region. For example, this phase-detection autofocus element can be a phase detection autofocus (PDAF) element. Accordingly, the actual offset of the image region can be determined by the PDAF element in the image sensor, and the offset determined by the PDAF element can also be called the defocus amount.

[0054] It is understandable that the object being captured in an image region refers to the object (which can be a point or object in the scene being captured) that is imaged onto each pixel in that image region. Since the number of pixels in an image region is relatively limited, the object being captured can be at least one complete physical object (such as an object, building, plant, icon, person, animal, or dust particle), or it can be a portion of at least one physical object.

[0055] S102, for each image region, based on the configured conversion relationship between offset and object distance and the actual offset of the image region, determine the actual object distance of the captured object in the image region.

[0056] The conversion relationship between offset and object distance is obtained through pre-testing. For example, when a reference object corresponds to multiple different object distances, the conversion relationship is determined based on the multiple offsets required to focus on the reference object using the image sensor. This conversion relationship can be a mapping relationship between multiple sets of offsets and object distances. In particular, to more accurately convert the offsets corresponding to different object distances, the conversion relationship between offset and object distance can be a pre-measured and determined conversion function.

[0057] Correspondingly, this conversion relationship can be used to convert the actual object distance corresponding to the actual offset of the image region, thus obtaining the actual object distance of the captured object in the image region.

[0058] In this application, the object distance is the distance between the object (such as the object being acquired) and the reference plane associated with the image acquisition module. This reference plane can have several possible values.

[0059] For example, the reference plane can be the focal plane of the object when the image acquisition module acquires an image of the object; or it can be the lens of the image acquisition module (i.e., the plane where the lens is located). Correspondingly, the actual object distance of the acquired object in the image area can be the distance from the acquired object to the focal plane corresponding to the acquired object in the image acquisition area, or the distance from the acquired object to the lens of the image acquisition module.

[0060] 103. Determine the depth information of the image based on the actual object distance of the captured object in each image region.

[0061] The depth information of an image reflects the actual distance between the object and the lens of the image acquisition module at each pixel or location region (such as each image region or each pixel of an image region). Accordingly, the depth information can include depth values ​​corresponding to different locations within the image. For example, the depth information can include the depth values ​​of each image region or the depth values ​​corresponding to each pixel in the image.

[0062] Specifically, the closer an object in a certain region of an image is to the lens, the smaller the depth value corresponding to that region; conversely, the farther an object in a certain region of an image is from the lens, the larger the depth value corresponding to that region.

[0063] Understandably, the depth information of an image can be used to determine the depth information of areas other than the target object's location during image blurring; it can also be used for 3D image reconstruction, and so on, without limitation.

[0064] As can be seen from the above, this application predetermines and configures the conversion relationship between offset and object distance. After determining the actual offset of different image regions in the image acquired by the image sensor, the actual object distance of the acquired object in the image region can be determined based on the conversion relationship and the actual offset of the image region. On this basis, the depth information of the image can be determined based on the actual object distance of the acquired object in different image regions. This enables the determination of image depth information without the need for two image acquisition modules to work together, which naturally reduces the power consumption generated by starting two image acquisition modules, thereby reducing the power consumption required to determine the depth information of the image.

[0065] Furthermore, since each image acquisition and depth information determination only requires activating the image acquisition module, there is no need to separately activate another image acquisition module after the image is acquired. Compared to determining depth information through data processing by a processor, activating another image acquisition module takes a relatively long time. Therefore, this application can determine the depth information of an image using only one image acquisition module, thus reducing the time required to determine the depth information.

[0066] In the embodiments of this application, there are multiple possible implementations for determining different image regions in the image sensor.

[0067] In one possible implementation, before determining the actual offset of each image region in the image, in order to more reasonably divide the image regions in the image acquired by the image sensor so as to determine the actual offset corresponding to each image region, this application may also divide the image acquired by the image sensor into at least one image region based on the distribution state of the phase focusing unit in the image sensor.

[0068] The distribution of phase-detection autofocus units refers to the number and arrangement of these units within the image sensor. Each phase-detection autofocus unit determines the amount of lens movement required for the image sensor to focus on an object within its detection range. Because the distribution of phase-detection autofocus units within the image acquisition unit varies, the number and distribution area of ​​physical pixels in the image sensor corresponding to each unit differs. Therefore, for the image acquired by the image sensor, the size of the image region from which each phase-detection autofocus unit can determine the actual offset will also vary. Based on this, by combining the distribution of phase-detection autofocus units within the image sensor, it is possible to reasonably determine the appropriate image regions to be divided from the acquired image, ensuring that each image region corresponds to at least one phase-detection autofocus unit. This reduces the likelihood of situations where the actual offset corresponding to an image region cannot be determined.

[0069] In practical applications, an image sensor can have multiple physical pixels corresponding to one phase-detection autofocus unit (PDAF unit); alternatively, each physical pixel can have its own PDAF unit. For example, if the image sensor is a full PDAF sensor (also known as an all PDAF sensor), then each physical pixel corresponds to one PDAF unit. Therefore, the image sensor can determine the actual offset of each pixel in the acquired image.

[0070] In order to reasonably divide the image region, different methods can be used to determine the image region for two different situations of the phase focusing unit in the image sensor.

[0071] Specifically, in response to the fact that each physical pixel in the image sensor has a phase focusing unit, the image acquired by the image sensor is divided into at least one image region based on the arrangement of the physical pixels in the image sensor. Each image region includes at least one pixel.

[0072] In this case, each pixel can be considered as an image region, and the actual offset of the image region is the offset output by the phase-detection autofocus unit corresponding to that region. Alternatively, an image region can be defined as a region consisting of a number of neighboring target pixels. For example... Figure 2This shows an example diagram of dividing an image into multiple image regions. Figure 2 In the image, the area segmented by the dashed lines represents the image region. It can be seen that... Figure 2 The image is divided into 12. Taking 13 image regions as an example, each image region can include multiple adjacent pixels. Therefore, each image region can correspond to multiple phase-detection autofocus units. Based on this, the actual offset corresponding to the image region can be the average of the offsets determined by multiple phase-detection autofocus units.

[0073] Accordingly, in response to the correspondence between multiple physical pixels in the image sensor and one phase focusing unit, the image acquired by the image sensor is divided into at least one image region according to the correspondence between each physical pixel in the image sensor and the phase focusing unit. In this case, each image region includes a target number of pixels, where the target number is the number of physical pixels corresponding to a single phase focusing unit. In this case, the image region whose actual offset can be determined by each phase focusing unit is fixed, and each image region corresponding to each phase focusing unit can be directly used as the image region divided from the image, so that each phase focusing unit can determine the actual offset of an image region.

[0074] In this application, there are multiple possible implementations for determining the depth information of an image based on the actual object distance corresponding to the image region, and no limitation is imposed on this. To determine the image depth more efficiently and accurately, the following describes... Figure 3 Let's take one implementation method as an example to illustrate.

[0075] like Figure 3 This illustration shows another flowchart of the depth information determination method provided in this application. The method of this embodiment can be applied to the aforementioned electronic devices. The method of this embodiment may include:

[0076] S301, determine the actual offset of different image regions in the image acquired by the image sensor.

[0077] The actual offset is used to characterize the amount of lens movement required to focus on the captured object in the image area.

[0078] S302, for each image region of the image, based on the configured conversion relationship between offset and object distance and the actual offset of the image region, determine the actual object distance of the acquired object in the image region.

[0079] The above steps can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0080] S303, determine the minimum distance between the actual objects corresponding to each image region in the image.

[0081] For example, if an image is divided into 300 image regions, then the minimum value needs to be determined from the 300 actual object distances corresponding to these 300 image regions. This minimum value is also the minimum actual object distance.

[0082] S304, determine the depth compensation value based on the minimum value.

[0083] The depth compensation value refers to the numerical compensation required to convert the actual object distance into depth information (or depth value). This depth compensation value can be determined based on the required depth value of the image region corresponding to the minimum value.

[0084] It is understandable that the actual object distance corresponding to an image region reflects the distance between the object in the scene region corresponding to the image region and the lens or a reference plane such as the focal plane. Therefore, the actual object distance corresponding to an image region can directly or indirectly reflect the distance between the captured object in the image region and the lens. Based on this, the minimum actual object distance corresponding to each image region is the distance between the captured object closest to the lens and the lens among the captured objects in each image region of that image.

[0085] Based on this, since the depth information of an image represents the actual distance between the object presented at each pixel or location in the image and the lens, in order to simplify the complexity of determining the depth information, this application uses the distance between the object closest to the lens and the lens (i.e., the minimum value) as a benchmark to define the distance between the object in other image regions and the lens. In other words, the minimum value of the actual object distance is used as a benchmark to determine the data required to compensate for converting the distances of other actual objects into depth values.

[0086] For example, the depth value of the image region corresponding to the minimum value can be set as the target depth value, and then the difference between the target depth value and the minimum value can be determined as the depth compensation value. For example, if the target depth value is 5 and the minimum value is 2, then the depth compensation value is 3. Alternatively, the absolute value of the minimum value can be determined as the target depth value of the image region corresponding to that minimum value, and the difference between the target depth value and the minimum value can be determined as the depth compensation value.

[0087] In one possible scenario, since the value range of depth information is typically 0-255, in this application, the depth compensation value can be determined by setting the depth information value (i.e., the depth value) of the image region corresponding to the minimum value to 0. Correspondingly, the difference between zero and this minimum value can be determined as the depth compensation value. For example, assuming the minimum actual object distance is 10, then the depth compensation value is -10.

[0088] It is understandable that setting the depth value corresponding to the minimum actual object distance to 0 means that the smaller the actual object distance in the image region, the smaller the depth value of the image region in the image. This not only more intuitively reflects the correspondence between the actual object distance and the depth value, but also makes it easier and more efficient to determine the depth compensation value, which is beneficial for the subsequent more efficient conversion of the depth values ​​of each image region.

[0089] S305, based on the depth compensation value, converts the actual object distance of the acquired object in the image region into the depth value of the image region, thereby obtaining the depth information of the image.

[0090] The depth information includes the depth values ​​of each image region in the image. Of course, since an image region includes at least one pixel, the depth value of each pixel in the image region can be considered as the depth value corresponding to that image region. Based on this, the depth information can also be regarded as including the depth value of each pixel in the image.

[0091] For example, the depth compensation value can be added to the actual distance of the object being captured in the image region to obtain the depth value corresponding to the image region.

[0092] Of course, the specific implementation of converting the actual object distance into a depth value may also differ depending on the method of converting the minimum value into a depth compensation value, and there are no restrictions on this.

[0093] In this embodiment, after determining the actual object distance of the captured object in each image region of the image, the minimum value of the actual object distance corresponding to each image region is determined. Since the actual object distance corresponding to the image region can directly or indirectly reflect the distance between the captured object in the image region and the lens, determining the depth compensation value based on the minimum value of the actual object distance and compensating for the actual object distance of each image region allows the compensated value to reflect the distance between the image region and the lens, and can naturally be used as the depth value of each image region in the image.

[0094] Moreover, by using this minimum value as a benchmark, determining the depth compensation value, and compensating for the actual object distance in each image region based on the depth compensation value, the depth value of each image region in the image can be obtained. This reduces the amount of data computation required to determine the depth information of the image and lowers the complexity of determining the depth information of the image.

[0095] It is understandable that the actual object distance of the object being captured in the image area in this application can be of various possibilities, which will be described below in conjunction with several possible scenarios.

[0096] In one possible scenario, the object distance is the focal distance, which is the distance between the object and its corresponding focal plane. In this case, the conversion relationship between offset and object distance is the same as that between offset and focal distance. Accordingly, based on the configured conversion relationship between offset and focal distance, the actual focal distance corresponding to the actual offset of the image region can be determined, where the actual focal distance is the distance between the captured object and its focal plane in the image region. Based on this, the actual focal distance of the image region can be determined as the actual object distance of the captured object in the image region.

[0097] In this context, the objective lens distance is a positive number. However, the focal distance can be a directional value; therefore, it can be either positive or negative. Relative to the focal plane, if the object is closer to the lens objective (commonly known as the object being in front of the focal plane), the focal distance is negative; conversely, if the object is farther from the lens objective (commonly known as the object being behind the focal plane), the focal distance is positive.

[0098] In this application, the conversion relationship between offset and focal distance is obtained through prior testing.

[0099] For example, in one possible implementation, the conversion relationship between offset and object-focal distance is: a conversion function determined based on multiple measured offsets when there are multiple different object-focal distances between the reference object and the focal plane corresponding to the reference object.

[0100] To facilitate understanding, the process of determining the conversion relationship between offset and focal distance is illustrated below using one implementation method as an example. Figure 4 This illustration shows a flowchart of an implementation process for determining the conversion relationship between offset and focal distance in this application. The method of this embodiment may include:

[0101] S401: When the lens of the image acquisition module has finished focusing on the reference object, determine the first objective lens distance between the reference object and the lens, and control the autofocus function of the image acquisition module to be locked.

[0102] In this application, the reference object used for testing can be selected according to actual needs, without any restrictions. For example, the reference object can be a picture or a doll. In particular, considering that black and white colors have a more prominent contrast and are easier to focus quickly and accurately, the reference object in this application can be an image with a white background and multiple black graphics. For example, the reference object can be a target image output by a monitor, which has a white background and includes multiple black diamond patterns arranged in rows.

[0103] When the lens of the image acquisition module completes focusing on the reference object, it means that the reference object is on its corresponding focal plane. At this time, the image sensor of the image acquisition module can present a clear image of the reference object.

[0104] In this application, objective lens distance refers to the distance between the object (such as a reference object or the object being acquired) and the lens of the image acquisition module. For ease of distinction, when the lens is focused on a reference object, the distance from that reference object to the lens is called the first objective lens distance.

[0105] In this context, locking the autofocus function means that the electronic device controls the image acquisition module to prevent it from starting autofocus, thus fixing the lens position (i.e., fixing the relative position between the lens and the image sensor). When the autofocus function is locked, regardless of the movement of the reference object, the autofocus system, consisting of the phase-detection autofocus unit in the image acquisition module and the motor driving the lens, will stop working and will naturally no longer adjust the lens position.

[0106] S402, with the autofocus function locked, adjusts the objective lens distance between the reference object and the lens, and obtains the test offset determined by the image acquisition module under multiple second objective lens distances between the reference object and the lens, thus obtaining the test offset corresponding to different second objective lens distances.

[0107] The second objective distance differs from the first objective distance. For ease of distinction, the distance between the reference object and the lens when the lens cannot focus on the reference object is called the second objective distance.

[0108] The test offset is the amount of lens movement required for the image acquisition module to focus on the reference object. This test offset can be determined by the phase-detection autofocus unit set in the image sensor of the image acquisition module. For example, the test offset can be the defocus amount output by PDAF.

[0109] Understandably, when the reference object and the lens are at a first objective distance, the lens achieves focus on the reference object (also known as focusing), indicating that no lens adjustment is needed. Therefore, in this case, the test offset determined by the image sensor (based on the phase-detection autofocus unit) is 0. When autofocus is locked, the image acquisition module does not automatically adjust the lens for focusing. Therefore, if the objective distance between the reference object and the lens is not the first objective distance, the lens cannot focus on the reference object. Consequently, the image acquisition module outputs a test offset characterizing the amount of lens adjustment required, and this test offset is not 0.

[0110] S403, for each second objective distance, the focal plane is located with reference to the position of the object at the first objective distance, and the focal distance corresponding to each second objective distance is determined.

[0111] It is understandable that when the distance between the reference object and the lens is the first objective lens distance, the reference object is located at the focal plane corresponding to the reference object. Based on this, when the distance between the reference object and the lens is the second objective lens distance, the difference between the second objective lens distance and the first objective lens distance is the focal distance between the reference object and the focal plane of the reference object.

[0112] For easier understanding, please refer to Figure 5 . Figure 5 Example diagrams showing multiple sets of objective lens distances and focal distances corresponding to the reference object during the test process are shown.

[0113] exist Figure 5 In this example, we'll use the case where the position of the reference object remains unchanged, but the position of the image acquisition module is adjusted to change the objective lens between the lens and the reference object. Of course, the same principle applies if the position of the image acquisition module is kept constant while adjusting the position of the reference object. Figure 5 The following example illustrates this using the image acquisition module outputting the test offset via a PDAF sensor.

[0114] exist Figure 5 The first straight line represents the objective distance between the reference object and the lens, and the second straight line represents the focal distance between the reference object and the focal plane.

[0115] Depend on Figure 5 As can be seen, when the objective distance between the reference object and the lens is 80cm, the reference object is on the focal plane, and at this time, the test offset output by PDAF is 0.

[0116] Based on this, the objective lens distance between the reference object and the lens is adjusted according to a set step size (such as 5cm or 10cm). At different objective lens distances, the test offset output by the image acquisition module can be obtained. Furthermore, since the focal distance between the reference object and the focal plane is 0 when the objective lens distance is 80cm, the focal distance between the reference object and the focal plane can be calculated for other objective lens distances.

[0117] like Figure 5 When the distance between the reference object and the lens objective is 10cm, the focal distance between the reference object and the focal plane (i.e., at a position of 80cm) is -70cm. Therefore, a focal distance of -70cm indicates that the reference object is in front of the focal plane and 70cm away from it. Similarly, when the distance between the reference object and the lens objective is 160cm, the focal distance between the reference object and the focal plane is 80cm.

[0118] S404, based on the test offset and focal distance corresponding to each second objective lens distance, determines the correspondence between multiple sets of test offsets and focal distances.

[0119] As introduced above, there is a one-to-one correspondence between the test offset corresponding to the second objective lens distance and the object focal distance corresponding to the second objective lens distance, thus allowing us to obtain multiple sets of correspondences between test offsets and object focal distances.

[0120] S405, based on the correspondence between the test offset and the object-focus distance of each group, determines the conversion relationship between the offset and the object-focus distance.

[0121] For example, based on the correspondence between the test offsets and the object-focus distance for each group, a conversion function between the offset and the object-focus distance is fitted. There are multiple ways to fit this conversion function, and no specific restrictions are imposed.

[0122] For example, the offset and the focal distance have the following functional relationship:

[0123] ;

[0124] in, and These are different conversion coefficients. By using multiple sets of test offsets and object-focus distances, these two coefficients can be fitted to obtain the conversion function between the offset and the object-focus distance.

[0125] Of course, this example uses a linear function relationship between the offset and the focal distance. In practical applications, the function relationship between the offset and the focal distance can take other forms, which can be set according to actual needs without restriction.

[0126] It is understandable that the above Figure 4 This example illustrates one implementation method that tests the conversion relationship between offset and focal distance. This embodiment is also applicable to other implementation methods.

[0127] To facilitate understanding, the specific implementation process of image depth information is determined by combining the pre-tested conversion relationship between offset and focal distance. The following section will explain this process. Figure 6 Please provide an explanation. For example... Figure 6 This illustration shows another implementation flowchart of the depth information determination method provided in this application. The method in this embodiment may include:

[0128] S601, determine the actual offset of different image regions in the image acquired by the image sensor.

[0129] The way different image regions are divided is related to the distribution of phase focusing units in the image acquisition module. The division method can be pre-configured according to actual needs or determined in real time, without any restrictions.

[0130] The actual offset is used to characterize the amount of lens movement required to achieve focus on the captured object in the image area.

[0131] S602, for each image region, based on the configured conversion relationship between offset and focal distance, determine the actual focal distance corresponding to the actual offset of the image region.

[0132] The actual object-focal distance is the distance between the object being acquired and the focal plane of the object in the image region.

[0133] In this embodiment, the actual focal distance is taken as the actual object distance of the object being captured in the image region. Simultaneously with the image sensor acquiring the image, the image sensor can determine multiple image regions based on the distribution of phase detection elements such as PDAF, and determine the actual lens offset required for each image region. For example... Figure 2 Taking the example of each physical pixel in an image sensor corresponding to one PDAF, in order to reduce the amount of data processing, the image can be divided into 12... There are 13 image regions, each containing multiple pixels. The actual offset of each image region is determined by combining the actual offsets of the PDAF outputs within that region.

[0134] exist Figure 2 Based on this, the actual offset corresponding to each image region can be converted into the distance between the captured object and the focal plane of the captured object in the image region, i.e., the actual object-focal distance, according to the pre-configured conversion relationship.

[0135] S603, determine the minimum actual focal distance corresponding to each image region in the image.

[0136] Since the actual focal distance corresponding to the image region can be positive or negative, the minimum value can also be positive or negative.

[0137] S604 determines the difference between zero and the minimum value as the depth compensation value.

[0138] S605, for each image region in the image, add the actual focal distance of the object in the image region to the depth compensation value to obtain the depth value corresponding to the image region.

[0139] It is understandable that the smaller the actual focal distance corresponding to an image region, the closer the object being captured in that image region is to the lens. In this embodiment, the depth value of the image region corresponding to the minimum actual focal distance is 0. Therefore, the difference between zero and this minimum value is the depth compensation value required to convert the minimum actual focal distance into a depth value of 0. Correspondingly, adding this depth compensation value to the actual focal distances of other image regions yields the depth values ​​of those other image regions relative to the image region with a depth value of 0, thus providing the relative depth value of each image region in the image.

[0140] For easier understanding, please refer to Figure 7 An example diagram is shown that converts the actual offset of an image region into the corresponding depth value of the image region.

[0141] exist Figure 7 The image shows the lens of the image acquisition module and multiple objects within the lens's acquisition range. These objects are designated Object 1, Object 2, Object 3, and Object 4. Each of these four objects is at a different distance from the lens. When the image acquisition module simultaneously acquires these four objects, assuming it needs to focus on Object 2, then Object 2 is located at the focal plane, and each object is at a different distance from this focal plane.

[0142] exist Figure 7 In the middle, three straight lines are shown from top to bottom:

[0143] In this diagram, the values ​​on the first straight line represent the actual offset (e.g., Defocus) required by the image acquisition module to focus on each object. Figure 7 It can be seen that object 2 is the object focused by the image acquisition module. Object 2 is located on its corresponding focal plane. Accordingly, in the image acquired by the image acquisition module, the actual offset of the image region containing at least part of the content of object 1 is 0.

[0144] The second straight line represents the distance of each object from the focal plane. Figure 7 As can be seen, object 1 is in front of the focal plane, and the plane containing object 1 is the foreground. The distance between object 1 and the focal plane is -10cm. Object 2 is 0cm away from the focal plane. Objects 3 and 4 are 10cm and 16cm away from the focal plane, respectively, and are located behind the focal plane. The plane containing these two objects is the background.

[0145] The third line represents the depth values ​​of each object. Figure 7Since object 1 is the closest to the focal plane, its depth value can be set to 0, and its depth compensation value will necessarily be 10. Based on this, the depth value of object 2 is 0 + 10 = 10, and similarly, the depth values ​​of object 3 and object 4 are 20 and 26 respectively. Correspondingly, the depth value of the pixels or image region covered by object 1 is 0, the depth value of the pixels or image region covered by object 2 is 10, the depth value of the pixels or image region covered by object 3 is 20, and the depth value of the pixels or image region covered by object 4 is 26.

[0146] S606 generates depth information of the image based on the depth values ​​of each image region.

[0147] The depth information includes the depth values ​​of each image region in the image.

[0148] It is understandable that image depth information is the distance information of each image region, which can be interpreted as the relative distance of each object to its closest object in the lens. Based on this, and by combining the image's depth information, the relative distance of each object to its closest object in the lens can be reasonably determined. For easier understanding, please refer to... Figure 8 , Figure 8 It shows Figure 2 The depth value of a portion of the image region in the image. Figure 2 Taking the bottom border of a book as an example, where the actual focal distance is the smallest, the depth value of the image area near the border is 0, while the depth value of other image areas is greater than zero.

[0149] Understandably, to further reduce the complexity of determining depth information, the conversion relationship between offset and object distance configured in this application can also be a conversion relationship between offset and objective lens distance. Accordingly, for each image region in the image acquired by the image acquisition module, this application can determine the actual objective lens distance corresponding to the actual offset of the image region based on the configured conversion relationship between offset and objective lens distance. As mentioned earlier, the actual objective lens distance is the distance between the object being acquired in the image region and the lens of the image acquisition module. Based on this, this application can determine the actual objective lens distance as the actual object distance of the object being acquired in the image region.

[0150] To facilitate understanding of this situation, the following will be combined with... Figure 9 Please provide an explanation. For example... Figure 9 This illustration shows another implementation flowchart of the depth information determination method provided in this application. The method in this embodiment may include:

[0151] S901, determine the actual offset of different image regions in the image acquired by the image sensor.

[0152] The actual offset is used to characterize the amount of lens movement required to achieve focus on the captured object in the image area.

[0153] This step can be found in the previous related introduction, and will not be repeated here.

[0154] S902 determines the actual objective distance corresponding to the actual offset of the image region based on the conversion relationship between the configured offset and the objective distance.

[0155] The offset is used to characterize the amount of lens movement required to achieve focus on the reference object.

[0156] The actual objective lens distance is the distance between the object being captured in the image area and the lens of the image acquisition module. In this embodiment, the actual objective lens distance is taken as the actual object distance of the object being captured in the image area.

[0157] In this embodiment, the conversion relationship between the offset and the objective lens distance is a conversion function determined based on multiple measured offsets, given that there are multiple different objective lens distances between the reference object and the lens of the image acquisition module. The specific method for determining this conversion relationship between the offset and the objective lens distance is not limited.

[0158] For example, when determining the conversion relationship between this offset and the objective lens distance, one can do so as follows: Figure 4 As described in the embodiment, multiple test offsets corresponding to different second objective lens distances can be obtained through steps S401 and S402. Based on this, function fitting can be performed by combining the test offsets corresponding to different second objective lens distances to fit the conversion function between objective lens distance and offset. The specific fitting process is not limited.

[0159] S903, determine the minimum actual objective distance corresponding to each image region in the image.

[0160] S904, the difference between zero and this minimum value is determined as the depth compensation value.

[0161] S905, for each image region in the image, add the depth compensation value to the actual objective distance of the object in the image region to obtain the depth value corresponding to the image region.

[0162] The above steps S903 to S905 are similar to the implementation process of the previous embodiment, and will not be described again.

[0163] Depend on Figure 5It can be seen that the closer the reference object is to the focal plane, the closer the reference object is to the lens. Correspondingly, when the minimum objective distance between the reference object and the lens is 10cm, the objective distance is also the smallest, although the depth compensation value will differ. For example, to ensure that the depth value of the image region corresponding to the minimum objective distance is 0, the depth compensation value should be -10; while to ensure that the depth value converted from the minimum objective distance is 0, the depth compensation value should be 70. However, the depth values ​​at other locations remain the same after compensation. For instance, when the objective distance between the lens and the reference object is 80cm, the objective distance of the reference object from the focal plane is 0, but after compensation with their respective depth compensation values, the determined depth value is always 70.

[0164] As can be seen from the above, by changing the objective lens distance between the reference object and the lens, the image offset under different objective lens distances can be directly obtained. Naturally, there is no need to convert the relationship between the offset of the reference object and the focal distance. The conversion relationship between objective lens distance and offset can be determined more conveniently and efficiently, reducing the testing complexity before the electronic equipment is produced, and thus reducing the complexity of determining the depth information of the image.

[0165] S906, Determine the depth information of the image based on the depth values ​​of each image region.

[0166] This step can be found in the previous description and will not be repeated here.

[0167] It is understood that, in any of the above embodiments of this application, after the depth information of the image is determined, there are various application scenarios based on the depth information of the image, and there are no limitations on these.

[0168] For example, in one possible implementation, after determining the depth information of the image, this application can also determine the target object in the image, and accordingly, based on the depth information of the image, perform background blurring processing on other areas in the image outside the area where the target object is located.

[0169] The target object in the image can be a pre-configured object type. For example, the user may have pre-defined the object type to be captured. For instance, if the object type is set to a person, then based on the image's depth information, background blurring can be applied to areas outside the person. Alternatively, the object type could be plants or buildings, with no restrictions.

[0170] The target object in the image can be a target object selected by the user from the image, or an object located in the central area of ​​the image or in focus, without any specific restrictions.

[0171] This application also provides an electronic device in its embodiments. For example... Figure 10 As shown, it illustrates a schematic diagram of the composition structure of the electronic device, which includes at least an image acquisition module 1001 and a processor 1002;

[0172] The image acquisition module 1001 includes an image sensor 1003 and a lens 1004. The image sensor is equipped with at least one phase focusing unit (not shown in the figure).

[0173] Image sensor 1003 is used to determine the actual offset of different image regions in the image acquired by the image sensor based on the phase focusing unit. The actual offset is used to characterize the amount of lens movement required to achieve focusing of the image sensor on the acquired object in the image region.

[0174] The processor 1002 is configured to determine the actual object distance of the captured object in the image region based on the configured conversion relationship between offset and object distance and the actual offset of the image region; and to determine the depth information of the image based on the actual object distance of the captured object in the image region.

[0175] Furthermore, the image acquisition module can also divide the image acquired by the image sensor into at least one image region based on the distribution of the phase focusing units in the image sensor.

[0176] For details regarding the specific operations performed by the image sensor and processor, please refer to the relevant descriptions in the preceding embodiments, which will not be repeated here.

[0177] Furthermore, in this application, the electronic device may also include: a memory 1005 for storing programs required for the processor to perform operations; and a display unit 1006 for displaying the acquired images.

[0178] Of course, the electronic device can also have more than Figure 10 There are no restrictions on the number of components, whether more or fewer.

[0179] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the depth information determination methods provided in this application.

[0180] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the depth information determination methods provided in this application.

[0181] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0183] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0184] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for determining depth information, comprising: determining an actual shift amount of different image regions in an image captured by an image sensor, the actual shift amount being used to represent a required lens shift amount for achieving focus on a captured object in the image region; determining an actual object distance of the captured object in the image region based on a conversion relationship between a configured shift amount and an object distance and the actual shift amount of the image region; determining depth information of the image based on the actual object distance of the captured object in the image region. 2.The method of claim 1, wherein the determining an actual object distance of the captured object in the image region based on a conversion relationship between a configured shift amount and an object distance and the actual shift amount of the image region comprises: determining an actual object distance of the captured object in the image region based on a conversion relationship between a configured shift amount and an object distance and the actual shift amount of the image region. 3.The method of claim 1, wherein the determining an actual object distance of the captured object in the image region based on a conversion relationship between a configured shift amount and an object distance and the actual shift amount of the image region comprises: determining an actual object distance of the captured object in the image region based on a conversion relationship between a configured shift amount and an object distance and the actual shift amount of the image region. 4.The method of any one of claims 1 to 3, wherein the determining depth information of the image based on the actual object distance of the captured object in the image region comprises: determining a minimum value of the actual object distance of each image region in the image; determining a depth compensation value based on the minimum value; converting the actual object distance of the captured object in the image region into a depth value of the image region based on the depth compensation value to obtain the depth information of the image, the depth information comprising the depth value of each image region in the image. determining the depth compensation value as a difference between zero and the minimum value; the converting the actual object distance of the captured object in the image region into a depth value of the image region based on the depth compensation value comprises: adding the actual object distance of the captured object in the image region to the depth compensation value to obtain the depth value of the image region. 6.The method of claim 1, further comprising: dividing the image captured by the image sensor into at least one image region based on a distribution state of phase focus units in the image sensor. ​ ​ ​ 5. The depth information determination method according to claim 4, wherein the determining a depth compensation value based on the minimum value includes: ​ ​ ​ ​ ​ 7. The depth information determination method of claim 6, wherein the dividing the image captured by the image sensor into at least one image region based on the distribution state of the phase focusing units in the image sensor comprises: in response to each physical pixel point in the image sensor having one phase focusing unit, dividing the image captured by the image sensor into at least one image region based on the distribution state of the physical pixel points in the image sensor, each image region comprising at least one pixel point; or in response to a plurality of physical pixel points in the image sensor corresponding to one phase focusing unit, dividing the image captured by the image sensor into at least one image region according to the correspondence between each physical pixel point in the image sensor and the phase focusing unit, each image region comprising a target number of pixel points, the target number being the number of physical pixel points corresponding to a single phase focusing unit.

8. The depth information determination method of claim 1, further comprising: determining a target object in the image; and performing a background blurring process on other regions of the image other than a region in which the target object is located based on the depth information of the image.

9. The depth information determination method of claim 2 or 3, wherein the conversion relationship between the offset and the object-focal distance is a conversion function determined based on a plurality of measured offsets in a case where the reference object and the focal plane corresponding to the reference object have a plurality of different object-focal distances; or the conversion relationship between the offset and the object-lens distance is a conversion function determined based on a plurality of measured offsets in a case where the reference object and the lens of the image capturing module have a plurality of different object-lens distances.

10. An image capturing module and a processor, wherein the image capturing module comprises an image sensor and a lens, the image sensor being provided with at least one phase focusing unit; the image sensor is configured to determine actual offsets of different image regions in an image captured by the image sensor based on the phase focusing unit, the actual offset being used to represent a required lens movement amount for focusing on a captured object in the image region; and the processor is configured to determine an actual object distance of the captured object in the image region based on a conversion relationship between a configured offset and an object distance and the actual offset of the image region, and determine depth information of the image based on the actual object distance of the captured object in the image region.

11. The image capturing module and the processor of claim 10, wherein the conversion relationship between the configured offset and the object distance is a conversion function determined based on a plurality of measured offsets in a case where a reference object and a focal plane corresponding to the reference object have a plurality of different object-focal distances; or the conversion relationship between the configured offset and the object-lens distance is a conversion function determined based on a plurality of measured offsets in a case where the reference object and a lens of the image capturing module have a plurality of different object-lens distances. ​ ​ ​ ​ 10. An electronic device comprising: ​ ​ ​ ​

Images