Shooting method, electronic device, readable medium and program product

Abstract

The invention relates to the technical field of intelligent terminals, and discloses a shooting method, electronic equipment, a readable medium and a program product. The shooting method comprises the following steps: in the process of controlling a main path camera and an auxiliary path camera to shoot, if the difference of line-by-line exposure time of the main path camera and the auxiliary path camera is detected not to meet a shooting condition, for example, the difference of line-by-line exposure time of the main path camera and the auxiliary path camera is greater than an exposure threshold value; if the main road image data and the auxiliary road image data are read line by line, aligning the image data to be shot by the two cameras at a certain moment at a certain moment in the reading time corresponding to each line of image data in the process of reading the main road image data and the auxiliary road image data line by line so as to reduce the exposure time difference of the main road image data and the auxiliary road image data; therefore, the image difference between the main road image data and the auxiliary road image data is reduced, and the accuracy of depth information obtained based on the main road image data and the auxiliary road image data is improved.

Description

Technical Field

[0001] This application relates to the field of smart terminal technology, and in particular to a shooting method, electronic device, readable medium, and program product. Background Technology

[0002] With the increasing sophistication of electronic devices, they can now blur images to highlight the subject and create a better sense of space. Typically, electronic devices use dual cameras (including a main road camera and auxiliary road cameras) to acquire image data from the main road and auxiliary road cameras. A depth image is then calculated using this data, and the depth information of each pixel in the depth image is used to blur the background of the main road image. For example, given that pixels in the foreground have lower depth values ​​and pixels in the background have higher depth values, different degrees of blurring can be applied to pixels with different depth values. This keeps the foreground pixels in the main road image sharp while gradually increasing the blur in the background pixels as the depth value increases, thus blurring the background pixels in the main road image and highlighting the subject.

[0003] When electronic devices capture moving subjects, the image data output by the main road camera and the auxiliary road camera differs significantly. The depth image calculated using these two image data is inaccurate, potentially leading to bokeh failures or false blurring. Bokeh failure refers to the background in the main road image data not being blurred, while false blurring refers to the foreground in the main road image data being blurred. Summary of the Invention

[0004] To address the issue of significant differences between images captured by two cameras in motion scenarios, leading to image blurring failure, this application provides a shooting method, electronic device, readable medium, and program product.

[0005] In a first aspect, embodiments of this application provide a shooting method applied to an electronic device, the electronic device including a first camera and a second camera, the method comprising: detecting a shooting command; determining that the difference between a first progressive exposure time of the first camera and a second progressive exposure time of the second camera does not meet shooting conditions; during the process of the first camera progressively acquiring first image data, adjusting the first progressive exposure time of at least one line of image data to a third progressive exposure time to obtain first image data, wherein the difference between the third progressive exposure time and the second progressive exposure time meets shooting conditions; and obtaining target depth image data based on the first image data and the second image data acquired by the second camera based on the second progressive exposure time.

[0006] In this embodiment of the application, when the electronic device detects the operation of opening the camera application, it can determine that it has received a shooting instruction from the user. For example, the electronic device detects the operation of opening the video recording function of the camera application or clicking the shooting function of the camera application.

[0007] It is understood that the first camera can be a main road camera, and the second camera can be a secondary road camera. In other embodiments, the first camera can be a secondary road camera, and the second camera can be a main road camera.

[0008] In this embodiment, by reducing the difference between the third progressive exposure time of the first camera and the second progressive exposure time of the second camera, the image difference between the first image data and the second image data is reduced, thereby improving the accuracy of the depth information obtained based on the first image data and the second image data. Therefore, in scenarios utilizing depth information, such as image blurring applications, the accuracy of blurring processing is improved, avoiding blurring failures or erroneous blurring.

[0009] In one possible implementation, the first progressive exposure time includes a first progressive exposure duration, the second progressive exposure time includes a second progressive exposure duration, and the shooting conditions include: the difference between the first progressive exposure duration and the second progressive exposure duration is less than or equal to an exposure threshold.

[0010] It is understandable that if the difference between the first progressive exposure time of the first camera and the second progressive exposure time of the second camera does not meet the shooting conditions, it indicates that the difference between the first image data captured by the first camera and the second image data captured by the second camera is large. In this case, the accuracy of the depth information calculated based on the first image data and the second image data is low.

[0011] In some embodiments, the exposure threshold can be determined based on the difference in line-by-line exposure time between the main road camera and the auxiliary road camera when the accuracy of the depth information calculated from the main road image data and the auxiliary road image data is high.

[0012] In one possible implementation, the first progressive exposure duration is longer than the second progressive exposure duration, and during the process of the first camera acquiring the first image data line by line, the first progressive exposure time of at least one line of image data is adjusted to the third progressive exposure time to obtain the first image data, including: based on the first adjustment coefficient, shortening the first progressive exposure duration of at least one line of image data to the third progressive exposure duration during the process of the first camera acquiring the first image data line by line to obtain the first image data.

[0013] In this embodiment, the third progressive exposure duration = the first progressive exposure duration ÷ the first adjustment coefficient.

[0014] It is understandable that by shortening the first line-by-line exposure time to the third line-by-line exposure time, the difference in line-by-line exposure time between the first camera and the second camera can be reduced, so that the reading start times of corresponding lines in the first image data and the second image data are relatively aligned.

[0015] In one possible implementation, during the process of acquiring first image data line by line by the first camera, the first line-by-line exposure time of at least one line of image data is adjusted to the third line-by-line exposure time to obtain the first image data, including: during the process of acquiring first image data line by line by the first camera, the first line-by-line exposure duration of each line of image data is adjusted to the third line-by-line exposure duration to obtain the first image data.

[0016] In one possible implementation, the first adjustment coefficient is determined by: determining the movement speed of the subject captured by the first camera based on historical image data captured by the first camera; and determining the first adjustment coefficient based on the movement speed and the historical image brightness value corresponding to the historical image data.

[0017] In one possible implementation, determining a first adjustment coefficient based on the motion speed and the historical image brightness value corresponding to the historical image data includes: determining a second adjustment coefficient based on the motion speed and the historical image brightness value corresponding to the historical image data, wherein the motion speed is proportional to the second adjustment coefficient, and the historical image brightness value is proportional to the second adjustment coefficient; corresponding to the product of the second adjustment coefficient and the first sensitivity during the process of the first camera acquiring the first image data line by line being less than or equal to the upper limit of the sensitivity of the first camera, and greater than the lower limit of the sensitivity of the first camera, the second adjustment coefficient is used as the first adjustment coefficient; corresponding to the product of the second adjustment coefficient and the first sensitivity being greater than the upper limit of the sensitivity, the ratio of the upper limit of the sensitivity to the first sensitivity is used as the first adjustment coefficient; corresponding to the product of the second adjustment coefficient and the first sensitivity being less than or equal to the lower limit of the sensitivity, the ratio of the lower limit of the sensitivity to the first sensitivity is used as the first adjustment coefficient.

[0018] In one possible implementation, the method further includes: increasing the first sensitivity to a second sensitivity during the process of the first camera acquiring first image data line by line. The method for determining the second sensitivity includes: taking the upper limit of sensitivity as the second sensitivity when the product of the second adjustment coefficient and the first sensitivity is greater than the upper limit of sensitivity of the first camera; and determining the second sensitivity based on the second adjustment coefficient and the first sensitivity when the product of the second adjustment coefficient and the first sensitivity is less than or equal to the upper limit of sensitivity.

[0019] In one possible implementation, the first progressive exposure time further includes the first progressive exposure start time of the first row of pixels in the first image data, and the second progressive exposure time further includes the second progressive exposure start time of the first row of pixels in the second image data. The first progressive exposure start time and the second progressive exposure start time are the same, and the shooting conditions include: the difference between the first readout time of the first image data and the second readout time of the second image data is less than or equal to a preset threshold, wherein the first readout time is equal to the difference between the last readout time and the first readout time of the first image data, and the second readout time is equal to the difference between the last readout time and the first readout time of the second image data.

[0020] In one possible implementation, during the process of acquiring first image data line by line by the first camera, the first line-by-line exposure time of at least one line of image data is adjusted to the third line-by-line exposure time to obtain the first image data, including: during the process of acquiring first image data line by line by the first camera, the first line-by-line exposure start time of the first row of pixels is adjusted to the third line-by-line exposure start time to obtain the first image data.

[0021] In one possible implementation, during the process of acquiring first image data line by line by the first camera, the first line-by-line exposure start time of the first row of pixels is adjusted to the third line-by-line exposure start time to obtain the first image data. This includes: corresponding to a first line-by-line exposure duration greater than a second line-by-line exposure duration, based on a first time, advancing the first line-by-line exposure start time to the third line-by-line exposure start time to obtain the first image data; or, corresponding to a first line-by-line exposure duration less than or equal to a second line-by-line exposure duration, based on a first time, delaying the first line-by-line exposure start time to the third line-by-line exposure start time to obtain the first image data. The time at 1 / n between the last line reading time and the first line reading time of the first image data is the same as the time at 1 / n between the last line reading time and the first line reading time of the second image data.

[0022] In one possible implementation, the first time is determined by: acquiring the first readout time of the first camera and the second readout time of the second camera; calculating the readout time difference based on the first readout time and the second readout time; calculating the exposure duration difference based on the first line-by-line exposure duration and the second line-by-line exposure duration; and determining the first time based on the readout time difference and the exposure duration difference.

[0023] In one possible implementation, the first time is equal to the sum of the ratios of the exposure time difference and the readout time difference to n.

[0024] In one possible implementation, the method further includes: based on the target depth image data, blurring the background image in the second image data to obtain the target image corresponding to the shooting command.

[0025] In a second aspect, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for implementing any of the shooting methods provided by the first aspect and various possible implementations of the first aspect.

[0026] Thirdly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to implement any of the shooting methods provided by the first aspect and various possible implementations of the first aspect.

[0027] Fourthly, embodiments of this application provide a computer program product, which includes computer instructions. When executed by an electronic device, the electronic device implements any of the shooting methods provided by the first aspect and various possible implementations of the first aspect. Attached Figure Description

[0028] Figure 1 A schematic diagram of an image data generation method is shown;

[0029] Figure 2 A schematic diagram of a method for calculating depth values ​​is shown;

[0030] Figure 3 A schematic diagram of a blurred scene is shown;

[0031] Figure 4 A schematic diagram of an image data generation method using a binocular camera is shown.

[0032] Figure 5 According to an embodiment of this application, a flowchart of a shooting method is shown;

[0033] Figure 6 According to an embodiment of this application, a schematic diagram of an image data generation method is shown;

[0034] Figure 7 According to an embodiment of this application, a schematic diagram of another image data generation method is shown;

[0035] Figure 8 According to an embodiment of this application, a flowchart of an exposure duration adjustment method is shown;

[0036] Figure 9 According to an embodiment of this application, a flowchart of a first-time determination method is shown;

[0037] Figure 10 According to an embodiment of this application, a schematic diagram of another image data generation method is shown;

[0038] Figure 11 According to an embodiment of this application, a schematic diagram of another image data generation method is shown;

[0039] Figure 12 According to an embodiment of this application, a schematic diagram of another shooting method is shown;

[0040] Figure 13 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 10 is shown. Detailed Implementation

[0041] The technical terms used in this application will be explained below.

[0042] Exposure time: Exposure time refers to the length of time during photography when the camera shutter is open, allowing light to reach the photosensitive material (such as film or the image sensor of a digital camera). The length of exposure time affects the brightness of the image and the dynamic range of the scene. Exposure time is usually controlled by shutter speed, which determines the length of time the shutter is open. The faster the shutter speed, the shorter the exposure time, and the less light enters the camera; the slower the shutter speed, the longer the exposure time, and the more light enters the camera.

[0043] Where, exposure time = number of exposure rows * line-by-line exposure duration, and line-by-line exposure duration = end time of line-by-line exposure for the i-th pixel - start time of line-by-line exposure for the i-th pixel. The start time of line-by-line exposure refers to the point in time when the image sensor begins to sense light. The end time of line-by-line exposure refers to the point in time when the image sensor outputs image data after completing its light-sensing process.

[0044] For image data exposed line by line, such as... Figure 1 As shown, the progressive exposure start time for the first row of pixels is t1, the progressive exposure end time is t3, and the progressive exposure duration for the first row of pixels is t3-t1; the progressive exposure start time for the second row of pixels is t5, the progressive exposure end time is t7, and the progressive exposure duration for the second row of pixels is t7-t5, and so on. The progressive exposure start time for the kth row of pixels is t16, the progressive exposure end time is t18, and the progressive exposure duration for the kth row of pixels is t18-t16. Figure 1 The image data shown has an exposure start time of t1 and an exposure end time of t18. Figure 1 The exposure time of the image data shown is k×(t3-t1)=k×(t7-t5)=k×(t18-t16).

[0045] As you can understand, progressive exposure means that after the first row of pixels has been exposed, data is read out. Once the first row of pixels has been completely read out, the second row of pixels is exposed, and so on, until the last row of pixels has been completely read out.

[0046] It's understandable that exposure time, aperture size, and ISO sensitivity together determine the amount of light exposure in an image. These three parameters are interrelated and collectively affect the brightness and quality of a photograph.

[0047] ISO sensitivity: ISO sensitivity refers to the degree to which a camera's light-sensitive device (such as the image sensor of film or camcorder) responds to light. ISO sensitivity is usually expressed as an ISO value. A higher ISO value indicates that the photosensitive material is more sensitive to light, allowing for image capture even in darker environments; a lower ISO value indicates that the photosensitive material is less sensitive to light, typically used in well-lit environments.

[0048] Depth map: A grayscale image used to describe the three-dimensional structure of a scene. A depth map includes distance information of the subject from the camera for each pixel.

[0049] When calculating depth images based on main road and auxiliary road image data acquired by binocular cameras, the positional differences (i.e., disparity) of the same objects in the main road and auxiliary road image data can be compared, and then a stereo matching algorithm is used to calculate a disparity map. The disparity map includes the disparity value of each pixel in the scene, and the disparity value is inversely proportional to the depth information. Using the disparity map and the geometric relationship between the cameras, the depth value of each pixel can be calculated, thus obtaining the depth image.

[0050] For example, such as Figure 2 As shown, O1 represents the main camera, O2 represents the auxiliary camera, P represents the same pixel captured by the main camera and the auxiliary camera, D represents the distance from spatial point P to the baseline (bs), i.e., the depth value; bs represents the distance between the main camera and the auxiliary camera; f represents the camera focal length, and d1+d2 represents the parallax. According to the principle of similar triangles, we can obtain D = bs*f / (d1+d2).

[0051] It's understandable that binocular bokeh refers to a technique that uses two cameras to achieve a background blur effect. The main camera and the auxiliary camera have different hardware characteristics, such as lens aperture and sensor, and to ensure image quality, their exposure times also differ. Compared to static scenes, the interaction between the subject and camera exposure in dynamic scenes can easily generate additional parallax changes. These additional parallax changes can be confused with the parallax changes caused by depth changes, resulting in a discrepancy between the calculated parallax value and the parallax caused by depth changes, leading to a deviation in the depth value. Specifically, when the main camera and the auxiliary camera capture images simultaneously, if the subject is a moving object, the different exposure and readout times will cause the object's position in the captured image to shift, resulting in a deviation in the depth value. Furthermore, the movement of the subject may also change the lighting conditions for the main and auxiliary cameras during the exposure time, affecting image brightness and contrast, interfering with the performance of stereo matching algorithms, and leading to inaccurate depth estimation.

[0052] For example, the main road camera captured images such as Figure 3 (a) shows the main road image data, and the auxiliary road camera captured the following data: Figure 3 (b) shows the auxiliary road image data. For better display... Figure 3 (a) and Figure 3 (b) Differences, merge Figure 3 (a) and Figure 3 (b) obtained Figure 3 (c). For example Figure 3 As shown in (c), the position of the subject captured by the main road camera and the auxiliary road camera is significantly different, resulting in inaccurate depth images and potential issues such as failed or incorrect blurring.

[0053] It is understood that the depth images mentioned in the technical solutions of this application can be used not only for image blurring scenarios, but also for other scenarios that require depth information from depth images, such as 3D reconstruction scenarios, such as building reconstruction scenarios, sculpture reconstruction scenarios, virtual reality (VR) and augmented reality (AR), autonomous driving, robot navigation, medical imaging, security monitoring, etc. For ease of explanation, the embodiments of this application mainly take image blurring scenarios as an example.

[0054] The image acquisition process is described below.

[0055] It can be understood that the image acquisition process of a camera includes two parts. The first part is exposure, which refers to the process of light shining on the image sensor. After exposure, the image sensor performs the second part of reading, which refers to the process of the image sensor reading out the data and transmitting it to the image signal processor (ISP) or display device (or the readout process).

[0056] The following section uses line-by-line exposure and line-by-line readout as an example to introduce the image acquisition process.

[0057] like Figure 4 As shown, the image sensor 0 of the main road camera generates main road image data according to the principle of line-by-line exposure. Image sensor 0 starts exposing the first row of pixels in the main road image data at time t1, completes the exposure of the first row of pixels at time t3, and starts reading the first row of pixels in the main road image data. Image sensor 0 completes the reading of the first row of pixels at time t5 and starts exposing the second row of pixels in the main road image data. Image sensor 0 completes the exposure of the second row of pixels at time t7 and starts reading the second row of pixels in the main road image data. Image sensor 0 completes the reading of the second row of pixels at time t9 and begins to expose the third row of pixels in the main path image data. Image sensor 0 completes the exposure of the third row of pixels at time t11 and begins to read the third row of pixels in the main path image data. This process continues until image sensor 0 begins to expose the last row of pixels in the main path image data at time t16, completes the exposure of the last row of pixels at time t18, and begins to read the last row of pixels in the main path image data. Image sensor 0 completes the reading of the last row of pixels at time t20, generating the main path image data.

[0058] It can be understood that the line-by-line exposure time of the main road camera = t3-t1 = t7-t5 = t11-t9 = t18-t16, and the line-by-line reading time of the main road camera = t5-t3 = t9-t7 = t13-t11 = t20-t18.

[0059] Similarly, the auxiliary road image data also follows the line-by-line exposure principle. Image sensor 1 of the auxiliary road camera begins exposing the first row of pixels in the auxiliary road image data at time t1, completes the exposure of the first row of pixels at time t4, and begins reading the first row of pixels in the auxiliary road image data. Image sensor 1 completes the reading of the first row of pixels at time t6 and begins exposing the second row of pixels in the auxiliary road image data. Image sensor 1 completes the exposure of the second row of pixels at time t9 and begins reading the second row of pixels in the auxiliary road image data. Image sensor 1 completes the reading of the second row of pixels at time t11 and begins exposing the third row of pixels in the auxiliary road image data. Image sensor 1 completes the exposure of the third row of pixels at t14 and begins reading the third row of pixels in the auxiliary path image data. This process continues until image sensor 1 begins the exposure of the last row of pixels in the auxiliary path image data at t19, completes the exposure of the last row of pixels at t22, and begins reading the last row of pixels in the auxiliary path image data. Image sensor 1 completes the reading of the last row of pixels at time t24, generating the auxiliary path image data.

[0060] It can be understood that the line-by-line exposure time of the auxiliary road camera = t4-t1 = t9-t6 = t14-t11 = t22-t19, and the line-by-line reading time of the auxiliary road camera = t6-t4 = t11-t9 = t16-t14 = t24-t22.

[0061] pass Figure 4 It can be observed that if the exposure times of the main path image data and the auxiliary path image data are inconsistent, the readout time (or frame output time) of each row of pixels will also be inconsistent. This results in significant differences between the main path image data and the auxiliary path image data, leading to inaccurate depth images. This is especially true when shooting moving subjects in low-light conditions, where the differences in exposure time between the main path image data and the auxiliary path image data are further amplified. The differences in readout time of each row of pixels in the main path image data and the auxiliary path image data are also amplified, further magnifying the image gap between them. This makes the calculated depth image even more inaccurate, potentially leading to bokeh failures or false bokeh effects.

[0062] To address the issue of significant differences in images captured by two cameras in motion scenarios, leading to failed image bokeh, this application provides a shooting method. Specifically, during the shooting process controlled by the main and auxiliary cameras, if the difference in line-by-line exposure time between the main and auxiliary cameras does not meet the shooting conditions (e.g., the difference exceeds an exposure threshold), then for the image data to be captured by the two cameras at a certain moment, during the line-by-line reading of the main and auxiliary image data, a certain moment within the readout time of each line of image data is aligned. This reduces the difference in exposure time between the main and auxiliary image data, thereby reducing image differences between them and improving the accuracy of depth information obtained from the main and auxiliary image data. Thus, in scenarios utilizing depth information, such as image bokeh applications, the accuracy of bokeh processing is improved, avoiding bokeh failure or erroneous bokeh.

[0063] For example, in some embodiments of this application, the electronic device can align the start times (or read times) of reading corresponding rows in the main path image data and the auxiliary path image data. Specifically, considering the difference in the start times of reading the main path image data and the auxiliary path image data, mainly due to the difference in the line-by-line exposure time of the main path camera and the auxiliary path camera, where the line-by-line exposure time includes the line-by-line exposure start time, the line-by-line exposure end time, and the line-by-line exposure duration, and the line-by-line exposure duration = line-by-line exposure end time - line-by-line exposure start time. Therefore, the start times of reading each row of pixels in the main path image data and the auxiliary path image data can be aligned by shortening or lengthening the line-by-line exposure duration of one of the cameras.

[0064] It's understandable that the difference in line-by-line exposure time between the main and auxiliary cameras might not meet the shooting requirements, necessitating low-light shooting scenarios such as nighttime or low-light conditions. In such scenarios, the camera's sensitivity is generally low. Extending the line-by-line exposure time under low sensitivity conditions would reduce image sharpness, and the subject's movement would cause blurring. Therefore, in the above solution, to align the start time of data reading, the line-by-line exposure time of either the main or auxiliary camera can be shortened, while increasing the sensitivity of the corresponding image data. This increases the shared exposure time of the main and auxiliary image data, reducing the difference in line-by-line exposure time between the main and auxiliary cameras, thereby minimizing image differences between the main and auxiliary image data.

[0065] For example, for Figure 4The scenario shown can shorten the line-by-line exposure time of the auxiliary road camera to reduce the difference in line-by-line exposure time between the auxiliary road camera and the main road camera, thereby reducing the image difference between the main road image data and the auxiliary road image data.

[0066] Specifically, in some embodiments of this application, the electronic device can determine the specific exposure time that needs to be shortened based on the preview image of the captured scene or based on the video frames already captured during video recording. For example, it can obtain the brightness value of the preview image or historical image data of the captured scene and the movement speed of the subject being captured, determine a first adjustment coefficient based on the brightness value and movement speed, and reduce the exposure time of the camera with the longer exposure time in the main road camera and auxiliary road camera based on the first adjustment coefficient, and increase the sensitivity.

[0067] It is understandable that by shortening the line-by-line exposure time of the main road camera or the auxiliary road camera, the difference in line-by-line exposure time between the main road camera and the auxiliary road camera can be reduced, so that the reading start time of each corresponding line in the main road image data and the auxiliary road image data is relatively aligned.

[0068] In other embodiments of this application, the electronic device may further align the time at 1 / n between the last read time and the first read time of the main path image data with the time at 1 / n between the last read time and the first read time of the auxiliary path image data. Where 0 < 1 / n < 1, and readout time = last read time - first read time.

[0069] For example, when n=2, for Figure 4 In the scenario shown, the auxiliary road image data can be advanced by one time, where the first time is equal to the difference in line-by-line exposure time between the main road camera and the auxiliary road camera plus the difference in readout time between the main road camera and the auxiliary road camera × 1 / 2; this allows the main road image data and the auxiliary road image data to be aligned at the halfway point of the readout time.

[0070] It is understandable that by aligning the main road image data and the auxiliary road image data at 1 / n of the readout time, the exposure time of the main road camera and the auxiliary road camera can be increased, thereby increasing the light captured by the main road camera and the auxiliary road camera at the same time and reducing the difference between the main road image data and the auxiliary road image data.

[0071] This application applies to electronic devices equipped with binocular cameras, which may include a main camera and an auxiliary camera. In one possible implementation, the binocular camera may include a main camera supporting 1x-3.5x zoom and an auxiliary camera (or ultra-wide-angle camera). In other embodiments, the binocular camera may also include a main camera (or telephoto camera) supporting zooms of 3.5x or higher and an auxiliary camera (or main camera). In still other embodiments, the electronic device may include multiple cameras, such as three cameras, at least two of which can be used to implement the functions of the aforementioned binocular camera; this application does not limit this aspect.

[0072] The following is a detailed description of the imaging method provided in the embodiments of this application, which is applied to electronic devices. Figure 5 A schematic diagram of a shooting method according to an embodiment of this application is shown, such as... Figure 5 As shown, it includes:

[0073] 101: Shooting command detected.

[0074] In this embodiment, when the electronic device detects the operation of opening the camera application, it can determine that it has received a shooting instruction from the user. For example, the electronic device detects the operation of opening the video recording function of the camera application or clicking the shooting function of the camera application. When the electronic device detects a shooting instruction, it can acquire in real time the first progressive exposure time of the main camera, the second progressive exposure time of the auxiliary camera, the brightness value of the image, and the speed of motion of the subject in the captured image, such as the preview image or historical image data.

[0075] 102: Determine whether the difference between the first progressive exposure time of the main road camera and the second progressive exposure time of the auxiliary road camera meets the shooting conditions. If the result is yes, proceed to 105: Obtain the target depth image data based on the main road image data collected by the main road camera based on the first progressive exposure time and the auxiliary road image data collected by the auxiliary road camera based on the second progressive exposure time. If the result is no, proceed to 103: During the process of the main road camera collecting main road image data line by line, adjust the first progressive exposure time of at least one line of image data to the third progressive exposure time to obtain the main road image data, wherein the difference between the third progressive exposure time and the second progressive exposure time meets the shooting conditions.

[0076] In this embodiment, the first progressive exposure time includes a first progressive exposure duration, a first progressive exposure start time, and a first progressive exposure end time, wherein the first progressive exposure duration = the first progressive exposure end time - the first progressive exposure start time. The second progressive exposure time includes a second progressive exposure duration, a second progressive exposure start time, and a second progressive exposure end time, wherein the second progressive exposure duration = the second progressive exposure end time - the second progressive exposure start time.

[0077] In this embodiment, the shooting conditions include the difference between the first line-by-line exposure time and the second line-by-line exposure time being less than or equal to an exposure threshold. The electronic device can acquire the first line-by-line exposure time of the main road camera and the second line-by-line exposure time of the auxiliary road camera, and calculate the difference between the first line-by-line exposure time and the second line-by-line exposure time. If the difference is greater than the exposure threshold, it indicates a large difference between the main road image data captured by the main road camera and the auxiliary road image data captured by the auxiliary road camera. In this case, the accuracy of the depth information calculated based on the main road image data and the auxiliary road image data is low. Therefore, it is determined that the difference between the first line-by-line exposure time of the main road camera and the second line-by-line exposure time of the auxiliary road camera does not meet the shooting conditions. That is, the difference between the first line-by-line exposure time of the main road camera and the second line-by-line exposure time of the auxiliary road camera does not meet the shooting conditions. Proceed to step 103: During the process of acquiring main road image data line-by-line by the main road camera, the first line-by-line exposure time of at least one line of image data is adjusted to a third line-by-line exposure time to obtain main road image data, wherein the difference between the third line-by-line exposure time and the second line-by-line exposure time meets the shooting conditions.

[0078] If the difference is less than or equal to the exposure threshold, it indicates that the difference between the main road image data captured by the main road camera and the auxiliary road image data captured by the auxiliary road camera is small. At this time, the accuracy of the depth information calculated based on the main road image data and the auxiliary road image data is high. It is determined that the difference between the first progressive exposure time of the main road camera and the second progressive exposure time of the auxiliary road camera meets the shooting conditions. That is, the difference between the first progressive exposure time of the main road camera and the second progressive exposure time of the auxiliary road camera meets the shooting conditions. Proceed to 105: Based on the main road image data collected by the main road camera based on the first progressive exposure time and the auxiliary road image data collected by the auxiliary road camera based on the second progressive exposure time, the target depth image data is obtained.

[0079] In some embodiments, the exposure threshold can be determined based on the difference in line-by-line exposure time between the main road camera and the auxiliary road camera when the accuracy of the depth information calculated from the main road image data and the auxiliary road image data is high.

[0080] In this embodiment of the application, the electronic device acquires the first progressive exposure duration of the main road camera and the second progressive exposure duration of the auxiliary road camera, and calculates the difference between the first progressive exposure duration and the second progressive exposure duration. This includes: the electronic device acquires the first progressive exposure duration of the main road camera and the second progressive exposure duration of the auxiliary road camera, aligns the first line reading time of the main road camera and the first line reading time of the auxiliary road camera, and then calculates the difference between the first progressive exposure duration and the second progressive exposure duration.

[0081] For example, for Figure 6 The scene shown, Figure 6 (a) shows the first progressive exposure duration (expo0) of the main road image data acquired by image sensor 0 of the main road camera, which is t2-t1, and the first line readout time is t2; the second progressive exposure duration (expo0) of the auxiliary road image data acquired by image sensor 1 of the auxiliary road camera is t4-t3, and the first line readout time is t4. The first line readout time t2 of the main road camera and the first line readout time t4 of the auxiliary road camera can be aligned to obtain... Figure 6 (b) shows a schematic diagram, which determines the difference between the first progressive exposure duration and the second progressive exposure duration (△expo) = t1 - t3.

[0082] For example, regarding Figure 4 In the scenario shown, the first progressive exposure time of the main road camera is t3-t1, and the first line readout time is t3; the second progressive exposure time of the auxiliary road camera is t4-t1, and the first line readout time is t4. The first line readout time t3 of the main road camera can be aligned with the first line readout time t4 of the auxiliary road camera to obtain... Figure 7 The diagram shows the difference between the first progressive exposure duration and the second progressive exposure duration, which is t2 - t1.

[0083] It is understood that, in the embodiments of this application, the electronic device can align the first line readout time of the main camera with the first line readout time of the auxiliary camera to calculate the difference between the first line-by-line exposure time and the second line-by-line exposure time. In other embodiments, the electronic device can align the first line readout time of the auxiliary camera with the first line readout time of the main camera to calculate the difference between the first line-by-line exposure time and the second line-by-line exposure time.

[0084] In other embodiments, the shooting conditions include the difference between the first readout time of the first image data and the second readout time of the second image data being less than or equal to a preset threshold, wherein the first readout time = the last line readout time of the first image data - the first line readout time of the first image data, and the second readout time = the last line readout time of the second image data - the first line readout time of the second image data. If the judgment result is yes, then proceed to 105: obtain target depth image data based on the main road image data collected by the main road camera based on the first line-by-line exposure time and the auxiliary road image data collected by the auxiliary road camera based on the second line-by-line exposure time. If the judgment result is no, then proceed to 103: during the process of the main road camera collecting main road image data line by line, adjust the first line-by-line exposure time of at least one line of image data to the third line-by-line exposure time to obtain main road image data, wherein the difference between the third line-by-line exposure time and the second line-by-line exposure time satisfies the shooting conditions; the preset threshold can be determined based on empirical values.

[0085] In other embodiments, the shooting conditions include the difference between the first line-by-line exposure duration and the second line-by-line exposure duration being less than or equal to an exposure threshold; and the difference between the first readout time of the first image data and the second readout time of the second image data being less than or equal to a preset threshold. If the difference between the first line-by-line exposure duration and the second line-by-line exposure duration is greater than the exposure threshold, and the difference between the first readout time of the first image data and the second readout time of the second image data is greater than the preset threshold, then the determination result is negative, and proceed to step 103: During the process of acquiring main road image data line by line by the main road camera, the first line-by-line exposure time of at least one line of image data is adjusted to a third line-by-line exposure time to obtain main road image data, wherein the difference between the third line-by-line exposure time and the second line-by-line exposure time satisfies the shooting conditions. If the difference between the first progressive exposure time and the second progressive exposure time is less than or equal to the exposure threshold; and / or, the difference between the first readout time of the first image data and the second readout time of the second image data is less than or equal to the preset threshold, then the judgment result is yes, and proceed to 105: Based on the main road image data collected by the main road camera based on the first progressive exposure time and the auxiliary road image data collected by the auxiliary road camera based on the second progressive exposure time, the target depth image data is obtained.

[0086] 103: During the process of collecting main road image data line by line by the main road camera, the first line exposure time of at least one line of image data is adjusted to the third line exposure time to obtain the main road image data, wherein the difference between the third line exposure time and the second line exposure time satisfies the shooting conditions.

[0087] In this embodiment, corresponding to the determination that the first progressive exposure duration of the main road camera is greater than the second exposure duration of the auxiliary road camera, during the process of the main road camera acquiring main road image data line by line, the first progressive exposure duration of at least one line of image data can be shortened to a third progressive exposure duration, so that the difference between the third progressive exposure duration and the second progressive exposure duration meets the shooting conditions; that is, the difference between the third progressive exposure duration and the second progressive exposure duration is less than or equal to the exposure threshold, or the difference between the third progressive exposure time and the second progressive exposure time meets the shooting conditions; that is, the difference between the third progressive exposure time and the second progressive exposure time is less than or equal to the exposure threshold. The progressive exposure duration can be adjusted according to the movement speed of the subject in the captured image, historical image data, or preview image, as well as the brightness value of the image. The method for adjusting the progressive exposure duration is as follows: Figure 8 The embodiments shown will be described in detail here, and will not be repeated.

[0088] Corresponding to the determination that the first line-by-line exposure time of the main road camera is less than or equal to the second exposure time of the auxiliary road camera, during the process of the auxiliary road camera collecting auxiliary road image data line by line, the second line-by-line exposure time of at least one line of image data can be shortened to the fourth line-by-line exposure time, so that the difference between the fourth line-by-line exposure time and the first line-by-line exposure time meets the shooting conditions, even if the difference between the fourth line-by-line exposure time and the first line-by-line exposure time is less than or equal to the exposure threshold.

[0089] For example, for Figure 4 The scenario shown can shorten the line-by-line exposure time of the auxiliary road camera to reduce the difference in line-by-line exposure time between the auxiliary road camera and the main road camera, thereby reducing the image difference between the main road image data and the auxiliary road image data.

[0090] In other embodiments, corresponding to the determination that the first progressive exposure duration of the main road camera is greater than the second exposure duration of the auxiliary road camera, the first progressive exposure duration of each line of image data can be shortened to a third progressive exposure duration during the process of the main road camera acquiring main road image data line by line, so that the difference between the third progressive exposure duration and the second progressive exposure duration meets the shooting conditions, even if the difference between the third progressive exposure duration and the second progressive exposure duration is less than or equal to the exposure threshold, or in other words, the difference between the third progressive exposure time and the second progressive exposure time meets the shooting conditions, even if the difference between the third progressive exposure time and the second progressive exposure time is less than or equal to the exposure threshold.

[0091] Corresponding to the determination that the first line-by-line exposure time of the main road camera is less than or equal to the second exposure time of the auxiliary road camera, the second line-by-line exposure time of each line of image data can be shortened to the fourth line-by-line exposure time during the process of the auxiliary road camera collecting auxiliary road image data line by line, so that the difference between the fourth line-by-line exposure time and the first line-by-line exposure time meets the shooting conditions, that is, the difference between the fourth line-by-line exposure time and the first line-by-line exposure time is less than or equal to the exposure threshold.

[0092] For example, for Figure 4 The scenario shown can shorten the line-by-line exposure time of the auxiliary road camera to reduce the difference in line-by-line exposure time between the auxiliary road camera and the main road camera, thereby reducing the image difference between the main road image data and the auxiliary road image data.

[0093] In other embodiments, corresponding to the electronic device determining that the first progressive exposure duration of the main road camera is greater than the second exposure duration of the auxiliary road camera, the second progressive exposure duration of at least one line of image data during the process of the auxiliary road camera acquiring main road image data line by line can be amplified to a fourth progressive exposure duration, so that the difference between the fourth progressive exposure duration and the first progressive exposure duration meets the shooting conditions; that is, the difference between the fourth progressive exposure duration and the first progressive exposure duration is less than or equal to the exposure threshold. Corresponding to the electronic device determining that the first progressive exposure duration of the main road camera is less than or equal to the second exposure duration of the auxiliary road camera, the first progressive exposure duration of at least one line of image data during the process of the main road camera acquiring main road image data line by line can be amplified to a third progressive exposure duration, so that the difference between the third progressive exposure duration and the second progressive exposure duration meets the shooting conditions; that is, the difference between the third progressive exposure duration and the second progressive exposure duration is less than or equal to the exposure threshold.

[0094] In other embodiments, corresponding to the electronic device determining that the first progressive exposure duration of the main road camera is greater than the second exposure duration of the auxiliary road camera, the second progressive exposure duration of each line of image data acquired line by line by the auxiliary road camera can be amplified to a fourth progressive exposure duration, so that the difference between the fourth progressive exposure duration and the first progressive exposure duration meets the shooting conditions; that is, the difference between the fourth progressive exposure duration and the first progressive exposure duration is less than or equal to the exposure threshold. Corresponding to the electronic device determining that the first progressive exposure duration of the main road camera is less than or equal to the second exposure duration of the auxiliary road camera, the first progressive exposure duration of each line of image data acquired line by line by the main road camera can be amplified to a third progressive exposure duration, so that the difference between the third progressive exposure duration and the second progressive exposure duration meets the shooting conditions, that is, the difference between the third progressive exposure duration and the second progressive exposure duration is less than or equal to the exposure threshold.

[0095] In this way, by increasing or decreasing the line-by-line exposure time of one of the main road cameras and the auxiliary road cameras, the start time of reading each row of pixels in the main road image data and the auxiliary road image data are aligned, thereby increasing the exposure time of the main road image data and the auxiliary road image data and reducing the difference in line-by-line exposure time between the main road camera and the auxiliary road camera, thus reducing the image difference between the main road image data and the auxiliary road image data.

[0096] It is understood that, in some embodiments, the electronic device may also align the main path image data and the auxiliary path image data at 1 / n of the readout time. Here, 0 < 1 / n < 1, and the readout time refers to the time from reading out the first row of pixels to reading out the last row of pixels.

[0097] For example, electronic devices can advance the exposure start time of the camera with the longer exposure time among the main and auxiliary cameras by a first time; or, they can delay the exposure start time of the camera with the shorter exposure time among the main and auxiliary cameras by a first time, thereby aligning the main and auxiliary image data at the moment of 1 / n of the readout time. The method for determining the first time is described in... Figure 9 The embodiments shown will be described in detail here, and will not be repeated.

[0098] In other embodiments, the electronic device can shorten the exposure time of the camera with the longer exposure time among the main camera and auxiliary camera, and advance the exposure start time by an earlier time. In other embodiments, the electronic device can also extend the exposure time of the camera with the shorter exposure time among the main camera and auxiliary camera, and delay the exposure start time by an earlier time.

[0099] 104: Target depth image data is obtained based on the main road image data and the auxiliary road image data collected by the auxiliary road camera based on the second line-by-line exposure time.

[0100] In this embodiment, the difference between the third progressive exposure duration of the main camera and the second progressive exposure duration of the auxiliary camera is less than or equal to the exposure threshold. Therefore, the reading start times of corresponding rows in the main image data captured by the main camera based on the third progressive exposure duration and the auxiliary image data collected by the auxiliary camera based on the second progressive exposure duration are relatively aligned. Even when shooting moving subjects in low-light shooting scenarios, such as night scenes or low-light scenes, the image difference between the main image data and the auxiliary image data is small, which improves the accuracy of the depth information of the target depth image data obtained based on the main image data and the auxiliary image data.

[0101] In the embodiments of this application, the format of the main road image data and the format of the auxiliary road image data can be any one of the following: raw (RAW) format, JPEG format, PNG format, etc.

[0102] As we can understand, RAW format images, also known as raw image data, are the original data generated when an image sensor converts captured light source signals into digital signals. RAW files record the camera's raw information, including metadata such as ISO settings, shutter speed, aperture value, and white balance. RAW format is unprocessed and uncompressed.

[0103] In some embodiments of this application, the electronic device can blur the background image in the main road image data based on the target depth image data to obtain the target image corresponding to the shooting command. In other embodiments, the electronic device can blur the background image in the auxiliary road image data based on the target depth image data to obtain the target image corresponding to the shooting command.

[0104] 105: Target depth image data is obtained based on the main road image data collected by the main road camera based on the first line-by-line exposure time and the auxiliary road image data collected by the auxiliary road camera based on the second line-by-line exposure time.

[0105] It is understandable that the difference between the first progressive exposure time of the main road camera and the second progressive exposure time of the auxiliary road camera meets the shooting conditions. The difference between the main road image data collected by the main road camera based on the first progressive exposure time and the auxiliary road image data collected by the auxiliary road camera based on the second progressive exposure time is small, and the accuracy of the depth information of the obtained target depth image data is high.

[0106] In some embodiments of this application, the electronic device can blur the background image in the main road image data based on the target depth image data to obtain the target image corresponding to the shooting command. In other embodiments, the electronic device can blur the background image in the auxiliary road image data based on the target depth image data to obtain the target image corresponding to the shooting command.

[0107] It is understandable that when the difference in exposure time between the main road image data and the auxiliary road image data does not meet the shooting conditions, the position of the moving subject in the image will inevitably differ, and the greater the difference in exposure time between lines, the greater the difference in the position of the moving subject in the image. This application aligns a certain moment within the readout time of each line of image data during the process of reading the main road image data and the auxiliary road image data line by line, thereby reducing the difference in exposure time between the main road image data and the auxiliary road image data, thus reducing the image differences between the two and improving the accuracy of the depth information obtained based on the main road image data and the auxiliary road image data. Therefore, in scenarios utilizing depth information, such as image blurring applications, the accuracy of blurring processing is improved, avoiding blurring failures or erroneous blurring.

[0108] The following section introduces the method for adjusting the line-by-line exposure time. For example... Figure 8 As shown, it includes:

[0109] 201: Obtain the speed of motion of the subject being photographed.

[0110] In this embodiment, when the electronic device detects a shooting command, it can acquire the movement speed of the subject in the main image data in real time. In other embodiments, when the electronic device detects a shooting command, it can acquire the movement speed of the subject in the auxiliary image data in real time. The shooting command includes operations such as the user opening a shooting application on the electronic device, such as a camera application, opening the video recording function of the shooting application, or clicking the shooting function of the shooting application.

[0111] For example, when an electronic device detects a shooting command, it can detect changes in the position of the subject in consecutive frames of the main image data and calculate the subject's speed based on the distance and time of these changes. For instance, the electronic device can use a deep learning model, such as YOLOv8, to detect the subject in the main image data and track its position in consecutive frames, estimating its speed based on the changes in its position and the time intervals between frames.

[0112] 202: Obtain the brightness value (LV) of the captured image.

[0113] In the embodiments of this application, when the electronic device detects the operation of opening the camera application, it can determine that it has received a user's shooting instruction. For example, the electronic device detects the operation of opening the video recording function of the camera application or clicking the shooting function of the camera application. When the electronic device detects a shooting instruction, it can acquire the brightness value (LV) of the captured image in the main channel image data in real time. In other embodiments, when the electronic device detects a shooting instruction, it can acquire the brightness value (LV) of the captured image in the auxiliary channel image data in real time. The captured image includes at least one of a preview image of the captured image and historical image data.

[0114] In some embodiments, the electronic device may use the average brightness of the main path image data or the auxiliary path image data as the brightness value LV of the scene, that is, add up all the pixel values ​​of the main path image data or the auxiliary path image data and divide by the total number of pixels to obtain the average brightness, and use the average brightness as the brightness value LV of the current scene.

[0115] In other embodiments, the electronic device may also use the OpenCV library to read the main path image data or the auxiliary path image data, convert the main path image data or the auxiliary path image data into grayscale images, and then calculate brightness statistics such as average, median, maximum and minimum values ​​to determine the brightness value LV of the main path image data or the auxiliary path image data.

[0116] It is understood that in some embodiments, step 201 (obtaining the speed of the subject being photographed) can be executed first, followed by step 202 (obtaining the brightness value LV of the captured image). In other embodiments, step 202 (obtaining the brightness value LV of the captured image) can be executed first, followed by step 201 (obtaining the speed of the subject being photographed). In still other embodiments, steps 201 (obtaining the speed of the subject being photographed) and 202 (obtaining the brightness value LV of the captured image) can be executed simultaneously.

[0117] 203: Determine the velocity gain R and velocity weight.

[0118] In this embodiment, the electronic device can calculate the speed gain R and the speed factor weight based on the brightness value LV and the movement speed of the subject being photographed. It can be understood that the speed gain R and the speed factor weight can be empirical values ​​calculated to achieve optimal results in actual shooting, and can be obtained by looking up a table during practical use. It can be understood that the faster the movement speed of the subject, the greater the speed gain R. When the scene brightness is insufficient, the weight of the speed gain R can be reduced to ensure controllable noise.

[0119] 204: Determine the second adjustment factor (ratio2 = R * weight + (1 - weight) * 1.0).

[0120] In the embodiments of this application, the electronic device can calculate ratio2 (or the second adjustment coefficient) based on the speed gain R and the speed factor weight, where ratio2 = R × weight + (1 - weight) × 1.0, or ratio2 = R × weight + (1 - weight). Ratio2 is used to adjust the exposure time and the first sensitivity (or the original sensitivity of the image data). The method of adjusting the exposure time and the first sensitivity based on ratio2 is described below.

[0121] It's understandable that the faster the subject moves, the greater the speed gain R, and the greater the second adjustment coefficient; that is, the speed of movement is directly proportional to the second adjustment coefficient. When the scene brightness is insufficient, to ensure controllable noise, the weight of the speed gain R can be reduced, i.e., the second adjustment coefficient can be reduced, meaning the historical image brightness value is directly proportional to the second adjustment coefficient.

[0122] 205: Determine whether the product of the first ISO and the second adjustment factor is greater than the upper limit of ISO (gain*ratio > threshold).

[0123] To ensure that the overall brightness of the image data remains unchanged after adjusting the exposure time, the camera of an electronic device also needs to consider the ISO sensitivity when adjusting the exposure time. For example, when shortening the exposure time, the initial ISO sensitivity is increased; or when lengthening the exposure time, the initial ISO sensitivity is decreased. The initial ISO sensitivity can be identified by a gain value.

[0124] It can be understood that the original exposure time (expoTime) × first ISO (gain) = new exposure time (newExpoTime) × second ISO (newGain); where exposure time × ISO represents the exposure value, which is used to measure the overall exposure level of the image, the first ISO is the original ISO of the camera to be adjusted, and the original exposure time is the line-by-line exposure time of the camera to be adjusted.

[0125] In this embodiment, to avoid the adjusted sensitivity from exceeding the maximum gain value that can control the image noise, or in other words, the adjusted sensitivity from exceeding the upper limit of sensitivity (ThreshHold), ratio2 can be detected, i.e., it can be determined whether gain × ratio2 is greater than threshold. If the determination result is yes, then proceed to 207: the second sensitivity is equal to the upper limit of sensitivity, and the first adjustment coefficient is equal to the ratio of the upper limit of sensitivity to the first sensitivity (newGain = threshold, Ratio1 = threshold / gain). If the determination result is no, then proceed to 206: the first adjustment coefficient is equal to the second adjustment coefficient, and the second sensitivity is equal to the product of the first sensitivity and the first adjustment coefficient (ratio1 = ratio2, newGain = gain * ratio1).

[0126] In other embodiments, to prevent the adjusted sensitivity (or second sensitivity) from being less than or equal to the minimum gain value for controllable image noise, or the adjusted sensitivity (or second sensitivity) from being less than or equal to the lower limit of sensitivity, ratio2 can be detected, i.e., it can be determined whether gain × ratio2 is less than or equal to the lower limit of sensitivity; if the determination result is yes, then newGain = lower limit of sensitivity, ratio1 = lower limit of sensitivity / gain, newExpoTime = ExpoTime / ratio1; if the determination result is no, then ratio1 = ratio2, newGain = gain * ratio1, newExpoTime = ExpoTime / ratio1.

[0127] 206: The first adjustment factor is equal to the second adjustment factor, and the second sensitivity is equal to the product of the first sensitivity and the first adjustment factor (ratio1 = ratio2, newGain = gain * ratio1).

[0128] It is understandable that, for cases where the product of the first ISO and the second adjustment factor is less than or equal to the upper limit of ISO, the second ISO is equal to the first adjustment factor of the first ISO.

[0129] 207: The second sensitivity is equal to the upper limit of sensitivity, and the first adjustment factor is equal to the ratio of the upper limit of sensitivity to the first sensitivity (newGain = threshold, Ratio1 = threshold / gain).

[0130] It is understandable that when the product of the first ISO and the second adjustment factor is greater than the upper limit of ISO, the second ISO is the upper limit of ISO, and the first adjustment factor is the ratio of the upper limit of ISO to the first ISO.

[0131] 208: The new exposure time is equal to the ratio of the original exposure time to the first adjustment factor (newExpoTime = exposureTime / ratio).

[0132] In this embodiment, when the product of the first sensitivity and the second adjustment factor is less than or equal to the upper limit of sensitivity, the electronic device can set the sensitivity of the camera with the longer exposure time among the main camera and auxiliary camera to a multiple of the original sensitivity by the first adjustment factor, and set the new exposure time to a fraction of the original exposure time by the first adjustment factor. When the product of the first sensitivity and the second adjustment factor is greater than the upper limit of sensitivity, the sensitivity of the camera with the longer exposure time among the main camera and auxiliary camera can be set to the upper limit of sensitivity, the first adjustment factor can be set to the ratio of the upper limit of sensitivity to the first sensitivity, and the new exposure time can be set to a fraction of the original exposure time by the first adjustment factor.

[0133] Thus, this embodiment of the application adjusts the difference in line-by-line exposure time between the main road camera and the auxiliary road camera, while also adjusting the sensitivity of the image data. This achieves the goal of reducing the difference in line-by-line exposure time between the main road camera and the auxiliary road camera, while also ensuring the clarity of the main road image data and the auxiliary road image data.

[0134] The following example, using n=2, illustrates the method for determining the first time step. Figure 9 As shown, it includes:

[0135] 301: Obtain the readout time T0 of image sensor 0.

[0136] In this embodiment of the application, the electronic device can first determine the readout time T0 of the image sensor 0 of the main camera, where T0 = the first line readout time of the main camera - the last line readout time of the main camera, the first line readout time refers to the time when the first line of pixels is read, and the last line readout time refers to the time when the last line of pixels is read.

[0137] For example, for Figure 4 In the scenario shown, the first line of the main road image data is read at time t3 and the last line is read at time t18, then T0 = t18 - t3.

[0138] For example, regarding Figure 10 In the scenario shown in (a), the first line of the main road image data is read at time t2 and the last line is read at time t3, then T0 = t3 - t2.

[0139] 302: Obtain the readout time T1 of image sensor 1.

[0140] In this embodiment of the application, the electronic device can determine the readout time T1 of the image sensor 1 of the auxiliary channel camera, where T1 = first line readout time of the auxiliary channel camera - last line readout time of the auxiliary channel camera, the first line readout time refers to the time when the first line of pixels is read, and the last line readout time refers to the time when the last line of pixels is read.

[0141] For example, for Figure 4 In the scenario shown, the first line of auxiliary road image data is read at time t4 and the last line is read at time t22, then T1 = t22 - t4.

[0142] For example, regarding Figure 10 In the scenario shown in (a), the first line of auxiliary road image data is read at time t5 and the last line is read at time t6, then T1 = t6 - t5.

[0143] 303: Calculate the time difference ΔT (ΔT = T1 - T0) based on the readout time T0 and the readout time T1.

[0144] In this embodiment of the application, the electronic device can calculate the difference between the readout time of the image sensor 0 of the main road camera and the readout time of the image sensor 1 of the auxiliary road camera to obtain the readout time difference ΔT.

[0145] For example, for Figure 4 In the scenario shown, ΔT = t22 - t4 - (t18 - t3).

[0146] For example, for Figure 10 In the scenario shown, ΔT = t6 - t5 - (t3 - t2).

[0147] 304: Obtain the progressive exposure time expoTime0 for image sensor 0.

[0148] In this embodiment of the application, the electronic device can calculate the line-by-line exposure time based on the exposure start time and exposure end time of any row of pixels collected by the image sensor 0 of the main camera. That is, the line-by-line exposure time = the exposure end time of the i-th row of pixels - the exposure start time of the i-th row of pixels.

[0149] For example, for Figure 4 In the scenario shown, the exposure start time of the first row of pixels in the main image data is t1, and the exposure end time is t3. Therefore, expoTime0 = t3 - t1.

[0150] For example, regarding Figure 10 In the scenario shown in (a), the exposure start time of the first row of pixels in the main image data is t1 and the exposure end time is t2. Therefore, expoTime0 = t2 - t1.

[0151] 305: Obtain the progressive exposure time expoTime1 of image sensor 1.

[0152] For example, for Figure 4 In the scenario shown, the exposure start time of the first row of pixels in the auxiliary path image data is t1, and the exposure end time is t4. Therefore, expoTime1 = t4 - t1.

[0153] For example, regarding Figure 10 In the scenario shown in (a), the exposure start time of the first row of pixels in the auxiliary path image data is t4 and the exposure end time is t5. Therefore, expoTime1 = t5 - t4.

[0154] 306: Calculate the exposure time difference Δexpo (Δexpo=expoTime1-expoTime0) based on the exposure time expoTime0 and the exposure time expoTime1.

[0155] For example, for Figure 4 In the scenario shown, Δexpo = t4 - t1 - (t3 - t1).

[0156] For example, regarding Figure 10 In the scenario shown in (a), Δexpo = t5 - t4 - (t2 - t1).

[0157] 307: Calculate the first time (start0-start1=Δexpo+Δt / 2) based on the readout time difference ΔT and the exposure duration difference Δexpo.

[0158] In the embodiments of this application, (start0-start1) represents the first time, that is, the exposure advance time of the camera with a longer exposure time among the main road cameras or auxiliary road cameras; or the exposure delay time of the camera with a shorter exposure time among the main road cameras or auxiliary road cameras.

[0159] For example, for Figure 10 In the scenario shown in (a), the electronic device can advance the exposure time of the auxiliary road camera from start0-start1=Δexpo+Δt / 2, thus obtaining... Figure 10 (b) shows the schematic diagram.

[0160] For example, for Figure 4 In the scenario shown, the electronic device can delay the exposure time of the main camera by start0 - start1 = Δexpo + Δt / 2, resulting in... Figure 11 The diagram shown is shown in the image.

[0161] In other embodiments, the exposure time of the main or auxiliary camera may not be adjusted; only by... Figure 9The embodiment shown aligns the main road image data and auxiliary road image data at the time of 1 / n between the last line read time and the first line read time.

[0162] Thus, by aligning the main path image data and auxiliary path image data at the 1 / n time between the last readout time and the first readout time, the common exposure time and readout time of the main path image data and auxiliary path image data can be improved, thereby enhancing the consistency of the captured identical photons and the consistency of the image.

[0163] The following is based on Figure 12 Taking this as an example, the shooting method of this application will be introduced. For example... Figure 12 As shown, the process includes: After the main camera's image sensor 0 acquires the main stream (or main image data), it splits the main stream into two streams. One stream (tiny stream) is used for perception algorithm processing to determine ratio1, and the other stream is used as input to buffer 0. After the auxiliary camera's image sensor 1 acquires the auxiliary stream (or auxiliary image data), it is input to buffer 1. Then, the data stored in buffer 0 and buffer 1 are optimized by optimizing the exposure strategy, and processed by a blurring algorithm to obtain the final blurred image. The method for determining ratio1 is described in [link to documentation]. Figure 8 The embodiment shown illustrates a method for optimizing the data stored in cache 0 and cache 1 by optimizing the exposure strategy. (See also...) Figure 5 The embodiments shown will not be described in detail here.

[0164] In this embodiment, the exposure strategy of the main road camera or the auxiliary road camera is adjusted according to the scene information and exposure information output from the perception algorithm and the two sensors to reduce the exposure time difference between the main road image data and the auxiliary road image data, thereby reducing the image difference between the main road image data and the auxiliary road image data and improving the accuracy of the depth information obtained based on the main road image data and the auxiliary road image data.

[0165] It is understood that the technical solutions of this application are applicable to electronic devices with cameras, such as, but not limited to, mobile phones, smartwatches, televisions, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments of this invention do not impose any restrictions on the specific type of electronic device.

[0166] The hardware structure of the electronic device mentioned in this application will be described below using electronic device 10 as an example. Figure 13 As shown, the electronic device 10 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, buttons 101, and a display screen 102, etc.

[0167] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0168] Processor 110 may include one or more processing units, such as processing modules or circuits of a central processing unit (CPU), graphics processing unit (GPU), digital signal processing (DSP), microprocessor (MCU), artificial intelligence (AI) processor, or field-programmable gate array (FPGA). Different processing units may be independent devices or integrated within one or more processors. Processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in processor 110 is a cache memory 180.

[0169] It is understood that the shooting method in this embodiment can be executed by the processor 110 of the corresponding electronic device. The power module 140 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component is used to manage the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module is used to receive charging input from the charger; the power management module is used to connect the power supply and the processor 110. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 110, the display screen 102, the camera 170, and the wireless communication module 120, etc.

[0170] The mobile communication module 130 may include, but is not limited to, antennas, power amplifiers, filters, and low-noise amplifiers (LNAs). The mobile communication module 130 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the electronic device 10. The mobile communication module 130 can receive electromagnetic waves via the antenna, filter and amplify the received electromagnetic waves, and then transmit them to a modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some functional modules of the mobile communication module 130 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 130 and at least some modules of the processor 110 may be housed in the same device.

[0171] The wireless communication module 120 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 120 can provide solutions for wireless communication applications on the electronic device 10, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The electronic device 10 can communicate with networks and other devices through wireless communication technologies.

[0172] It is understood that, in this embodiment of the application, when the electronic device is a receiving device, it can receive video and audio data from other electronic devices in the recording group, as well as the recording content tags and time stamp information corresponding to each data, through the wireless communication module. And when the electronic device is a transmitting device, it can send video and audio data, as well as the recording content tags and time stamp information corresponding to each data, to other electronic devices in the recording group through the wireless communication module.

[0173] In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the electronic device 10 may also be located in the same module.

[0174] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a quantum dot light-emitting diode (QLED), etc.

[0175] The sensor module 190 may include proximity sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0176] Audio module 150 is used to convert digital audio information into analog audio signal output, or to convert analog audio input into digital audio signal. Audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, audio module 150 may be located in processor 110, or some functional modules of audio module 150 may be located in processor 110. In some embodiments, audio module 150 may include a speaker, earpiece, microphone, and headphone jack. Camera 170 is used to capture still images or videos. An object generates an optical image through the lens and projects it onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to image signal processing (ISP) to convert it into a digital image signal. Electronic device 10 can implement shooting functions through ISP, camera 170, video codec, graphics processing unit (GPU), display screen 102, and application processor, etc.

[0177] Interface module 160 includes an external storage interface, a USB interface, and a subscriber identification module (SIM) card interface. The external storage interface can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the electronic device 10. The external storage card communicates with the processor 110 through the external storage interface to perform data storage. The universal serial bus interface is used for communication between the electronic device 10 and other electronic devices. The subscriber identification module card interface is used to communicate with the SIM card installed in the electronic device 10, for example, to read or write phone numbers stored in the SIM card.

[0178] In some embodiments, the electronic device 10 further includes buttons 101, a motor, and indicators. The buttons 101 may include volume buttons, a power button, etc. The motor is used to generate a vibration effect in the electronic device 10, for example, vibrating when the user's electronic device 10 is called to prompt the user to answer the call. The indicators may include laser indicators, radio frequency indicators, LED indicators, etc.

[0179] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0180] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0181] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0182] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc-read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0183] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0184] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0185] It should be noted that in the examples and description of this patent, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Although this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application.

Claims

1. A shooting method applied to an electronic device, the electronic device comprising a first camera and a second camera, characterized in that, The method includes: A shooting command has been detected. It was determined that the difference between the first progressive exposure time of the first camera and the second progressive exposure time of the second camera did not meet the shooting conditions. During the process of acquiring first image data line by line by the first camera, the first line exposure time of at least one line of image data is adjusted to the third line exposure time to obtain the first image data, wherein the difference between the third line exposure time and the second line exposure time satisfies the shooting conditions. Target depth image data is obtained based on the first image data and the second image data acquired by the second camera based on the second line-by-line exposure time.

2. The shooting method according to claim 1, characterized in that, The first line-by-line exposure time includes the first line-by-line exposure duration, and the second line-by-line exposure time includes the second line-by-line exposure duration. The shooting conditions include: The difference between the first line-by-line exposure duration and the second line-by-line exposure duration is less than or equal to the exposure threshold.

3. The shooting method according to claim 2, characterized in that, The first progressive exposure duration is greater than the second progressive exposure duration, and During the process of acquiring first image data line by line by the first camera, the first line-by-line exposure time of at least one line of image data is adjusted to the third line-by-line exposure time to obtain the first image data, including: Based on the first adjustment coefficient, during the process of the first camera acquiring the first image data line by line, the first line-by-line exposure time of at least one line of image data is shortened to the third line-by-line exposure time to obtain the first image data.

4. The shooting method according to claim 3, characterized in that, In the process of acquiring first image data line by line by the first camera, the first line-by-line exposure time of at least one line of image data is adjusted to a third line-by-line exposure time to obtain the first image data, including: During the process of acquiring first image data line by line by the first camera, the first line-by-line exposure time of each line of image data is adjusted to the third line-by-line exposure time to obtain the first image data.

5. The shooting method according to claim 3, characterized in that, The first adjustment factor is determined in the following way: Based on historical image data captured by the first camera, the movement speed of the subject captured by the first camera is determined; The first adjustment coefficient is determined based on the motion speed and the historical image brightness value corresponding to the historical image data.

6. The shooting method according to claim 5, characterized in that, The step of determining the first adjustment coefficient based on the motion speed and the historical image brightness value corresponding to the historical image data includes: A second adjustment coefficient is determined based on the motion speed and the historical image brightness value corresponding to the historical image data, wherein the motion speed is directly proportional to the second adjustment coefficient, and the historical image brightness value is directly proportional to the second adjustment coefficient. If the product of the second adjustment coefficient and the first sensitivity during the process of the first camera acquiring the first image data line by line is less than or equal to the upper limit of the sensitivity of the first camera, and greater than the lower limit of the sensitivity of the first camera, then the second adjustment coefficient is used as the first adjustment coefficient. If the product of the second adjustment coefficient and the first sensitivity is greater than the upper limit of sensitivity, the ratio of the upper limit of sensitivity to the first sensitivity is used as the first adjustment coefficient. If the product of the second adjustment coefficient and the first sensitivity is less than or equal to the lower limit of sensitivity, the ratio of the lower limit of sensitivity to the first sensitivity is used as the first adjustment coefficient.

7. The shooting method according to claim 6, characterized in that, Also includes: The first sensitivity is increased to a second sensitivity during the process of the first camera acquiring the first image data line by line, wherein the method for determining the second sensitivity includes: If the product of the second adjustment coefficient and the first sensitivity is greater than the upper limit of the sensitivity of the first camera, the upper limit of the sensitivity is used as the second sensitivity. If the product of the second adjustment coefficient and the first sensitivity is less than or equal to the upper limit of sensitivity, the second sensitivity is determined based on the second adjustment coefficient and the first sensitivity.

8. The shooting method according to claim 1, characterized in that, The first line-by-line exposure time also includes the first line-by-line exposure start time of the first row of pixels in the first image data, and the second line-by-line exposure time also includes the second line-by-line exposure start time of the first row of pixels in the second image data. The first line-by-line exposure start time and the second line-by-line exposure start time are the same. The shooting conditions include: The difference between the first readout time of the first image data and the second readout time of the second image data is less than or equal to a preset threshold, wherein the first readout time is equal to the difference between the last readout time and the first readout time of the first image data, and the second readout time is equal to the difference between the last readout time and the first readout time of the second image data.

9. The shooting method according to claim 8, characterized in that, In the process of acquiring first image data line by line by the first camera, the first line-by-line exposure time of at least one line of image data is adjusted to a third line-by-line exposure time to obtain the first image data, including: During the process of acquiring first image data line by line by the first camera, the first line exposure start time of the first row of pixels is adjusted to the third line exposure start time to obtain the first image data.

10. The shooting method according to claim 9, characterized in that, In the process of acquiring first image data line by line by the first camera, the first line-by-line exposure start time of the first row of pixels is adjusted to the third line-by-line exposure start time to obtain the first image data, including: Since the first progressive exposure duration is longer than the second progressive exposure duration, based on the first time, the start time of the first progressive exposure is advanced to the start time of the third progressive exposure to obtain the first image data; or... Corresponding to the first progressive exposure duration being less than or equal to the second progressive exposure duration, based on the first time, the start time of the first progressive exposure is delayed to the start time of the third progressive exposure to obtain the first image data, wherein the time at 1 / n between the last line reading time and the first line reading time of the first image data is the same as the time at 1 / n between the last line reading time and the first line reading time of the second image data.

11. The shooting method according to claim 10, characterized in that, The first time is determined in the following way: Obtain the first readout time of the first camera and the second readout time of the second camera; Calculate the readout time difference based on the first readout time and the second readout time; Calculate the exposure time difference based on the first line-by-line exposure time and the second line-by-line exposure time; The first time is determined based on the readout time difference and the exposure duration difference.

12. The shooting method according to claim 11, characterized in that, The first time is equal to the sum of the exposure duration difference and the ratio of the readout time difference to n.

13. The shooting method according to claim 1, characterized in that, Also includes: Based on the target depth image data, the background image in the second image data is blurred to obtain the target image corresponding to the shooting command.

14. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of the electronic device, wherein the processor is one of the one or more processors of the electronic device for performing the imaging method according to any one of claims 1 to 13.

15. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the shooting method according to any one of claims 1 to 13.

16. A computer program product, characterized in that, The computer program product includes computer instructions, which, when executed by an electronic device, enable the electronic device to perform the photographing method as described in any one of claims 1 to 13.

Images