Image processing method and apparatus, electronic device, and computer-readable storage medium

Abstract

An image processing method, comprising: acquiring first image data and second image data which are output by a first photoelectric sensor, and third image data output by a second photoelectric sensor, the brightness of the first image data being greater than that of the second image data, and the brightness of the first image data being greater than that of the third image data (202); fusing the second image data and the first image data to obtain fused image data (204); determining depth information on the basis of the second image data and the third image data (206); and performing blurring processing on the fused image data on the basis of the depth information to obtain a target image (208).

Description

Image processing method, device, electronic device, and computer-readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on January 7, 2024, with application number 202410020555.6 and invention name “Image processing method, device, electronic device and computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of images, and in particular to an image processing method, device, electronic device, and computer-readable storage medium. Background Art

[0004] The essence of Portrait mode is to provide a composition method. By blurring the foreground and background, it reduces the cost of scene portrait composition, emphasizes the subject, strengthens the sense of layering and improves the expressiveness of the photo. The characteristics of Portrait mode: it is infinitely close to the effect of commercial shooting (fixed-focus SLR + photo editing + lighting).

[0005] In traditional technology, the preview process and the frame process of taking pictures are shot with high brightness images. For posed shots where the subject is stationary, and for shots where the subject is moving, the depth information is inaccurate and the blurring effect is poor. Summary of the Invention

[0006] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can accurately calculate depth information to accurately blur an image.

[0007] In a first aspect, the present application provides an image processing method. The method comprises:

[0008] Acquire first image data and second image data output by a first photosensor, and third image data output by a second photosensor; the brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data;

[0009] fusing the second image data with the first image data to obtain fused image data;

[0010] determining depth information based on the second image data and the third image data;

[0011] The fused image data is blurred according to the depth information to obtain a target image.

[0012] In a second aspect, the present application further provides an image processing device. The device comprises:

[0013] an acquisition module, configured to acquire first image data and second image data output by a first photosensor, and third image data output by a second photosensor; wherein the brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data;

[0014] a fusion module, configured to fuse the second image data with the first image data to obtain fused image data;

[0015] a depth calculation module, configured to determine depth information based on the second image data and the third image data;

[0016] The blurring module is configured to blur the fused image data according to the depth information to obtain a target image.

[0017] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it performs the following operations:

[0018] Acquire first image data and second image data output by a first photosensor, and third image data output by a second photosensor; the brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data;

[0019] fusing the second image data with the first image data to obtain fused image data;

[0020] determining depth information based on the second image data and the third image data;

[0021] The fused image data is blurred according to the depth information to obtain a target image.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following operations:

[0023] Acquire first image data and second image data output by a first photosensor, and third image data output by a second photosensor; the brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data;

[0024] fusing the second image data with the first image data to obtain fused image data;

[0025] determining depth information based on the second image data and the third image data;

[0026] The fused image data is blurred according to the depth information to obtain a target image.

[0027] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following operations:

[0028] Acquire first image data and second image data output by a first photosensor, and third image data output by a second photosensor; the brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data;

[0029] fusing the second image data with the first image data to obtain fused image data;

[0030] determining depth information based on the second image data and the third image data;

[0031] The fused image data is blurred according to the depth information to obtain a target image.

[0032] The above-mentioned image processing method obtains first and second image data output by a first photosensor, and third image data output by a second photosensor. The brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data. Therefore, the first image data can more clearly represent the information of stationary objects, and in the overexposed area corresponding to the first image data, the second image data can more clearly represent the information in this overexposed area, which can make the fused image data correspond to a higher dynamic range. Because the brightness of the second and third image data is lower than that of the first image data, the second and third image data are less likely to be blurred by moving areas, and there are corresponding positional differences between the second and third image data, more accurate depth information can be obtained, and the phenomenon of false blurring and missing blurring can be avoided. Finally, blurring is performed based on this depth information, and this depth information can be accurately represented in the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, without paying any creative work, they can also obtain drawings of other embodiments based on these drawings.

[0034] FIG1 is a diagram showing an application environment of an image processing method in one embodiment.

[0035] FIG2 is a flow chart of a photographing process of an image processing method in one embodiment.

[0036] FIG3 is a specific photographing flowchart of an image processing method in one embodiment.

[0037] FIG4 is a schematic diagram of a photographing effect in one embodiment.

[0038] FIG5 is a preview flowchart of an image processing method in one embodiment.

[0039] FIG6 is a preview effect diagram in one embodiment.

[0040] FIG7 is a flowchart of blurring processing in one embodiment.

[0041] FIG8 is a flowchart of selecting reference data in one embodiment.

[0042] FIG9 is a schematic diagram of exposure duration for image data acquisition in one embodiment.

[0043] FIG10 is a schematic diagram of selecting image pairs using a ZSL strategy in one embodiment.

[0044] FIG11 is a schematic diagram of selecting image pairs for a static scene in one embodiment.

[0045] FIG. 12 is a schematic diagram of selecting image pairs when the motion range is satisfied in one embodiment.

[0046] FIG13 is a flowchart of brightness adjustment of background image data in one embodiment.

[0047] FIG14 is a structural block diagram of an image processing apparatus in one embodiment.

[0048] FIG15 is a diagram of a computer device according to one embodiment. DETAILED DESCRIPTION

[0049] To facilitate understanding of the present application, a more comprehensive description of the present application will be provided below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present application.

[0050] The image processing method provided in the embodiments of the present application can be applied in the application environment shown in FIG1 . In this embodiment, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104 or placed on a cloud or other network server.

[0051] Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0052] In an exemplary embodiment, as shown in FIG2 , an image processing method is provided. The method is described by taking the application of the method to the terminal 102 in FIG1 as an example, and includes the following operations:

[0053] In operation 202 , first image data and second image data output by a first photosensor and third image data output by a second photosensor are acquired; the brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data.

[0054] The first photoelectric sensor and the second photoelectric sensor are used to convert light signals acquired at different angles into image data respectively. The first sensor may be a master sensor, and the second sensor may be a slave sensor. The first sensor is used to output data acquired by the first camera, and the second sensor is used to output data acquired by the second camera. The first photoelectric sensor may use a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) to sense light signals, and the second photoelectric sensor may use a charge coupled device or a complementary metal oxide semiconductor sensor to sense light signals.

[0055] The first image data is image data output by the first photoelectric sensor. The brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data, so that the first image data can more clearly represent the information of the stationary object.

[0056] Because the brightness of the first image data is greater than that of the second image data, the second image data can more clearly represent the information in the overexposed area corresponding to the first image data. Furthermore, because both the second image data and the first image data are output by the first photosensor, the difference in image depth between the two is minimal.

[0057] The third image data is image data output by the second photosensor. Because the first photosensor and the second photosensor process image data collected at different locations, the corresponding depth information can be calculated by combining the third image data with either the first or second image data. The second and third image data can be used to more accurately determine depth information.

[0058] Optionally, the brightness of the first image data, the second image data, and the third image data can be controlled by exposure time or aperture size. Optionally, when brightness is controlled by exposure time, exposure time and brightness are positively correlated; optionally, the exposure time of the first image data is greater than the exposure time of the second image data, and the exposure time of the first image data is greater than the exposure time of the third image data, so that the brightness of the first image data is greater than the brightness of the second image data, and the brightness of the first image data is greater than the brightness of the third image data.

[0059] When controlling brightness by exposure time, if there is a moving area in the first image data, the moving area of ​​the first image data may be blurred. However, since the exposure time of the second image data and the third image data is shorter than that of the first image data, the second image data and the third image data are still clear in this moving area.

[0060] In an exemplary embodiment, acquiring first image data and second image data output by a first photosensor, and third image data output by a second photosensor includes acquiring first image data and second image data output by a main photosensor, and third image data output by a slave photosensor.

[0061] In operation 204 , the second image data is fused with the first image data to obtain fused image data.

[0062] The fused image data is obtained by fusing image data of different brightness output by the same photoelectric sensor. Since the brightness of the second image data is different from that of the first image data, the image corresponding to the fused image data can have a higher dynamic range.

[0063] In one embodiment, fusing the second image data with the first image data includes: determining reference data and non-reference data in the second image data and the first image data; and fusing the non-reference data with the reference data corresponding to an image position. The corresponding image position may refer to corresponding pixels or corresponding image features.

[0064] In another embodiment, fusing the second image data with the first image data includes: determining an image acquisition scene; when the image acquisition scene is the first scene, using the first image data as reference data, fusing the second image data into reference data corresponding to the image position; when the image acquisition scene is the second scene, determining reference data and non-reference data in the second image data and the first image data; and fusing the non-reference data into the reference data corresponding to the image position. For example, the first scene is a static scene, and the second scene is a non-static scene. The non-static scenes can be further refined, in order of increasing dynamic range, into situations that satisfy a first motion range condition and situations that satisfy a second motion range condition.

[0065] In operation 206 , depth information is determined based on the second image data and the third image data.

[0066] Depth information is spatial position information in image data. Since the brightness of the second image data and the third image data is lower than that of the first image data, the second image data and the third image data are less likely to be blurred due to motion.

[0067] Because the second image data and the first image data are both output by the first sensor, the depth information corresponding to the second image data and the first image data is relatively close. And because the second image data and the third image data are output by different sensors, there are corresponding positional differences between the two image data, which can obtain more accurate depth information to avoid the phenomenon of false false information. False false information refers to the falsification of depth information during the image processing process; false false information refers to the inadequate extraction and utilization of image features during the image processing and analysis process, resulting in inaccurate depth information.

[0068] In an optional embodiment, determining depth information based on the second image data and the third image data includes: comparing the position difference of corresponding pixels in the second image data and the third image data to determine the depth information of the object; the position difference is the parallax of the same scene by photoelectric sensors at different positions.

[0069] In one optional embodiment, determining depth information based on the second image data and the third image data includes determining a disparity between corresponding pixels in the second image data and the third image data, and determining the depth information based on the disparity. For example, the electronic device may generate a depth map in which a value corresponding to each pixel represents the scene depth corresponding to the pixel.

[0070] In an optional embodiment, depth information is determined based on the second image data and the third image data, including: performing perspective correction on the second image data and the third image data, respectively, to obtain corrected second image data and corrected third image data, the perspectives of the corrected second image data and the corrected third image data corresponding to each other; extracting corner points or SIFT feature points from the corrected second image data and the corrected third image data to obtain feature points to be matched respectively from the second image data and the third image data; matching the feature points to be matched respectively from the second image data and the third image data to establish corresponding feature point pairs; calculating the depth value of each pixel according to the correspondence between the corresponding feature point pairs and the positional relationship between the first photoelectric sensor and the second photoelectric sensor.

[0071] In operation 208 , blurring is performed on the fused image data according to the depth information to obtain a target image.

[0072] In an optional embodiment, blurring the fused image data based on the depth information to obtain a target image includes blurring the fused image data based on a blur value indicated by the depth information to obtain the target image. Optionally, the blur value indicated by the depth information may be positively correlated with the depth information at different pixel locations, i.e., a greater depth information corresponds to a greater blur value. Optionally, the blur value indicated by the depth information may be related to a capture mode; optionally, a correspondence between the depth information and the blur value is determined based on the capture mode; and based on the correspondence, the depth information is mapped to the corresponding blur value.

[0073] In an optional embodiment, blurring the fused image data based on the depth information to obtain a target image includes blurring the fused image data based on blurring methods corresponding to different pixel locations according to the depth information to obtain the target image. The pixel regions affected by the blurring methods include, but are not limited to, foreground, background, portrait, and region of interest; and the blurring methods used include, but are not limited to, Gaussian blur or radial blur.

[0074] In an optional embodiment, the fused image data is blurred according to the depth information to obtain a target image, including: blurring the fused image data according to the blurring method corresponding to different pixel positions according to the depth information; blurring the fused image data according to the blurring method according to the blurring value indicated by the depth information to obtain the target image.

[0075] Depth information is used for blurring, and the depth information can be represented by the brightness, color, or texture of different pixels in the target image. Optionally, the target image is a photographed image, which is stored in an album or certain folders. After the target image is obtained, it can be displayed.

[0076] In the above-described image processing method, first and second image data output by a first photosensor, as well as third image data output by a second photosensor, are acquired. The brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data. Therefore, the first image data can more clearly represent information about stationary objects, and in overexposed areas corresponding to the first image data, the second image data can more clearly represent information in these overexposed areas, allowing the fused image data to correspond to a higher dynamic range. Because the brightness of both the second and third image data is lower than that of the first image data, the second and third image data are less susceptible to blurring due to motion, and the corresponding positional differences between the second and third image data allow for more accurate depth information to be obtained, while avoiding false or missed blurs. Finally, blurring is performed based on this depth information, allowing this depth information to be accurately represented within the target image.

[0077] In an exemplary embodiment, as shown in FIG7 , blurring the fused image data according to the depth information to obtain the target image includes:

[0078] In operation 702 , an image deformation matrix is ​​determined based on the first image data and the second image data.

[0079] The image deformation matrix is ​​a global position matrix of the first image data and the second image data. The image deformation matrix is ​​used to represent the relative position relationship of pixels between the first image data and the second image data. Optionally, the image deformation matrix is ​​a wrap matrix.

[0080] Since the first image data and the second image data are both output by the first photosensor, there is a frame difference between the first image data and the second image data. Therefore, the corresponding relationship between the first image data and the second image data at each pixel position needs to be represented by an image deformation matrix.

[0081] In an optional embodiment, determining an image deformation matrix based on the first image data and the second image data includes: performing feature point matching on each pixel in the first image data and each pixel in the second image data to obtain feature point pairs; and determining the image deformation matrix based on the pixel position difference in the feature point pairs.

[0082] In an optional embodiment, determining an image deformation matrix based on the first image data and the second image data includes: determining matching lines in the first image data and the second image data based on line features of the first image data and line features of the second image data; and determining the image deformation matrix based on pixel position differences in the matching lines. The matching lines may be obtained through a Hough transform.

[0083] In operation 704 , the depth information is converted into converted depth information corresponding to the first image data according to the image deformation matrix.

[0084] The converted depth information is adapted to the first image data. By converting the depth information according to the image deformation matrix, the original depth information can be adaptively adjusted to the frame difference between the first and second image data, thereby more accurately representing the depth information of the fused image data.

[0085] Optionally, the depth information may be mapped according to the image deformation matrix to obtain converted depth information corresponding to the first image data; the depth information may be mapped according to the image deformation matrix and other matrices respectively to obtain converted depth information corresponding to the first image data.

[0086] Operation 706 : blurring the fused image data according to the converted depth information to obtain a target image.

[0087] In an optional embodiment, blurring the fused image data according to the converted depth information to obtain the target image includes: blurring the fused image data according to the blur value indicated by the converted depth information to obtain the target image.

[0088] In an optional embodiment, blurring the fused image data according to the converted depth information to obtain a target image includes: blurring the fused image data according to the blurring method corresponding to different pixel positions of the converted depth information to obtain a target image.

[0089] In an optional embodiment, the fused image data is blurred according to the converted depth information to obtain a target image, including: blurring the fused image data according to the blurring method corresponding to different pixel positions according to the converted depth information; blurring the fused image data according to the blurring method according to the blurring value indicated by the converted depth information to obtain the target image.

[0090] In this embodiment, the image deformation matrix is ​​used to convert depth information into converted depth information suitable for fusing the unreliable image data. This converted depth information is more accurate, and the fused image data is then blurred based on this converted depth information, allowing the target image to more accurately represent the information in the actual captured scene. Therefore, at least in static scenes and scenes that partially meet the first motion range condition, this embodiment allows the target image to more accurately represent the converted depth information.

[0091] In one embodiment, the second image data is fused with the first image data to obtain fused image data, including: in the motion area, determining the second image data and the regional deformation matrix of the second image data; in the motion area, correcting the second image data by the regional deformation matrix to obtain corrected motion area data; in the motion area, fusing the first image data with the corrected motion area data to obtain fused image data.

[0092] Correspondingly, blurring the fused image data according to the depth information to obtain the target image includes: blurring the fused image data according to the depth information and the regional deformation matrix to obtain the target image.

[0093] The motion region is characterized by partial image data, indicating the region where a moving object resides in the image data. A moving object is an object that moves during capture. Optionally, the motion region is determined based on at least one of the first and second image data. Optionally, because both the first and second image data are output by the first photosensor, the time at which they are output differs. Therefore, optical flow calculation can be performed on the first and second image data to determine the motion region. Optionally, feature extraction at different scales can be performed on one of the first and second image data, and the motion region can be determined based on the features at different scales.

[0094] The regional deformation matrix is ​​a deformation matrix of the second image data and the second image data in the motion region, and is used to characterize the degree of distortion between the second image data and the second image data in the motion region. Optionally, the regional deformation matrix can be used to determine the pixel relationship between the second image data and the second image data in the motion region. The regional deformation matrix differs from the image deformation matrix: the regional deformation matrix is ​​a deformation matrix for a local area of ​​the image, while the image deformation matrix is ​​a deformation matrix for the entire image.

[0095] In an optional embodiment, in the motion area, the regional deformation matrix of the second image data and the second image data is determined, including: performing feature point matching on each pixel of the first image data in the motion area and each pixel of the second image data in the motion area to obtain feature point pairs of the motion area; and determining the image deformation matrix based on the pixel position difference in the feature point pairs of the motion area.

[0096] In an optional embodiment, determining a regional deformation matrix between the second image data and the second image data in the motion region includes: matching lines of the first image data and the second image data in the motion region based on line features of the first image data and the second image data in the motion region to obtain regional matching lines; and determining an image deformation matrix based on pixel position differences in the regional matching lines. The regional matching lines may be obtained via a Hough transform.

[0097] In an optional embodiment, in the motion area, the second image data is corrected by a regional deformation matrix to obtain corrected motion area data, including: in the motion area, the same group of adjacent pixel points of the second image data are offset corrected by a regional deformation matrix to obtain corrected motion area data; wherein, the same group of adjacent pixel points are adjacent pixel points, which can be four-neighborhood pixel points or eight-neighborhood pixel points; thereby, correction of the offset position is achieved.

[0098] In an optional embodiment, fusing the first image data with the corrected motion region data in the motion region to obtain fused image data includes fusing the corrected motion region data into the motion region of the first image data to obtain fused image data. Thus, using the first image data as reference data, the corrected motion region data is fused into this reference data, resulting in an image represented by the fused image data having higher definition.

[0099] In another optional embodiment, in the motion region, the first image data and the corrected motion region data are fused to obtain fused image data, including: replacing the motion region of the first image data with the corrected motion region data to obtain the fused image data.

[0100] In an optional embodiment, the fused image data is blurred according to the depth information and the regional deformation matrix to obtain a target image, including: deforming the pixels of the fused image data in the motion area according to the regional deformation matrix to obtain the fused image data after regional deformation; blurring the fused image data after regional deformation according to the depth information to obtain the target image.

[0101] In an optional embodiment, blurring the fused image data based on the depth information to obtain a target image includes: determining an image deformation matrix based on the first image data and the second image data; converting the depth information into converted depth information corresponding to the first image data according to the image deformation matrix; and blurring the fused image data based on the converted depth information to obtain the target image. Blurring the fused image data based on the converted depth information includes: blurring the fused image data based on the converted depth information and the regional deformation matrix to obtain the target image. Thus, combining the converted depth information with the regional deformation matrix improves the blurring effect in the moving area.

[0102] In this embodiment, since the brightness of the second image data is relatively low and the exposure time is relatively low, the clarity of the second image data in the motion area can be relatively high. In this case, the motion area of ​​the second image data is corrected separately through the regional deformation matrix, so that the corrected motion area data can more accurately represent the information of the motion area in the actual scene. Then, according to the depth information and the regional deformation matrix, the fused image data is blurred, so that the target image can more accurately represent the depth information of the motion area. Therefore, at least in the scene that partially meets the first motion range condition, the target image can be more accurately represented by this embodiment. The depth information is guaranteed to be overall image quality and clarity. In addition, replacing the original depth information with the above-mentioned converted depth information can enable the target image to more accurately represent the information of the motion area in the actual shooting scene.

[0103] The second image data and the third image data are collected synchronously.

[0104] Because the second image data and the third image data are output by different photoelectric sensors, the second image data and the third image data can be collected synchronously. When the two image data are output synchronously, the second image data and the third image data can be synchronized in time, that is, the two data can achieve frame synchronization.

[0105] Based on this, since synchronous acquisition will not be affected by the shooting environment, the depth information can be calculated more accurately based on the second image data and the third image data, so as to avoid the phenomenon of false positives and false negatives with a higher probability.

[0106] In an exemplary embodiment, fusing the second image data with the first image data to obtain fused image data includes: taking the first image data as reference data, fusing the second image data with the first image data to obtain fused image data.

[0107] In an optional embodiment, the first image data is used as reference data, and the second image data is fused with the first image data to obtain fused image data, including: fusing the second image data into the first image data according to the correspondence between pixel positions to obtain fused image data.

[0108] In an optional embodiment, the first image data is used as reference data, and the second image data is fused with the first image data to obtain fused image data, including: in the motion area, according to the correspondence between pixel positions, the second image data is fused into the first image data to obtain fused image data.

[0109] In an optional embodiment, the first image data is used as the reference data, and the second image data is fused with the first image data to obtain fused image data, including: using the content of the first image data as the reference content, adjusting the second image data according to the reference content to obtain adjusted second image data; and fusing the adjusted second image data into the first image data to obtain fused image data.

[0110] In this embodiment, since the brightness of the first image data is greater than the brightness of the second image data, the clarity of the first image data in most areas can be higher, thereby ensuring higher image quality.

[0111] In an exemplary embodiment, 2exp is used as input, meaning the sensor simultaneously outputs two frames: the first image data and the second image data. The primary sensor employs Less Blanking MultiFrame (LBMF) and digital overlap double exposure (2DOL) calculations. This ensures that snapshots are based on short frames, enabling clear motion capture. Furthermore, depth information is calculated using short frames, ensuring depth accuracy. This is because short frames are clear for both the primary and secondary cameras, enabling better matching.

[0112] In an exemplary embodiment, as shown in FIG3 , the first photoelectric sensor (main sensor) outputs first image data (long frames) and second image data (short frames) to an image front-end engine (IFE); after the image front-end engine processes the first image data and the second image data respectively, the image front-end engine inputs the processing result of the first image data and the processing result of the second image data by the image front-end engine into the Bayer processing segment (Bayer Processing Segment, Bayer) for processing, and the processing result of the Bayer processing segment is input into the image re-interpretation engine (Turbo raw) for fusion to obtain fused image data, and the fused image data is blurred (bokeh) according to the depth information to obtain the target image.

[0113] Correspondingly, the second photosensor (slave sensor) outputs third image data (short frames) for binning. The binned third image data is then input to the image front-end engine (IFE). After processing the binned third image data, the IFE inputs the results of the IFE's processing of the third image data to a Bayer processing segment (Bayer). The Bayer processing segment then inputs the results of the IFE's processing of the third image data to an image reinterpretation engine (Turbo Raw). This converts the second image data (short frames) into master-base raw data and the third image data (short frames) into slave-base raw data. The master-base raw data and the slave-base raw data are then input to an image processing engine (IPE) to obtain YUV data from the master and slave sensors. Depth information is determined based on the YUV data from the master and slave sensors.

[0114] The images generated by the first image data are shown in Figures (a) and (c) in Figure 4, and the target images are shown in Figures (b) and (d) in Figure 4. The moving figures and fallen leaves in the target images can be clearly displayed.

[0115] In an exemplary embodiment, the method further includes: in the case of preview, acquiring fourth image data and fifth image data output by the first photoelectric sensor, the brightness of the fourth image data being greater than that of the fifth image data; fusing the fourth image data and the fifth image data to obtain fused preview image data; and generating a preview image based on the fused preview image data.

[0116] The fourth image data is image data output by the first photosensor. The brightness of the fourth image data is greater than that of the fifth image data, so that the fourth image data can more clearly represent information about the stationary object. Optionally, the fourth image data and the first image data can be the same image data or different image data.

[0117] The fifth image data is image data output by the first photosensor. The brightness of the fourth image data is greater than that of the fifth image data. Therefore, in the overexposed areas corresponding to the fourth image data, the fifth image data can more clearly represent the information in these four overexposed areas. Since both the fifth image data and the fourth image data are output by the first photosensor, the difference in image depth between the two is minimal. Optionally, the fourth image data and the second image data can be the same image data or different image data.

[0118] The fused preview image data is the result of fusion of the fourth image data and the fifth image data. The dynamic range of the fused preview image data is greater than the dynamic range of the fourth image data, and the dynamic range of the fused preview image data matches the dynamic range of the fifth image data. Optionally, the fourth image data can be fused into the fifth image data, or the fifth image data can be fused into the fourth image data. The preview image is the image displayed on the screen during image acquisition.

[0119] In an exemplary embodiment, fusing the fourth image data and the fifth image data to obtain fused preview image data includes: determining, based on the brightness of the first image data and the second image data, fusion weights of the first image data and the second image data at different image positions; and fusing the first image data and the second image data at the different image positions based on the fusion weights to obtain the fused preview image data.

[0120] In an optional embodiment, generating a preview image based on the fused preview image data includes: drawing an image according to the fused preview image data, and using the drawn image as the preview image.

[0121] In an optional embodiment, generating a preview image based on the fused preview image data includes: correcting the fused preview image data to obtain corrected preview image data, and generating the preview image based on the corrected preview image data. Optionally, the fourth image data and the fifth image data may be fused via a sensor front-end output engine (SensorFrontEnd, SFE) to obtain fused preview image data; and generating the preview image based on the fused preview image data.

[0122] Optionally, the fused preview image data is sequentially input into an image front-end processing unit (Image Front End) and an image processing engine (Image Processing Engine) for processing to obtain corrected preview image data.

[0123] In this embodiment, when previewing the shooting scene, the third image data and the fifth image data are fused to obtain fused preview image data. The dynamic range of the fused preview image data is higher. A preview image is then generated based on the fused preview image data to obtain the preview image more quickly while taking into account the dynamic range of the preview image.

[0124] In an optional embodiment, generating a preview image based on the fused preview image data includes: converting the fused preview image data into the preview image according to image attribute data of the fourth image data.

[0125] Image attribute data is a characteristic of image data. This image attribute data facilitates image data analysis to accomplish corresponding tasks. Categories of image attribute data include image brightness and may also include, but are not limited to, image resolution, color space, data format, and other characteristics. Optionally, both the fourth and fifth image data are generated based on the first photosensor.

[0126] In an optional embodiment, converting the fused preview image data into a preview image according to image attribute data of the fourth image data includes: after correcting the fused preview image data to obtain corrected preview image data, converting the corrected preview image data into a preview image according to the image attribute data of the fourth image data.

[0127] In an optional embodiment, converting the fused preview image data into a preview image according to the image attribute data of the fourth image data includes generating a preview image frame according to the fused preview image data and using the image attribute data of the fourth image data as the image attribute data of the preview image. Thus, the preview image frame and image attribute data are complete.

[0128] In this embodiment, because the brightness of the fourth image data is greater than that of the fifth image data, the fourth image data can more clearly represent information about the stationary object. In this case, using the image attribute data of the fourth image data to convert the fused image data into a preview image can improve the clarity of the preview and create a sense of transparency.

[0129] In an exemplary embodiment, as shown in FIG5 , the first photoelectric sensor (main sensor) outputs the fourth image data (long frames) and the fifth image data (short frames) to the sensor front-end output engine (Sensor FrontEnd, SFE) to obtain fused preview image data; the fused preview image data is sequentially input into the image front-end processing unit (Image Front End) and the image processing engine (Image Processing Engine) for processing to obtain corrected preview image data; the corrected preview image data is then rendered in real time (RTbokeh) according to the image data of the fourth image data to obtain a preview image.

[0130] Correspondingly, the second photosensor (slave sensor) outputs the sixth image data, which is then sequentially input into the Image Front End and Image Processing Engine for processing and participates in the rendering of the Active Pixel Sensor (APS) to produce a preview image. Furthermore, the statistical information output by the Image Front End is used to synchronize the sixth image data with the fifth image data frame to ensure effective image processing.

[0131] In an exemplary embodiment, obtaining first image data and second image data output by a first photoelectric sensor includes: obtaining multiple image data pairs output by the first photoelectric sensor, each image data pair including first candidate image data and corresponding second candidate image data; the brightness of the first candidate image data is greater than the brightness of the second candidate image data; among the multiple image data pairs, determining a target image data pair according to image clarity, the target image data pair including the first image data and the corresponding second image data.

[0132] The image data pair is candidate image data output by the first photosensor and temporarily stored. Optionally, while the terminal is in preview mode, multiple image data pairs are acquired and temporarily stored in real time. Optionally, the image data can be temporarily stored in a cache. The image data pair includes first candidate image data and corresponding second candidate image data. Therefore, image clarity can be determined based on either the first candidate image data or the second candidate image data. Optionally, image clarity can be determined by taking the average clarity value of each image data pair.

[0133] The first candidate image data and the second candidate image data are output sequentially. The order in which the first candidate image data and the second candidate image data are output is not limited. Because the brightness of the first candidate image data is greater than that of the second candidate image data, the first candidate image data more clearly represents the information of the stationary object. In the overexposed areas corresponding to the first candidate image data, the second candidate image data more clearly represents the information in these four overexposed areas.

[0134] In one optional embodiment, a target image data pair is determined from multiple image data pairs based on image clarity, including: calculating image gradient values ​​for each of the multiple image data pairs in different directions; and determining the target image data pair based on the clarity represented by the image gradient values ​​in different directions; wherein, for the same scene, a higher image gradient value indicates a clearer image, and the image data pair with the highest clarity is the target image data pair. Exemplarily, the first candidate image data in the image data pair is processed using a Sobel operator to obtain image gradient values ​​in the horizontal and vertical directions, and then an average grayscale value of the image gradient values ​​is determined; a larger average grayscale value indicates a clearer image.

[0135] In an optional embodiment, among multiple image data pairs, a target image data pair is determined according to image clarity, including: respectively calculating the change values ​​of image gradient values ​​of the multiple image data pairs in different directions; and determining the target image data pair based on the clarity represented by the change values; wherein, the higher the change value of the image gradient value under the same scene, the clearer the image, and the image data pair with the highest clarity is the target image data pair.

[0136] In this embodiment, among multiple image data pairs, the target image pair is determined according to the clarity, so when linked with the moment of pressing the shutter, the first image data and the second image data can also have higher clarity, thereby making the clarity of the target image higher.

[0137] In an exemplary embodiment, before fusing the second image data with the first image data to obtain the fused image data, the method further includes: determining the image acquisition scene based on at least one of the first image data and the second image data; and when the image acquisition scene is a static scene, determining the first image data as the reference data.

[0138] The image acquisition scene is the scene in which the first image data and the second image data are captured. The static scene is a shooting scene in which no moving objects exist. Optionally, both the image acquisition scene and the static scene are represented by data.

[0139] In an optional embodiment, determining an image acquisition scene based on at least one of the first image data and the second image data includes: inputting one of the first image data and the second image data into a deep learning model for feature extraction; and determining the image acquisition scene based on the results of matching the extracted image features with preset templates; wherein each preset template is used to represent each image acquisition scene. Optionally, when the image features match a preset template used to represent a static scene, the image acquisition scene is a static scene; when the image features match a preset template used to represent a non-static scene, the image acquisition scene is a non-static scene; and the non-static scene can be further refined according to the different motion range conditions satisfied by the image features.

[0140] In an optional embodiment, determining the image capture scene based on at least one of the first image data and the second image data includes: performing optical flow calculation on the first image data and the second image data; and determining the image capture scene based on the result of the optical flow calculation. Optionally, if the result of the optical flow calculation represents a stationary scene, the image capture scene is a stationary scene; if the result of the optical flow calculation represents a non-stationary scene, the image capture scene is a non-stationary scene; the non-stationary scene may be further refined based on different motion range conditions satisfied by the result of the optical flow calculation.

[0141] In an optional embodiment, the second image data is fused with the first image data to obtain fused image data, including: when the image acquisition scene is a static scene, determining the first image data as reference data; taking the first image data as reference data, fusing the second image data with the first image data to obtain fused image data.

[0142] Correspondingly, the fused image data is blurred according to the depth information to obtain the target image, including: determining an image deformation matrix according to the first image data and the second image data; converting the depth information into converted depth information corresponding to the first image data according to the image deformation matrix; and blurring the fused image data according to the converted depth information to obtain the target image.

[0143] In this embodiment, when the image acquisition scene is a static scene, the first image data can be directly used as the reference data to ensure high image quality.

[0144] In an exemplary embodiment, the second image data is fused with the first image data to obtain fused image data, including: obtaining an image signal-to-noise ratio based on at least one of the first image data and the second image data when at least one of the first image data and the second image data satisfies a first motion range condition; and determining, in the first image data and the second image data, reference data for fusion according to a signal-to-noise ratio interval in which the image signal-to-noise ratio lies.

[0145] Satisfying the first motion range condition indicates that the degree of motion of the subject in the first and second image data falls within the first motion range. The first motion range condition indicates that the clarity of the first image data in the motion region cannot be determined based on the motion range. Optionally, the first motion range refers to an area of ​​the motion region being smaller than a preset area.

[0146] The image signal-to-noise ratio (SNR) is a measure of the image's signal-to-noise ratio (SNR). A higher SNR indicates a greater contribution of useful information relative to noise in the image. In this case, the SNR of the second image data can be used to determine the baseline data used for fusion between the first and second image data.

[0147] The signal-to-noise ratio interval is the range within which the image signal-to-noise ratio falls. The signal-to-noise ratio interval can be divided according to at least one critical value. Optionally, if only one critical value exists and the image signal-to-noise ratio is greater than this critical value, the first image data is determined as the baseline data for fusion; if only one critical value exists and the image signal-to-noise ratio is less than this critical value, the second image data is determined as the baseline data for fusion.

[0148] In an optional embodiment, the situation where at least one of the first image data and the second image data satisfies the first motion range condition may refer to: inputting one of the first image data and the second image data into a deep learning model for feature extraction; and determining that the image features satisfy the first motion range condition based on the result of matching the extracted image features with a preset template.

[0149] In an optional embodiment, the situation where at least one of the first image data and the second image data satisfies the first motion range condition may refer to: performing optical flow calculation on the first image data and the second image data; and determining that the result of the optical flow calculation satisfies the first motion range condition.

[0150] In an optional embodiment, obtaining a signal-to-noise ratio based on at least one image data includes: obtaining a signal-to-noise ratio of first image data; or obtaining a signal-to-noise ratio of second image data; or obtaining a signal-to-noise ratio of the first image data and a signal-to-noise ratio of the second image data, and calculating a mean signal-to-noise ratio between the signal-to-noise ratio of the first image data and the signal-to-noise ratio of the second image data to obtain the image signal-to-noise ratio.

[0151] In an optional embodiment, based on the signal-to-noise ratio interval of the image signal-to-noise ratio, baseline data for fusion is determined in the first image data and the second image data, including: if the image signal-to-noise ratio is in the first signal-to-noise ratio interval, determining the first image data as the baseline data for fusion; if the image signal-to-noise ratio is in the second signal-to-noise ratio interval, determining the second image data as the baseline data for fusion.

[0152] In an optional embodiment, based on the signal-to-noise ratio interval of the image signal-to-noise ratio, baseline data for fusion is determined in the first image data and the second image data, including: when the image signal-to-noise ratio is less than an image signal-to-noise ratio threshold, determining the first image data as the baseline data for fusion; when the image signal-to-noise ratio is greater than the image signal-to-noise ratio threshold, determining the second image data as the baseline data for fusion.

[0153] Optionally, in the case where the first image data is the reference data for fusion, the operation of obtaining the target image is described. Correspondingly, based on the depth information, the fused image data is blurred to obtain the target image, including: determining an image deformation matrix based on the first image data and the second image data; converting the depth information into converted depth information corresponding to the first image data according to the image deformation matrix; and blurring the fused image data based on the converted depth information to obtain the target image. Specifically, blurring the fused image data based on the converted depth information includes: blurring the fused image data based on the converted depth information and the regional deformation matrix to obtain the target image. Thus, combining the converted depth information and the regional deformation matrix achieves a better blurring effect in the moving area.

[0154] In this embodiment, when the baseline data cannot be determined by the motion range condition, the baseline data is adaptively selected from the first image data and the second image data based on the signal-to-noise ratio interval of the image signal-to-noise ratio, so that the clarity of the target image is higher and the image quality effect is achieved.

[0155] In an exemplary embodiment, before fusing the second image data with the first image data to obtain the fused image data, the method includes: if at least one of the first image data and the second image data satisfies a second motion range condition, determining the second image data as the reference data for fusion.

[0156] When the second motion range condition is satisfied, the degree of motion of the subject captured in the first and second image data is characterized as being within the second motion range. The second motion range condition is used to indicate that the first image data is unclear in the motion region. Optionally, the first motion range refers to an area of ​​the motion region being greater than a preset area.

[0157] In an optional embodiment, the situation where at least one of the first image data and the second image data satisfies the second motion range condition may refer to: inputting one of the first image data and the second image data into a deep learning model for feature extraction; and determining that the image features satisfy the second motion range condition based on the result of matching the extracted image features with a preset template.

[0158] In an optional embodiment, the situation where at least one of the first image data and the second image data satisfies the second motion range condition may refer to: performing optical flow calculation on the first image data and the second image data; and determining that the result of the optical flow calculation satisfies the second motion range condition.

[0159] When the second image data is determined to be the reference data for fusion, the second image data is fused with the first image data to obtain fused image data, including: taking the second image data as the reference data, fusing the second image data with the first image data to obtain fused image data.

[0160] In this embodiment, when the second motion range condition is met, since the brightness of the second image data is lower than that of the first image data, the second image data is used as the reference data, and the second image data is used to represent the large-scale motion of the moving object to achieve the goal of capturing the image.

[0161] In an exemplary embodiment, before fusing the second image data with the first image data to obtain the fused image data, the method further includes: performing artifact removal and foreground and background separation processing on the first image data and the second image data, respectively, to obtain first artifact-free portrait data corresponding to the first image data and second artifact-free portrait data corresponding to the second image data; and determining reference data for fusion based on at least one of the first artifact-free portrait data and the second artifact-free portrait data.

[0162] The first artifact-free portrait data is obtained by performing artifact removal and foreground / background separation on the first image data. The second artifact-free portrait data is obtained by performing artifact removal and foreground / background separation on the second image data. Both the first and second artifact-free portrait data are not background data, but rather portrait data after artifacts in the foreground have been removed.

[0163] Optionally, the methods for removing artifacts include but are not limited to median filtering, wavelet denoising, Wiener filtering, etc.; the foreground and background separation processing includes but is not limited to segmenting the foreground and background based on texture features, and segmenting the foreground and background based on facial features.

[0164] In an optional embodiment, the first image data and the second image data are subjected to artifact removal and foreground and background separation processing respectively, including: performing artifact removal on the first image data and the second image data respectively to obtain the first image data after artifact removal and the second image data after artifact removal; performing foreground and background separation processing on the first image data after artifact removal and the second image data after artifact removal.

[0165] In an optional embodiment, based on at least one image data of the first de-artifacted portrait data and the second de-artifacted portrait data, the reference data for fusion is determined, including: when the image acquisition scene is a static scene, determining the first de-artifacted portrait data as the reference data; using the first de-artifacted portrait data as the reference data, fusing the second de-artifacted portrait data with the first de-artifacted portrait data to obtain fused image data.

[0166] In an optional embodiment, based on at least one image data in the first de-artifacted portrait data and the second de-artifacted portrait data, determining the baseline data for fusion includes: when at least one image data in the first de-artifacted portrait data and the second de-artifacted portrait data satisfies a first motion range condition, obtaining the image signal-to-noise ratio based on the at least one image data; and determining the baseline data for fusion in the first de-artifacted portrait data and the second de-artifacted portrait data according to the signal-to-noise ratio interval in which the image signal-to-noise ratio is located.

[0167] In an optional embodiment, based on at least one image data among the first de-artifacted portrait data and the second de-artifacted portrait data, the baseline data for fusion is determined, including: when at least one image data among the first de-artifacted portrait data and the second de-artifacted portrait data satisfies the second motion range condition, the second de-artifacted portrait data is determined as the baseline data for fusion.

[0168] In this embodiment, artifact removal and foreground and background separation processing are first performed to improve the clarity of the portrait data, and then the reference data for fusion is determined to further improve the clarity of the fused image data.

[0169] In an exemplary embodiment, a frame selection strategy is implemented based on the portrait 2exp method. Since two data streams (long frames and short frames) are output simultaneously, we need to consider many points: 1. The dimension of clarity, which set of data is clear; 2. The dimension of snapshot, whether there is motion in the current scene, and whether to use the short frame as the base; 3. The timing of snapshot, we need to choose a moment close to the shutter to ensure the uniformity of the snapshot timing.

[0170] In an exemplary embodiment, as shown in FIG8 , it is determined based on the input whether there is a design for ghost removal and foreground and background separation (Ai depth), and then whether it is a static scene is determined. If it is a static scene, the first image data is selected as the base data. If the first motion range condition is met, it is a small-range motion, and whether a long second image data fusion is to be performed is determined based on the signal-to-noise ratio. The long second image data fusion uses the first image data as the base data, and the second image data is fused onto the first image data. If the second motion range condition is met, it is a large-range motion, and the second image data needs to be used as the base data for fusion processing.

[0171] Optionally, the exposure time of the first image data is twice the exposure time of the second image data, as shown in FIG9 .

[0172] Optionally, in a static scene, when the first dynamic range condition or the first dynamic range condition is met, the benchmark data is determined by taking frames after clicking the shutter (Post Shutter Lag, PSL) and / or taking frames forward with zero delay after clicking the shutter (Zero Shutter Lag, ZSL).

[0173] Optionally, as shown in FIG10 , when the shutter is clicked to take frames forward with zero delay, from the scene where the shutter is still and the first dynamic range condition or the second dynamic range condition is satisfied, the image data pair of the time point when the shutter is pressed and the image data pair forward of the time point when the shutter is pressed are taken to determine the target image data pair.

[0174] Optionally, as shown in Figure 11, in the cases of clicking the shutter to take frames forward with zero delay and clicking the shutter to take frames forward with zero delay, if the shutter is in a static scene, the time point when the shutter is pressed and multiple image data pairs after the time point when the shutter is pressed are taken, and the method of taking frames after clicking the shutter is adopted to determine the PSL image data pairs other than the zero-delay forward frame when the shutter is clicked, and the target image data pairs are determined by the multiple image data pairs after the time point when the shutter is pressed and the PSL image data pairs other than the zero-delay forward frame when the shutter is clicked.

[0175] Optionally, as shown in Figure 12, in the case of clicking the shutter to take frames with zero delay forward and clicking the shutter to take frames with zero delay forward, if the first dynamic range condition or the second dynamic range condition is met, multiple image data pairs after the time point when the shutter is pressed are taken, and the method of taking frames after the shutter is clicked is adopted to determine the PSL image data pairs other than the zero-delay forward frame when the shutter is clicked, and the target image data pairs are determined by the multiple image data pairs after the time point when the shutter is pressed and the PSL image data pairs other than the zero-delay forward frame when the shutter is clicked.

[0176] Based on this, we implement a new input form based on 2exp, and combine it with the long and short frame strategies to implement a complete set of frame selection strategies to fully ensure the effect.

[0177] In an exemplary embodiment, as shown in FIG13 , the method further includes:

[0178] Operation 1302 : Determine portrait data and background image data; the portrait data and background image data are image data obtained by performing foreground and background separation processing on the first image data, the second image data, or the fused image data.

[0179] The background image data is image data other than the portrait data. Optionally, when the portrait data is in the foreground, the background image data is background data; alternatively, when the portrait data is in the background, the background image data is foreground data.

[0180] Optionally, the portrait data and the background image data are image data obtained by performing foreground and background separation processing on one of the first image data, the second image data, and the fused image data. Optionally, the portrait data and the background image data are image data obtained by performing foreground and background separation processing on the first image data and the second image data, respectively.

[0181] Operation 1304 : Determine the background light scene based on the brightness ratio between the portrait data and the background image data.

[0182] The brightness ratio is used to characterize the relationship between the portrait data and the background space. Optionally, the brightness ratio may be the ratio between the brightness of the portrait data and the brightness of the background space, the brightness ratio may be the variance between the brightness of the portrait data and the brightness of the background space, or the brightness ratio may be the variance between the brightness of the portrait data and the brightness of the background space.

[0183] The background light scene is used to characterize the ambient light type of the shooting scene. For example, the background light scene includes backlight scene, soft light scene and front light scene.

[0184] In an optional embodiment, determining the background light scene according to the brightness ratio between the portrait data and the background image data includes: mapping the brightness ratio between the portrait data and the background image data according to data in a preset mapping table to obtain the background light scene.

[0185] Operation 1306 : Adjust the brightness of the background image data according to the background light scene.

[0186] In an optional embodiment, the brightness of the background image data is adjusted according to the background light scene, including: mapping according to conditions satisfied by the background light scene to obtain a brightness adjustment result; and adjusting the brightness of the background image data according to the brightness adjustment result.

[0187] Optionally, after brightness adjustment is performed on the background image data, the image data to which the brightness-adjusted background image data belongs may continue to be processed. For example, after brightness adjustment is performed on the background image data of the fused image data, the fused image data carries the brightness-adjusted background image data, and the fused image data may be blurred based on the depth information to obtain the target image. In this case, because the fused image data in operation 208 carries the brightness-adjusted background image data, a brightness correspondence can be established between the portrait data and the background image data in the target image, thereby making the background spatial relationship of the portrait in the target image more consistent with the spatial sense of the real scene.

[0188] In this embodiment, the background lighting scene is determined based on the brightness ratio between the portrait data and the background image data. Based on the background lighting scene, the background image data is then brightness-adjusted, establishing a spatial relationship between the portrait data and the background image data. This spatial relationship makes the portrait data more prominent and the background image data more extensive.

[0189] In an optional embodiment, determining the background light scene according to the brightness ratio between the portrait data and the background image data includes: determining the background light scene according to the brightness ratio interval in which the brightness ratio between the portrait data and the background image data lies.

[0190] Correspondingly, the brightness of the background image data is adjusted according to the background light scene, including: adjusting the brightness of the background image data according to the brightness adjustment direction corresponding to the background light scene.

[0191] In an optional embodiment, the background light scene is determined based on the brightness ratio interval of the brightness ratio between the portrait data and the background image data, including: if the brightness ratio between the portrait data and the background image data is in a first brightness ratio interval, the background light scene is a backlight scene; if the brightness ratio between the portrait data and the background image data is in a second brightness ratio interval, the background light scene is a soft light scene; if the brightness ratio between the portrait data and the background image data is in a third brightness ratio interval, the background light scene is a front light scene; wherein the value of the first brightness ratio interval is smaller than the value of the second brightness ratio interval, and the value of the second brightness ratio interval is smaller than the value of the second brightness ratio interval.

[0192] Exemplarily, the first brightness ratio interval is greater than 0 and less than or equal to 0.3; the second brightness ratio interval is greater than or equal to 0.5 and less than or equal to 0.3; and the third brightness ratio interval is greater than or equal to 0.8 and less than or equal to 1.3.

[0193] In an optional embodiment, the brightness of the background image data is adjusted according to the brightness adjustment direction corresponding to the background light scene, including: when the background light scene is a backlight scene, the brightness of the background image data is lowered; when the background light scene is a soft light scene, the brightness of the background image data is not adjusted; when the background light scene is a front light scene, the brightness of the background image data is increased.

[0194] The principle for direct light is to raise the EV0 base tone to ensure smooth, crisp light and shadows, creating a rich, transparent image without a dead black or dull atmosphere. For backlighting, the principle is to lower the background tone from face EV0 to fusion. The background should be overexposed with details and gradients, creating a contrast between the backlight and the backlight, and creating a harmonious sense of space between the subject and the background. Current issues include backlit backgrounds or mid-high light backgrounds that tend to be foggy and lack gradients. For soft light or overcast conditions, the principle is to maintain the base tone at EV0 or slightly lower it to ensure a soft, delicate image with natural transitions and a textured feel.

[0195] In this embodiment, background light scenes are determined by brightness ratio intervals, making them distinct from one another. This allows accurate determination of the brightness adjustment direction for each background light scene, allowing for detailed brightness adjustment of the background image data. This allows for a more detailed spatial relationship between the portrait data and the background image data, established through brightness.

[0196] In an exemplary embodiment, the method further includes: when determining the background image data, determining whether the background image data meets the brightness overexposure condition; the background image data is image data obtained by performing foreground and background separation processing on the first image data, the second image data, or the fused image data; if so, lowering the brightness of the background image data.

[0197] The brightness overexposure condition is used to determine whether the background image data is overexposed. Optionally, the brightness overexposure condition is set for each image region of the background image data, and each image region can be a plurality of pixels under a certain specification.

[0198] In an optional embodiment, determining whether the background image data satisfies the brightness overexposure condition includes determining whether the background image data is used to indicate pixels having brightness values ​​greater than an overexposure threshold; if so, the brightness overexposure condition is satisfied.

[0199] In an optional embodiment, determining whether the background image data satisfies the brightness overexposure condition includes: determining whether the background image data is used to indicate pixels whose brightness values ​​are greater than an overexposure threshold; if so, determining the area of ​​the overexposed region for the pixels whose brightness values ​​are greater than the overexposure threshold; if the area of ​​the overexposed region is greater than the overexposure region area threshold, the brightness overexposure condition is satisfied.

[0200] In an optional embodiment, the method includes: determining portrait data and background image data; the portrait data and background image data are image data obtained by performing foreground and background separation processing on the first image data, the second image data or the fused image data; determining the background light scene according to the brightness ratio between the portrait data and the background image data; adjusting the brightness of the background image data according to the background light scene to obtain first adjusted background image data; judging whether the first adjusted background image data meets the brightness overexposure condition; the first adjusted background image data is image data obtained by performing foreground and background separation processing on the first image data, the second image data or the fused image data; if satisfied, lowering the brightness of the first adjusted background image data to obtain second adjusted background image data.

[0201] Optionally, if the brightness of the background image data is lowered based on whether the brightness overexposure condition is satisfied, the image data to which the brightness-adjusted background image data belongs may continue to be processed. For example, after the brightness of the background image data of the fused image data is lowered based on whether the brightness overexposure condition is satisfied, this fused image data carries the brightness-adjusted background image data, and this fused image data can be blurred based on the depth information to obtain the target image. In this case, because the fused image data in operation 208 carries the brightness-adjusted background image data, a brightness correspondence can be established between the portrait data and the background image data in the target image, so that the spatial relationship of the portrait and the background in the target image more closely matches the spatial sense of the real scene.

[0202] In this embodiment, it is determined whether the background image data meets the brightness overexposure condition. If so, it represents an indoor environment. For indoor environments, in addition to using the brightness of the background image data under the background light scene condition, the brightness of the background image data can also be lowered separately to make the background details clearer, thereby avoiding blurry or hazy background image data.

[0203] It should be understood that, although each operation in the flow chart that each embodiment as above relates to is shown in sequence according to the indication of arrow, these operations are not necessarily performed in sequence according to the order indicated by arrow.Unless clear instructions are arranged in this article, the execution of these operations does not have strict order restriction, and these operations can be performed in other orders.And, at least a portion of operations in the flow chart that each embodiment as above relates to can comprise multiple operations or multiple stages, and these operations or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these operations or stages is not necessarily carried out in sequence, but can be performed in turn or alternately with at least a portion of the operations or stages in other operations or other operations.

[0204] Based on the same inventive concept, embodiments of the present application also provide an image processing device for implementing the aforementioned image processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following image processing device embodiments can be found in the above-described limitations on the image processing method and will not be further elaborated here.

[0205] In one embodiment, as shown in FIG14 , an image processing apparatus is provided, including:

[0206] an acquisition module 1402 configured to acquire first image data and second image data output by a first photosensor, and third image data output by a second photosensor; wherein the brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data;

[0207] a fusion module 1404, configured to fuse the second image data with the first image data to obtain fused image data;

[0208] a depth calculation module 1406, configured to determine depth information based on the second image data and the third image data;

[0209] The blur module 1408 is configured to perform blur processing on the fused image data according to the depth information to obtain a target image.

[0210] In one embodiment, the virtualization module is configured to:

[0211] determining an image deformation matrix based on the first image data and the second image data;

[0212] converting the depth information into converted depth information corresponding to the first image data according to the image deformation matrix;

[0213] The fused image data is blurred according to the converted depth information to obtain a target image.

[0214] In one embodiment, the fusion module 1404 is configured to:

[0215] In the motion region, determining the second image data and a regional deformation matrix of the second image data;

[0216] In the motion region, correcting the second image data using the regional deformation matrix to obtain corrected motion region data;

[0217] In the motion region, the first image data and the corrected motion region data are fused to obtain fused image data.

[0218] The virtualization module is used to:

[0219] The fused image data is blurred according to the depth information and the regional deformation matrix to obtain a target image.

[0220] In one embodiment, the acquisition module 1402 is configured to:

[0221] In the case of preview, acquiring fourth image data and fifth image data output by the first photosensor, wherein the brightness of the fourth image data is greater than that of the fifth image data;

[0222] fusing the fourth image data and the fifth image data to obtain fused preview image data;

[0223] A preview image is generated based on the fused preview image data.

[0224] In one embodiment, the acquisition module 1402 is configured to:

[0225] The fused preview image data is converted into a preview image according to the image attribute data of the fourth image data.

[0226] In one embodiment, the second image data and the third image data are acquired synchronously.

[0227] In one embodiment, the acquisition module 1402 is configured to:

[0228] Acquire multiple image data pairs output by the first photosensor, each image data pair including first candidate image data and corresponding second candidate image data; the brightness of the first candidate image data is greater than the brightness of the second candidate image data;

[0229] Among the plurality of image data pairs, a target image data pair is determined according to image definition, and the target image data pair includes the first image data and the corresponding second image data.

[0230] In one embodiment, the fusion module 1404 is configured to:

[0231] The first image data is used as reference data, and the second image data is fused with the first image data to obtain fused image data.

[0232] In one embodiment, the fusion module 1404 is configured to:

[0233] before fusing the second image data with the first image data to obtain fused image data, determining an image acquisition scene based on at least one of the first image data and the second image data;

[0234] In a case where the image acquisition scene is a static scene, the first image data is determined to be the reference data.

[0235] In one embodiment, the fusion module 1404 is configured to:

[0236] before fusing the second image data with the first image data to obtain fused image data, obtaining an image signal-to-noise ratio based on at least one of the first image data and the second image data if the at least one of the first image data and the second image data satisfies a first motion range condition;

[0237] According to the signal-to-noise ratio interval of the image signal-to-noise ratio, reference data for fusion is determined in the first image data and the second image data.

[0238] In one embodiment, the fusion module 1404 is configured to:

[0239] Before fusing the second image data with the first image data to obtain fused image data, if at least one of the first image data and the second image data satisfies a second motion range condition, the second image data is determined as reference data for fusion.

[0240] In one embodiment, the fusion module 1404 is configured to:

[0241] fusing the second image data with the first image data to obtain fused image data, and then performing artifact removal and foreground and background separation processing on the first image data and the second image data, respectively, to obtain first artifact-free portrait data corresponding to the first image data and second artifact-free portrait data corresponding to the second image data;

[0242] Based on at least one of the first artifact-free portrait data and the second artifact-free portrait data, reference data for fusion is determined.

[0243] In one embodiment, the fusion module 1404 is configured to:

[0244] Determining portrait data and background image data; the portrait data and the background image data are image data obtained by performing foreground and background separation processing on the first image data, the second image data, or the fused image data;

[0245] determining a background light scene according to a brightness ratio between the portrait data and the background image data;

[0246] The background image data is brightness-adjusted according to the background light scene.

[0247] In one embodiment, the fusion module 1404 is configured to:

[0248] determining a background light scene according to a brightness ratio interval of a brightness ratio between the portrait data and the background image data;

[0249] The brightness of the background image data is adjusted according to the brightness adjustment direction corresponding to the background light scene.

[0250] In one embodiment, the fusion module 1404 is configured to:

[0251] In the case of determining background image data, determining whether the background image data meets a brightness overexposure condition; the background image data is image data obtained by performing foreground and background separation processing on the first image data, the second image data, or the fused image data;

[0252] If the conditions are met, the brightness of the background image data is lowered.

[0253] Each module in the above-mentioned image processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0254] In one embodiment, a computer device is provided, which may be a terminal. Its internal structure diagram may be as shown in FIG15 . The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and an external device. The communication interface of the computer device is configured to communicate with an external terminal via wired or wireless communication, where the wireless communication may be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements an image processing method. The display unit of the computer device is configured to produce a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0255] Those skilled in the art will understand that the structure shown in FIG15 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0256] The present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the operations of the image processing method.

[0257] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0258] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0259] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0260] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0261] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image processing method, characterized in that, Including: Obtaining first image data and second image data output by a first optoelectronic sensor, and third image data output by a second optoelectronic sensor; The brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data; Fusing the second image data and the first image data to obtain fused image data; Determining depth information according to the second image data and the third image data; And Performing a blurring process on the fused image data according to the depth information to obtain a target image.

2. The method according to claim 1, wherein The performing a blurring process on the fused image data according to the depth information to obtain a target image includes: Determining an image deformation matrix according to the first image data and the second image data; Converting the depth information into converted depth information corresponding to the first image data according to the image deformation matrix; and Performing a blurring process on the fused image data according to the converted depth information to obtain a target image.

3. The method according to claim 1 or 2, characterized in that, The fusing the second image data and the first image data to obtain fused image data includes: In a motion area, determining a regional deformation matrix of the second image data and the second image data; In the motion area, correcting the second image data through the regional deformation matrix to obtain corrected motion area data; In the motion area, fusing the first image data and the corrected motion area data to obtain fused image data; and The performing a blurring process on the fused image data according to the depth information to obtain a target image includes: Performing a blurring process on the fused image data according to the depth information and the regional deformation matrix to obtain a target image.

4. The method according to claim 1, wherein The method further includes: In a preview scenario, obtaining fourth image data and fifth image data output by a first optoelectronic sensor, and the brightness of the fourth image data is greater than that of the fifth image data; Fusing the fourth image data and the fifth image data to obtain fused preview image data; and Generating a preview image based on the fused preview image data.

5. The method according to claim 4, wherein The generating a preview image based on the fused preview image data includes: Converting the fused preview image data into a preview image according to the image attribute data of the fourth image data.

6. The method according to claim 1, characterized in that The second image data and the third image data are collected synchronously.

7. The method according to claim 1, characterized in that The obtaining first image data and second image data output by a first optoelectronic sensor includes: Obtaining a plurality of image data pairs output by a first optoelectronic sensor, each image data pair including first candidate image data and corresponding second candidate image data; the brightness of the first candidate image data is greater than that of the second candidate image data; and Among the plurality of image data pairs, determining a target image data pair according to image sharpness, and the target image data pair includes the first image data and the corresponding second image data.

8. The method according to claim 7, wherein The determining a target image data pair according to image sharpness among the plurality of image data pairs includes: Calculate the image gradient values of the multiple pairs of image data in different directions respectively; Determine the target pair of image data according to the sharpness characterized by the image gradient values in different directions.

9. The method according to claim 1, characterized in that, The fusing the second image data with the first image data to obtain fused image data includes: Taking the first image data as reference data, fusing the second image data with the first image data to obtain fused image data.

10. The method according to claim 9, characterized in that, Before the fusing the second image data with the first image data to obtain fused image data, the method further includes: Determine an image acquisition scenario based on at least one of the first image data and the second image data; and When the image acquisition scenario belongs to a static scenario, determine the first image data as the reference data.

11. The method according to claim 1, wherein Before the fusing the second image data with the first image data to obtain fused image data, the method further includes: When at least one of the first image data and the second image data satisfies a first motion range condition, obtain an image signal-to-noise ratio based on the at least one of the image data; and Determine the reference data for fusion from the first image data and the second image data according to the signal-to-noise ratio interval where the image signal-to-noise ratio is located.

12. The method according to claim 11, wherein, The determining the reference data for fusion from the first image data and the second image data according to the signal-to-noise ratio interval where the image signal-to-noise ratio is located includes: When the image signal-to-noise ratio is less than the image signal-to-noise ratio threshold, determine the first image data as the reference data for fusion; When the image signal-to-noise ratio is greater than the image signal-to-noise ratio threshold, determine the second image data as the reference data for fusion.

13. The method according to claim 1, characterized in that, Before the fusing the second image data with the first image data to obtain fused image data, the method includes: When at least one of the first image data and the second image data satisfies a second motion range condition, determine the second image data as the reference data for fusion.

14. The method according to claim 1, wherein Before the fusing the second image data with the first image data to obtain fused image data, the method further includes: Perform artifact elimination and foreground / background separation processing on the first image data and the second image data respectively to obtain first artifact-free portrait data corresponding to the first image data and second artifact-free portrait data corresponding to the second image data; and Determine the reference data for fusion based on at least one of the first artifact-free portrait data and the second artifact-free portrait data.

15. The method according to claim 1, characterized in that, The method further includes: Determine portrait data and background image data; the portrait data and the background image data are image data obtained by performing foreground / background separation processing on the first image data, the second image data, or the fused image data; Determine a background light scenario according to the brightness ratio between the portrait data and the background image data; and Adjust the brightness of the background image data according to the background light scenario.

16. The method according to claim 15, characterized in that, Determining a background light scene according to a brightness ratio between the portrait data and the background image data includes: Determining a background light scene according to a brightness ratio interval in which the brightness ratio between the portrait data and the background image data lies; and Adjusting the brightness of the background image data according to the background light scene includes: Adjusting the brightness of the background image data according to a brightness adjustment direction corresponding to the background light scene.

17. The method according to claim 1, wherein The method further includes: When the background image data is determined, determining whether the background image data satisfies a brightness overexposure condition; the background image data is image data obtained by performing foreground / background separation processing on the first image data, the second image data, or the fused image data; and If so, reducing the brightness of the background image data.

18. An image processing apparatus, characterized in that, including: An acquisition module configured to acquire first image data and second image data output by a first photoelectric sensor, and third image data output by a second photoelectric sensor; The brightness of the first image data is greater than that of the second image data, and the brightness of the first image data is greater than that of the third image data; A fusion module configured to fuse the second image data and the first image data to obtain fused image data; A depth calculation module configured to determine depth information according to the second image data and the third image data; A blurring module configured to perform blurring processing on the fused image data according to the depth information to obtain a target image.

19. An electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that, When the computer program is executed by the processor, the processor is caused to perform the operations of the image processing method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the operations of the method according to any one of claims 1 to 17 are implemented.

Images