Exposure parameter adjustment method, device, and storage medium

Abstract

The application provides an exposure parameter adjustment method and device and a storage medium, and relates to the technical field of image processing. In the application, when a preview image based on a preview stream is displayed, if a face image is detected from the preview image, a real face part can be segmented from the face image first, an exposure parameter is calculated based on the average brightness of the real face part, and then the exposure parameter of a camera is adjusted. By segmenting the real face part from the face image, the influence of face coverings such as masks, glasses, beards and hair and background images on the overall brightness calculation can be reduced, the exposure parameter calculated based on the average brightness of the real face part is more accurate, the brightness of the finally captured image is more in line with the user demand, the problem of constantly flickering of the captured image is avoided, the image quality is improved, and the user's shooting experience is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an exposure parameter adjustment method, device and storage medium. Background Technology

[0002] With the development of smart terminals, the camera function has become a common feature of electronic devices.

[0003] Currently, most electronic devices are equipped with automatic exposure parameter adjustment functions. After the camera is turned on, if there is a human figure in the preview video stream captured by the camera, the electronic device can calculate the overall brightness of the face area, and then adjust the camera's exposure parameters based on this overall brightness to capture an image that meets the brightness requirements.

[0004] However, when the face of a person in the preview video stream is obscured by a mask, glasses, beard, or hair, if the color difference between the obscuring object and the facial skin is large, it may affect the overall brightness of the face area. This will result in lower accuracy of the exposure parameters calculated based on the overall brightness of the face area, leading to overexposure or underexposure and affecting image quality. Summary of the Invention

[0005] This application provides an exposure parameter adjustment method, device, and storage medium, which solves the technical problem that exposure parameters are not accurate enough when the face is covered by occlusions such as masks, glasses, beards, or hair.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, embodiments of this application provide a method for adjusting exposure parameters. This method may include:

[0008] The preview interface displays a first image based on a first data stream; a first face image is cropped from a second image based on the first data stream; a first mask image corresponding to the first face image is acquired, the first mask image including at least data of the face area not covered by occlusions; based on the data of the face area not covered by occlusions, the first face area is determined in a third image; a first exposure parameter is obtained based on the brightness of the first face area; and the camera is controlled to acquire a second data stream based on the first exposure parameter. The first image is an RGB image converted from a RAW image acquired at a first moment, the second image is a YUV image converted from a RAW image acquired at a second moment, and the third image is a YUV image converted from a RAW image acquired at a first moment. The first data stream is the data stream acquired by the camera before adjusting to the first exposure parameter, and the second data stream is the data stream acquired by the camera after adjusting to the first exposure parameter.

[0009] In the above solution, when displaying a preview image based on a preview stream, if a face image is detected in the preview image, the actual face portion can be segmented from the face image first. Then, exposure parameters are calculated based on the brightness of the actual face portion, and the camera's exposure parameters are adjusted accordingly. By segmenting the actual face portion from the face image, the influence of facial obstructions such as masks, glasses, beards, or hair, as well as the background image, on the overall brightness calculation can be reduced. This makes the exposure parameters calculated based on the average brightness of the actual face portion more accurate, resulting in a final image brightness that better meets the user's needs. It also avoids the problem of continuous flickering in the captured image, improving the quality of the final image and enhancing the user's photography experience.

[0010] In one possible implementation, RGB and YUV are two standard color encoding methods. In RGB color encoding, each pixel has three colors: red, green, and blue. In YUV color encoding, Y represents luminance, and U and V represent chrominance; luminance and chrominance specify the color of a pixel. For displays, images are typically shown using the RGB model, but the YUV model is usually used when processing image data because it saves more storage space.

[0011] In one possible implementation, cropping a first face image from a second image based on a first data stream includes: obtaining face bounding box information using a face detection algorithm, the face bounding box information including the number of faces in the second image and the coordinates of each face bounding box, with each face bounding box corresponding to one face; determining a second face region based on the face bounding box information, the second face region being the face bounding box region in the second image whose area is greater than a preset size and whose face bounding box area is the largest; and cropping a first face image from the second image if the size of the second face region is greater than or equal to the preset size, the first face image being the image of the second face region. The preset size can be a preset area, or it can be a preset height and a preset width. It is understood that the size of each frame of the YUV image is usually fixed. If the face region with the largest face frame in the i-th frame of the YUV image is smaller than the preset size, it means that the face occupies a small proportion of the YUV image in this frame and has little impact on the brightness of the entire image. It is not the focus of the shot, so the brightness of this face region should not be used as the basis for adjusting the exposure parameters. If the face region with the largest face frame in a YUV image is greater than or equal to the preset size, it means that the face occupies a large proportion of the YUV image in this frame and may be the focus of the shot.

[0012] In one possible implementation, after cropping the first face image from the second image, the method further includes: converting the first face image to RGB format; and adjusting the converted RGB format first face image to a preset size. It is understandable that since AI models typically acquire MSAK images based on RGB format images, the face region image needs to be converted from the YUV domain to the RGB domain before being input into the AI ​​model. Furthermore, taking a preset size of 64 pixels * 64 pixels as an example, since the size of the face region with the largest face bounding box may far exceed 64 pixels * 64 pixels, directly inputting it into the AI ​​model could lead to a large computational load. To reduce the computational load on the AI ​​model, the MSAK module can first adjust the face region image to 64 pixels * 64 pixels before inputting it into the AI ​​model.

[0013] In one possible implementation, obtaining the first mask image corresponding to the first face image includes: inputting the first face image into an AI model to obtain the first mask image. The mask image is a matrix with the same dimensions as the original image (i.e., the i-th frame YUV image). The elements in the mask image determine whether to retain, modify, or occlude image pixels at corresponding positions in the i-th frame YUV image. The mask image can be used to select real face regions and filter non-face regions (such as background images and areas covered by occluded objects). As an example, in the mask image of a real face, a pixel value of 0 indicates that the pixel is located in the real face region, and a pixel value of 1 indicates that the pixel is located in the non-face region. As another example, a pixel value of 1 indicates that the pixel is located in the real face region, and a pixel value of 0 indicates that the pixel is located in the non-face region. It's understandable that if the color of the background image or the area covered by occlusions differs significantly from the actual color of the face, the calculated overall brightness of the face area may be inaccurate, leading to inaccurate exposure parameters based on that brightness. Therefore, by using an AI model, an MSAK image carrying real face data can be obtained, allowing non-face areas to be ignored or removed.

[0014] In one possible implementation, a first face image is input into an AI model to obtain a first mask image. This includes: arranging the RGB domain data of the first face image and the depth information data of the first depth map to obtain a first face depth map, which is a depth map acquired by a depth estimation device at a second time step; and inputting the first face depth map into the AI ​​model to obtain the first mask image. It is understandable that in some scenarios, when the preview stream includes a distant image, and the distant image is a large-sized face image (such as a face image on a lightbox), the electronic device might determine that the face image on the lightbox is the face image with the largest face bounding box area and obtain exposure parameters based on the face image on the lightbox. Although the face image on the lightbox is large, users are usually more concerned with close-up images than distant images; therefore, the exposure parameters obtained based on the face image on the lightbox may not be accurate. By arranging the RGB domain data of the first face image and the depth information data of the first depth map, the RGB domain data and depth information data can be fused in the face depth map.

[0015] In one possible implementation, the AI ​​model stores multiple frames of historical portraits, which are face depth maps obtained based on RAW images captured before the second time step. The first face depth map is input into the AI ​​model to obtain a first mask image, including: inputting the first face depth map into the AI ​​model; and calling the AI ​​model to obtain the first mask image corresponding to the third image based on the first face depth map and the multiple historical portraits. It can be understood that by inputting the face depth map, which integrates RGB domain data and depth information data, into the AI ​​model, a judgment of the overall depth of the portrait can be added to the original algorithm, making the final mask image output by the AI ​​model more accurate.

[0016] In one possible implementation, before displaying the first image based on the first data stream in the preview interface, the method further includes: acquiring a first depth map using a depth estimation device. The depth map carries depth information representing the distance from the object being photographed to the camera. Typically, the depth map consists of an array of m*n pixels. The larger the depth value of a pixel, the farther the object is from the depth estimation device. Since the depth estimation device is only used to roughly estimate the distance from the object to the camera and does not require complex calculations such as human detection, the resolution of the depth map acquired by the depth estimation device is smaller than the resolution of the RAW image acquired by the camera. For example, m = 30, n = 40. It should be noted that there is a one-to-one mapping between the frames in the data stream acquired by the camera and the frames in the data stream acquired by the depth estimation device. For example, each frame in both the camera and depth estimation device data streams is marked with a timestamp. Two frames marked with the same timestamp are two different forms of data acquired at the same time, used to present the same captured content.

[0017] In one possible implementation, determining the second face region based on face bounding box information includes: determining the second face region based on the face bounding box information and the first depth map. Specifically, the second face region is the face bounding box region in the second image whose depth is less than a preset depth, whose face bounding box area is greater than a preset size, and whose face bounding box area is the largest. It can be understood that by adding depth judgment to the portrait, face regions in distant images can be excluded.

[0018] In one possible implementation, determining the first face region in the third image based on data of the face region not covered by occlusions includes: determining the first face region in the third image based on the coordinates of the pixels in the face region not covered by occlusions, where the coordinates of the pixels in the first face region correspond to the coordinates of the pixels in the face region not covered by occlusions. It can be understood that since the position of the face region with the largest bounding box in the YUV image is known, and the position of the actual face region in the mask is known, the 3A module can determine the position of the actual face region in the YUV image and obtain the brightness value of each pixel in that actual face region, thereby calculating the brightness of the actual face based on the brightness value of each pixel.

[0019] In one possible implementation, the second time step is the time step prior to the first time step, and the second and third images are different YUV images. For example, if the preview frame currently displayed on the screen is the (i+k)th frame RGB image, a face detection algorithm can be used to obtain the face bounding box information of the (i+k)th frame YUV image. Then, based on the face bounding box information, a face image is cropped from the (i+k)th frame YUV image. Next, based on the face image of the (i+k)th frame YUV image, the MSAK image of the (i+k)th frame YUV image is obtained. Finally, based on the MSAK image of the (i+k)th frame YUV image, the brightness of the face region in the (i+k)th frame image is obtained. Alternatively, the second time step and the first time step are the same time step, and the second and third images are the same YUV image. For example, the electronic device can obtain the MSAK image of the (i+k)th frame YUV image based on the face bounding box information of the (i+k)th frame YUV image, and then obtain the brightness of the actual face in the (i+k)th frame YUV image based on the MSAK image.

[0020] In one possible implementation, the first mask image further includes at least one of the following: data of the face region covered by the occluded object and data of the background image. It is understood that overexposure or underexposure may occur when the face is obscured by an occluded object; overall exposure instability may also occur when the subject remains relatively stationary with respect to the electronic device, but the background image is constantly changing. By acquiring the mask image, the face region covered by the occluded object and the background image can be ignored and removed, resulting in the true face portion.

[0021] Secondly, this application provides an apparatus comprising units for performing the method described in the first aspect above. This apparatus corresponds to performing the exposure parameter adjustment method described in the first aspect above. For a detailed description of the units within this apparatus, please refer to the description in the first aspect above; for brevity, it will not be repeated here.

[0022] Thirdly, this application provides an electronic device including a memory and one or more processors. The memory stores computer program code, which includes computer instructions. When the computer instructions are invoked by the processor, the electronic device executes the exposure parameter adjustment method provided in any of the first aspects.

[0023] Fourthly, this application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When executed on an electronic device, the computer instructions cause the electronic device to perform an exposure parameter adjustment method as provided in the first aspect and any possible implementation thereof.

[0024] Fifthly, this application provides a computer program product. When the computer program product is run on a computer, it causes the computer to execute the exposure parameter adjustment method provided by the first aspect and any possible implementation thereof.

[0025] Sixthly, this application provides a chip. The chip includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The chip can be applied to electronic devices including communication modules and memory. The interface circuits are used to receive signals from the memory of the electronic device and send the received signals to the processor, the signals including computer instructions stored in the memory. When the processor invokes the computer instructions, the electronic device can execute an exposure parameter adjustment method as provided in the first aspect and any possible implementation thereof.

[0026] It is understood that the beneficial effects achieved by the apparatus of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, the computer program product of the fifth aspect, and the chip of the sixth aspect can be referred to the beneficial effects of the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0027] Figure 1 This application provides an operational schematic diagram for a portrait photography scenario.

[0028] Figure 2 A schematic diagram illustrating the acquisition of image frames based on exposure parameters provided in an embodiment of this application;

[0029] Figure 3 A schematic diagram illustrating an application scenario of an exposure parameter adjustment method provided in this application embodiment;

[0030] Figure 4 A schematic diagram illustrating an application scenario of another exposure parameter adjustment method provided in this application embodiment;

[0031] Figure 5 A schematic diagram illustrating an application scenario of another exposure parameter adjustment method provided in this application embodiment;

[0032] Figure 6 A schematic diagram illustrating an application scenario of another exposure parameter adjustment method provided in this application embodiment;

[0033] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0034] Figure 8 A schematic diagram of the architecture of an electronic device provided in an embodiment of this application;

[0035] Figure 9This application provides a module interaction diagram for shooting portrait scenes;

[0036] Figure 10 To and Figure 9 The module interaction diagram corresponds to a specific method flow diagram;

[0037] Figure 11 A schematic diagram illustrating the acquisition of an MSAK image carrying real face data, provided in an embodiment of this application;

[0038] Figure 12 This is a schematic diagram illustrating a sudden change in exposure when there are differences in image content, provided as an embodiment of this application.

[0039] Figure 13 This application provides another module interaction diagram in a portrait shooting scene;

[0040] Figure 14 A schematic diagram of a depth map acquired by a depth estimation device, provided for an embodiment of this application;

[0041] Figure 15 To and Figure 13 The module interaction diagram corresponds to a specific method flow diagram;

[0042] Figure 16 A schematic diagram illustrating the acquisition of a face depth map provided in an embodiment of this application;

[0043] Figure 17 This is a schematic diagram illustrating the acquisition of a mask image based on multiple face depth maps, as provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0045] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. In the description of this application, "and / or" is merely a way of describing the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone.

[0046] In the specification and claims of this application, the terms "first" and "second," etc., are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of the objects. For example, "first operation" and "second operation," etc., are used to distinguish different operations, rather than to describe a specific order of operations. In the embodiments of this application, "multiple" refers to two or more.

[0047] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in some embodiments," "in other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0048] With the development of terminal devices, cameras offer an increasing variety of shooting modes, such as still photography, portrait, video recording, night scene, aperture, panorama, and slow motion. Users can choose a shooting mode, such as still photography, according to their needs. Typically, after selecting a shooting mode, the terminal device displays a preview interface. If the user is satisfied with the preview, they can click the shutter button, and the terminal device will respond to the user's click to take a photo or video.

[0049] Currently, most electronic devices are equipped with automatic exposure (AE) and automatic white balance (AWB) functions. AE refers to the camera automatically adjusting exposure parameters, such as shutter speed and gain, so that the image sensor (e.g., the sensor itself) captures an image close to the target brightness at an appropriate exposure level, thus completing the automatic exposure control process. The shutter speed is related to the exposure time; the longer the shutter is open, the more light enters, and the brighter the image appears. The gain is related to ISO (ISO sensitivity), which describes the sensitivity of the image sensor to light. Higher ISO indicates greater sensitivity. In well-lit conditions, the ISO should be set to a relatively low value, while in low-light conditions, the ISO value can be appropriately increased. Exposure is the light intensity multiplied by the time it takes for the light to reach the image sensor; exposure is usually expressed in E (lux). For the AWB function, if the white in the original image captured by the image sensor is not processed by AWB, the white image may appear as different colors under different color temperatures. For example, it may appear bluish under low color temperature (such as on a cloudy day) and yellowish under high color temperature (such as on a sunny day). The AWB function restores the white after imaging under different color temperatures to the true white (usually the white that the human eye observes under natural sunlight ambient light) through the white balance algorithm.

[0050] In scenes where a human figure is present in the preview video stream captured by the camera, the electronic device can calculate the overall brightness of the face area, and then perform AE processing and / or AWB processing based on the overall brightness to capture an image that meets the brightness requirements.

[0051] Taking After Effects (AE) processing as an example, Figure 1 This diagram illustrates an operation in a portrait photography scenario.

[0052] like Figure 1As shown in (a), the phone displays icons for applications such as the camera on the desktop. When a user wants to take a picture, they can tap the camera icon 01. The phone responds to the user's tap on the camera icon 01 and launches the camera application. After the camera application completes initialization, it notifies the camera to capture an image via the camera frame and camera driver. The camera transmits the data stream acquired by the sensor to the image signal processor. The image signal processor performs a first format conversion on the data stream acquired by the camera to obtain a first data stream (also called a tiny stream, preview stream, or preview video stream). For example, the data stream acquired by the camera is in RAW format, and the tiny stream is in RGB format. The tiny stream has a relatively small resolution, such as several hundred pixels multiplied by several hundred pixels. The image signal processor sends the tiny stream to the screen, so that the tiny stream is presented as a preview image in the preview frame, for example, in... Figure 1 The preview image displayed in the preview interface of (b) is generated based on the tiny stream. For example... Figure 1 Taking the preview interface shown in (b) as an example, the preview interface may include a preview box 02, aperture mode selection options, night scene mode selection options, portrait mode selection options, photo mode selection options, video recording mode selection options, professional mode selection options, photo preview control 04, photo capture control 05, and other controls. The preview box 02 is used to display the preview video stream captured by the camera, such as a preview video stream composed of multiple frames of the user's selfie. When the user selects the photo mode selection option, a triangle control 03 is displayed below the photo mode selection option to indicate to the user that the photo mode has been successfully selected. The photo preview control 04 is used to display the user's most recently taken photo or video in thumbnail format. The photo capture control 05 is used by the user to confirm and trigger the phone to capture a photo or video.

[0053] During the display of the preview video stream, the phone can adjust the camera's exposure parameters in real time based on the brightness of the face area in the preview video stream, thereby adjusting the brightness of subsequently captured preview video streams. Specifically, the phone can obtain the overall brightness of the face area in the captured i-th frame image, and then adjust the camera's exposure parameters based on this overall brightness, thus capturing the i+j-th frame image based on the adjusted exposure parameters, where i and j are positive integers. For example... Figure 2As shown, the mobile phone can crop out a face image 1 from image frame 1 acquired at time t1 and determine exposure parameter 1 based on the overall brightness of face image 1; crop out a face image 2 from image frame 2 acquired at time t2 and determine exposure parameter 2 based on the overall brightness of face image 2; crop out a face image 3 from image frame 3 acquired at time t3 and determine exposure parameter 3 based on the overall brightness of face image 3... Then, at time t(1+j), image frame 1+j is acquired based on exposure parameter 1; at time t(2+j), image frame 2+j is acquired based on exposure parameter 2; at time t(3+j), image frame 3+j is acquired based on exposure parameter 3.

[0054] In such Figure 1 Taking the image displayed in preview box 02 shown in (b) as the i-th frame as an example, if the phone detects that the overall brightness of the face area (such as the rectangular area surrounded by the face frame) in the i-th frame is too dark, the phone can increase the camera's exposure value and / or gain value, and acquire the i+j-th frame image based on the increased exposure value and / or gain value, and then... Figure 1 The preview box shown in (c) displays the (i+j)th frame image. The (i+j)th frame image can be the next frame image after the (i)th frame image, or there can be at least one frame interval between the (i+j)th frame image and the (i)th frame image. It should be noted that this application uses grayscale to represent the brightness of the face image in the accompanying drawings; a darker grayscale indicates a lower brightness of the face image, and a lighter grayscale indicates a higher brightness of the face image. It is understood that different types of fill patterns or other drawing methods can also be used to represent the brightness of the face image; this application does not impose any specific limitations.

[0055] After the phone adjusts the camera's exposure parameters, the brightness of the image displayed in preview frame 02 will also change accordingly. As an example, when the user adjusts the camera's exposure parameters... Figure 1 When you are satisfied with the selfie with increased brightness shown in (c), such as... Figure 1 As shown in (d), the user can click the camera control 05. In response to the user's click on the camera control 05, the camera application, through the camera frame and camera driver, notifies the image signal processor to perform a second format conversion on the data stream acquired by the camera, obtaining a second data stream (also called the image stream). For example, the data stream acquired by the camera is in RAW format, and the second data stream is in RGB format. The image stream has a relatively high resolution, such as several thousand pixels multiplied by several thousand pixels. Because each frame in the image stream has a higher resolution and better image quality, the camera application usually generates the final photo or video based on the image stream, for example, in... Figure 1 The photos in the album interface (f) are generated based on the photo stream.

[0056] It's important to note that there's a one-to-one mapping between frames in the tiny stream and frames in the captured video stream. Each frame in both the tiny stream and the captured video stream is marked with a timestamp. Frame 001 in the tiny stream and frame 101 in the captured video stream are both marked with timestamp t0, indicating that frames 001 and 101 were converted from the data stream captured by the camera at time t0; frame 002 in the tiny stream and frame 102 in the captured video stream are both marked with timestamp t1, indicating that frames 002 and 102 were converted from the data stream captured by the camera at time t1; frame 003 in the tiny stream and frame 103 in the captured video stream are both marked with timestamp t2, indicating that frames 003 and 103 were converted from the data stream captured by the camera at time t2; and so on; frame 012 in the tiny stream and frame 112 in the captured video stream are both marked with timestamp t11, indicating that frames 012 and 112 were converted from the data stream captured by the camera at time t11. Furthermore, the formats of the tiny stream and the captured video stream can be the same or different. For example, both tiny streams and image streams can be in RGB format. The difference between tiny streams and image streams is that the resolution of a tiny stream is lower than that of an image stream. Taking frames 001 and 101 as examples, since both frames 001 and 101 are obtained by converting the data stream captured by the camera at time t0, theoretically, the image content of frames 001 and 101 should be the same. Frame 001 is obtained by the image signal processor performing a first format conversion on the data stream, resulting in a smaller resolution, such as 360*720 pixels. Frame 101 is obtained by the image signal processor performing a second format conversion on the data stream, resulting in a larger resolution, such as 1024*2048 pixels.

[0057] Furthermore, such as Figure 1 As shown in (e), after the phone completes taking a photo, it updates the thumbnail in the photo preview control 04 to the thumbnail of the photo taken this time. Because the thumbnail in the photo preview control 04 is small, the user may not be able to clearly see the specific content of the photo. In this case, as... Figure 1 As shown in (f), the user can click the photo preview control 04. In response to the user's click on the photo preview control 04, the phone opens the gallery application and, as shown in [the image / image / etc.], displays the preview image. Figure 1 The interface shown in (f) displays photo 06 taken this time.

[0058] In the above scheme, when a human figure is present in the preview video stream captured by the camera, the electronic device can calculate the overall brightness of the face area and then perform AWB processing based on this overall brightness. This allows for the capture of a brighter preview image when the face image is dark and a darker preview image when the face image is bright. Therefore, the brightness of the image captured by the electronic device better meets the user's needs. For specific implementation details of the electronic device's AWB processing based on the overall brightness of the face area, please refer to the descriptions of related technologies; they will not be elaborated upon here.

[0059] Typically, the overall brightness of the aforementioned face region is calculated as the average brightness based on the brightness of each pixel within the rectangular area enclosed by the face bounding box. In some applications, the rectangular area enclosed by the face bounding box can include not only the face image but also the background image. The face image can include not only the facial area (i.e., the exposed skin area not covered by occlusions, such as the forehead, glasses, nose, mouth, cheeks, ears, and chin) but also areas covered by occlusions. If the color of the background image or occlusion differs significantly from the color of the facial area, the overall brightness of the face region calculated by the electronic device may be inaccurate. This results in inaccurate exposure parameters calculated based on the overall brightness of the face region, leading to overexposure or underexposure by the camera, ultimately affecting image quality.

[0060] The following is combined with Figures 3 to 6 The application scenarios of the exposure parameter adjustment method provided in this application are illustrated with examples.

[0061] As a scenario example, such as Figure 3 As shown in (a), when the subject is not wearing a mask, the electronic device acquires the image using normal exposure parameters. Figure 3 As shown in (b), when the subject wears a light-colored mask (i.e., the mask's color is lighter than the subject's facial area), the electronic device calculates an overly high average brightness based on the brightness of each pixel in the face area. This results in a lower exposure and gain value based on the average brightness, leading to underexposure (i.e., the overall image brightness is darker than the actual brightness), resulting in a darker final photo or video. Figure 3 As shown in (c), when the subject is wearing a dark mask (i.e., the color of the mask is darker than the color of the subject's face), the average brightness calculated by the electronic device based on the brightness of each pixel in the face area is too low, resulting in an overexposure (i.e., the overall brightness of the image is brighter than the actual brightness), which leads to the final photo or video being too bright.

[0062] As another example of a scenario, such as Figure 4As shown in (a), when the subject is wearing clear lenses (or not wearing glasses), the electronic device acquires the image using normal exposure parameters. Figure 4 As shown in (b), when the subject wears light-colored lenses (i.e., the lens color is lighter than the color of the subject's face), the electronic device calculates an overly high average brightness based on the brightness of each pixel in the face area. This results in a lower exposure and gain value based on the average brightness, leading to underexposure (i.e., the overall image brightness is darker than the actual brightness), resulting in a darker final photo or video. Figure 4 As shown in (c), when the subject wears dark lenses (i.e., the color of the lenses is darker than the color of the subject's face area), the average brightness calculated by the electronic device based on the brightness of each pixel in the face area is too low, resulting in an overexposure (i.e., the overall brightness of the image is brighter than the actual brightness), which leads to the final photo or video being too bright.

[0063] As another example of a scenario, such as Figure 5 As shown in (a), when the subject is not bearded, the entire facial area is not obscured, and the electronic device captures the image according to normal exposure parameters. Figure 5 As shown in (b), when the subject has a beard, part of the face is obscured, leaving only the uncovered portion. Because the area obscured by the beard is darker than the area not obscured, the average brightness calculated by the electronic device based on the brightness of each pixel within the entire face frame is lower. This results in an overexposure (i.e., the overall brightness of the image is brighter than the actual brightness), leading to an overexposed photo or video.

[0064] As another example of a scenario, such as Figure 6 As shown in (a), when the subject is at position P1, the upper left corner of the face frame 1 is the background image 1. Figure 6 As shown in (b), when the subject moves to position P2, there is no background image in face frame 2. Figure 6 As shown in (c), when the subject moves to position P3, the upper right corner of the face frame 3 becomes the background image 2. Figure 6As shown in (d), when the subject moves to position P4, the left side of the face frame 4 is the background image 3. In the scenario where the subject moves sequentially from position P1 to positions P2, P3, and P4, the background image within the face frame continuously changes. For example, the relative position of the background image within the face frame, the image content of the background image, and the brightness of the background image may all change. Referring to the description of the above embodiment, the electronic device calculates the average brightness based on the brightness of each pixel within the entire face frame. As the background image within the face frame continuously changes, the exposure parameters calculated by the electronic device based on the average brightness of the face frame also continuously change. This requires the electronic device to constantly adjust the camera's exposure parameters, resulting in unstable overall exposure and causing flickering (i.e., inconsistent brightness) in the captured image.

[0065] It should be noted that the above scenario examples are merely illustrative and do not constitute a limitation on this application. It is understood that overexposure, underexposure, or overall exposure instability may also occur in other scenarios. For example, overexposure or underexposure may occur when the subject's hair obscures their face, or when the subject is wearing a face mask. Furthermore, overall exposure instability may occur when the subject and electronic device remain relatively stationary, but the background image is constantly changing.

[0066] In view of the above problems, this application provides an exposure parameter adjustment method. This method can be applied to scenarios where electronic devices are used to capture facial images. When displaying a preview image based on a preview stream, if a facial image is detected in the preview image, the electronic device can first segment the actual facial portion (i.e., the face region) from the facial image, then calculate the exposure parameters based on the average brightness of the actual facial portion, and then adjust the camera's exposure parameters. In this process, by segmenting the actual facial portion from the facial image, the influence of facial obstructions such as masks, glasses, beards, or hair, as well as the background image, on the overall brightness calculation can be reduced. This makes the exposure parameters calculated based on the average brightness of the actual facial portion more accurate, resulting in a final image brightness that better meets the user's needs, avoiding the problem of continuous flickering in the captured image, improving the image quality, and enhancing the user's photography experience.

[0067] In some embodiments, the aforementioned electronic device may be a terminal device. A terminal device is also called a terminal or user equipment (UE). For example, a terminal device may be a mobile phone, a personal computer (PC), a smart screen, a smart TV, a tablet computer, a wearable device, a computer with wireless transceiver capabilities, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, or a wireless terminal in a smart home, etc., or it may be other devices or apparatuses.

[0068] Figure 7 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0069] like Figure 7 As shown, the electronic device 100 may include a processor 110, a memory 120, buttons 130, a sensor module 140, a display screen 150, an audio module 160, a speaker 160A, a receiver 160B, a microphone 160C, a headphone jack 160D, a camera 170, a depth estimation device 180, etc.

[0070] The processor 110 can be used to execute the exposure parameter adjustment method in this embodiment based on the data stream acquired by the camera 170. The processor 110 may include one or more processing units, such as a central processing unit (CPU), a neural processing unit (NPU), a graphics processing unit (GPU), an application processor (AP), a digital signal processor (DSP), or an image signal processor (ISP). Different processing units can be independent devices or integrated into one or more processors; for example, the NPU can be located within the DSP.

[0071] The memory 120 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application (APP) required for a function, such as a camera application, a gallery application, etc. The data storage area may store configuration files for each APP, as well as data created during the use of the electronic device 100, such as YUV images obtained from RAW data streams captured by a camera after basic 3A processing and image signal processing (ISP). The 3A processing includes AE processing, AWB processing, and autofocus (AF) processing.

[0072] Buttons 130 include a power button, volume buttons, etc. Buttons 130 can be mechanical buttons or touch buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100, such as key signal inputs to trigger camera 170 to capture images.

[0073] Sensor module 140 may include image sensors, touch sensors, etc. An image sensor, also known as a photosensitive element, is a device that uses the photoelectric conversion function of an optoelectronic device to convert a light image on a photosensitive surface into an electrical signal proportional to the light image. For example, an image sensor may be a complementary metal-oxide-semiconductor image sensor (CMOS image sensor, CIS). A touch sensor, also called a "touch panel," can be disposed on the display screen 150. The touch sensor and the display screen 150 together form a touch screen, also called a "touch screen." The touch sensor is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event and provide visual output related to the touch operation through the display screen 150. In some embodiments, the touch sensor may also be disposed on the surface of the electronic device 100, in a different position than the display screen 150. It should be noted that the above sensors can be independent functional modules in the electronic device or can be disposed within certain functional devices; for example, an image sensor can be integrated into a camera 170.

[0074] The display screen 150 includes a display panel for displaying the desktop, a camera preview interface, various images in the gallery, etc.

[0075] Electronic device 100 can implement audio functions, such as music playback and recording, through an audio module 160, a speaker 160A, a receiver 160B, a microphone 160C, a headphone jack 160D, and an application processor. The audio module 160 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 160 can also be used for encoding and decoding audio signals. The speaker 160A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music or make hands-free calls through the speaker 160A. The receiver 160B, also called a "handpiece," is used to convert audio electrical signals into sound signals. When electronic device 100 answers a phone call or voice message, the receiver 160B can be brought close to the user's ear to hear the voice. The microphone 160C, also called a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to the microphone 160C, inputting the sound signal into the microphone 160C. The headphone jack 160D is used to connect wired headphones.

[0076] Camera 170 is used to capture still images or videos. Light from an object passes through a lens to generate an optical image, which is then projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 170, where N is a positive integer greater than 1.

[0077] The depth estimation device 180 is used to acquire a depth map and estimate the distance information of objects in the depth map, i.e., the distance from the object being photographed to the camera, using computer vision technology. The depth estimation device 180 may include a transmitting unit, an optical lens, an imaging unit, a control unit, and a computing unit. Through these components, the depth estimation device 180 can acquire a depth map, where each pixel corresponds to the depth information of a target object, and these pixels together form a depth image. In some embodiments, the depth estimation device 180 is an independent functional device in an electronic device. In other embodiments, the depth estimation device 180 may also be a sub-device of a functional device in an electronic device; for example, the depth estimation device 180 may be a TOF sensor in a camera module, and the camera module may also include a camera 170.

[0078] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0079] The following provides an example illustration of the software system of an electronic device. The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, or microservice architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software system architecture of an electronic device.

[0080] For example, Figure 8 A schematic diagram of the architecture of an electronic device provided in an embodiment of this application is shown.

[0081] like Figure 8 As shown, electronic devices can adopt a layered architecture, dividing the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software layers of the software architecture are divided from top to bottom as follows: application layer, application framework (FW) layer, system library (FWK LIB), hardware abstraction layer (HAL) layer, and kernel layer. This software architecture runs on top of the hardware layers, which may include a display screen (also called a screen), NPU, depth estimation device, and camera, etc.

[0082] The application layer, also known as the application layer, can include a series of application packages. For example, the application layer can include a camera application, a gallery application, etc. The camera application is used to call the camera to capture tiny streams and image streams, and display the captured preview interface; the gallery application is used to store and display the final photos obtained through the camera application. When these application packages are run, they can access the various service modules provided by the application framework layer through the application programming interface (API) and execute corresponding intelligent business logic.

[0083] The application framework layer provides APIs and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. It may include an Activity Manager Service (AMS), a Window Manager Service (WMS), and a camera service, among others. AMS manages the lifecycle of each application. WMS manages all windows in the system. The camera service includes a face detection module, a mask module, and a depth estimation module, among others. The specific implementation of each module within the camera service can be found in the description of the following embodiments.

[0084] The system library can include multiple functional modules, such as a surface manager, media libraries, a 2D graphics engine (e.g., SGL), and a 3D graphics processing library (e.g., OpenGLES). The surface manager manages the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats. The 2D graphics engine is the drawing engine for 2D graphics. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0085] Within the system libraries, the Android Runtime comprises the core libraries and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that Java calls, and the other part comprises the core Android libraries. The application layer and application framework layer run within the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used for managing object lifecycles, stack management, thread management, security and exception management, and garbage collection, among other functions.

[0086] The hardware abstraction layer has standard interfaces implemented by hardware vendors. For example, the hardware abstraction layer may include camera HAL and audio HAL, etc.

[0087] The kernel layer is the layer between hardware and software, belonging to the lower layers of the Android system. The kernel layer can contain various driver interfaces, such as camera drivers, display drivers, and audio drivers. Additionally, the kernel layer can include 3A modules. Specifically, 3A modules can include the AE module, AWB module, and AF module.

[0088] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are equally applicable to electronic devices based on operating systems such as iOS or Windows.

[0089] Below Figure 7 and Figure 8 Based on the provided functional modules, combined with Figures 9 to 17 The specific implementation of the exposure parameter adjustment method provided in the embodiments of this application will be illustrated by example.

[0090] For example, Figure 9 This diagram illustrates a modular interaction in a portrait photography scenario.

[0091] like Figure 9 As shown, a camera may include at least a lens and an image sensor (such as a CIS), and a CPU may include at least a face detection module, a mask module, an AE module, and a camera driver.

[0092] In response to a user opening the camera application, the camera application sends a shooting / image acquisition request to the camera service. This request may include parameters such as the camera identifier (ID) corresponding to the current shooting scene, frame rate range, and shooting mode. The camera service sends the camera ID and frame rate range corresponding to the current shooting scene to the camera driver via the camera HAL. The camera driver can then open the corresponding camera based on the camera ID for the current shooting scene. The opened camera begins acquiring data streams.

[0093] Light emitted or reflected by the photographed object (including at least one person) passes through the lens to generate an optical image that is projected onto the CIS (CMOS Image Sensor). The CIS converts the light signal into a RAW format electrical signal (i.e., a RAW image), and then passes the RAW image to the 3A module. The 3A module performs basic 3A processing on the RAW image and then passes it to the ISP (ISP for Image Signal Processor). The ISP converts it into a digital image signal. The ISP outputs the digital image signal to the DSP (DSP for DSP processing). The DSP converts the digital image signal into a standard format image signal (such as RGB and YUV formats), such as RGB and YUV images. It should be noted that RGB and YUV images can be converted to each other. In the RGB color coding method, each pixel has three colors: red, green, and blue. In the YUV color coding method, Y represents luminance, and U and V represent chrominance. Luminance and chrominance specify the color of a pixel. For displays, images are usually displayed using the RGB model, but the YUV model is usually used when processing image data because the YUV model saves more storage space. Based on this, the DSP performs the following processing flow: On the one hand, the DSP sends the RGB format preview stream (i.e., the RGB image) to the display, so that the user can see... Figure 1The preview interface is shown in (b) above. On the other hand, the DSP stores the YUV format preview stream (i.e., the YUV image) in memory so that the CPU can execute the exposure parameter adjustment method provided in this embodiment. It can be understood that since the RGB image and the YUV image are different formats generated based on the same digital image signal, their image content is consistent, such as containing the same face image. Thus, during the display of the RGB format preview stream, the CPU can obtain the YUV format preview stream from memory and adjust the camera's exposure parameters in real time based on the brightness of the face area in the YUV format preview stream to change the brightness of the subsequently acquired preview video stream.

[0094] Specifically, after processing the (i-1)th frame of the YUV image, the face detection module can read the i-th frame of the YUV image from memory, detect the i-th frame of the YUV image to obtain the face bounding box information (such as the number of faces and the coordinates of the face bounding boxes) in the current image, and then pass the i-th frame of the YUV image and the face bounding box information of the i-th frame of the YUV image to the MSAK module. Based on the i-th frame of the YUV image and the face bounding box information of the i-th frame of the YUV image, the MSAK module crops out the mask data of the real face corresponding to the maximum face area from the i-th frame of the YUV image, and then passes the mask data to the AE module. Based on the mask data and the i-th frame of the YUV image, the AE module calculates the brightness of the real face, then determines the exposure parameters corresponding to the brightness of the real face, and sends the exposure parameters to the camera through the camera driver. The camera can readjust the exposure parameters and acquire a new data stream based on the adjusted exposure parameters.

[0095] Figure 10 To and Figure 9 The module interaction diagram is a schematic diagram of the specific method flow.

[0096] This method can be applied to, for example Figure 7 and Figure 8 The electronic device shown may include at least a CPU, an ISP, a DSP, and a camera. The CPU may include at least functional modules such as a camera application, a face detection module, a mask module, a 3A module (including an AE module), and a camera driver.

[0097] like Figure 10 As shown, the method may include the following steps S101 to S124.

[0098] S101, the camera application receives the user's action of opening the camera application or the request from other applications to call the camera application.

[0099] For example, a camera app can receive user clicks on the camera app icon, or user selection of a shooting mode in the camera app's preview interface, or user actions in other apps that trigger the opening of the camera (such as video calls, taking photos, or scanning).

[0100] S102, the camera application sends a shooting request to the camera driver, instructing the acquisition of data streams.

[0101] S103, in response to the shooting request, the camera driver sends a shooting command to the camera to instruct it to capture the data stream.

[0102] For example, in response to a user opening the camera application or a request from another application to invoke the camera application, the camera application at the application layer identifies the current shooting scene and determines the camera corresponding to the current shooting scene (e.g., opening the rear camera in a distant shooting scene, or opening the front camera in a selfie scene), the camera's frame rate range (including the maximum and minimum frame rates), and the shooting mode. The camera application sequentially sends the camera ID, frame rate range, shooting mode, and other parameters corresponding to the current shooting scene to the camera driver through the camera service and camera HAL. The camera driver can open the corresponding camera based on the camera ID corresponding to the current shooting scene. The opened camera collects a data stream based on the frame rate range and shooting mode. Referring to the description of the above embodiment, the data stream collected by the camera is a RAW format electrical signal, i.e., a RAW image.

[0103] S104, the camera returns the acquired data stream, i.e., the RAW image, to the camera driver.

[0104] S105, the camera driver transmits the RAW image to the 3A module.

[0105] S106, the 3A module performs basic 3A processing on the RAW image and then transmits the processed RAW image to the ISP.

[0106] The basic 3A processing mentioned above can include the simplest pre-processing operations such as white balance correction and image cropping, but does not include AE ​​processing.

[0107] S107, the ISP converts the processed RAW image into a digital image signal and transmits the digital image signal to the DSP.

[0108] The S108 DSP converts digital image signals into image signals in standard formats (such as RGB and YUV formats), such as RGB and YUV images.

[0109] Referring to the description of the above embodiments, RGB and YUV images are two different color encoding methods that can be converted to each other. For displays, images are typically displayed using the RGB model. The more storage-efficient YUV model is usually used when processing image data. Therefore, after the DSP converts the digital image signal into RGB and YUV images, the DSP can perform the following processing flow: On one hand, the DSP sends a preview stream (i.e., the RGB image) in RGB format to the display, allowing the user to see... Figure 1 The preview interface is shown in (b) above. On the other hand, the DSP stores the YUV format preview stream (i.e., YUV image) into the memory. Then, for each frame of the preview stream stored in the memory, the face detection module can execute S109 as described below according to the acquisition order of each frame. That is, while the electronic device is displaying the preview interface based on the acquired preview stream, the electronic device can also adjust the camera's exposure parameters based on the brightness of the face in the acquired preview stream to adjust the brightness of the preview stream to be acquired.

[0110] The following explanation uses the YUV image read from the memory by the face detection module as the i-th frame YUV image as an example.

[0111] S109, the face detection module reads the i-th frame YUV image (referred to as the second image) from the memory.

[0112] S110, the face detection module determines whether the i-th frame YUV image contains a face.

[0113] The face detection module can employ face detection algorithms, such as template matching, singular value feature-based methods, subspace analysis, local preserving projection, or principal component analysis, to determine whether the i-th frame of the YUV image includes a face. If the i-th frame of the YUV image includes a face, the exposure parameter adjustment method provided in this embodiment can be used, i.e., the following steps S111-S124 are executed. If the i-th frame of the YUV image does not include a face, the exposure parameter adjustment method provided in this embodiment can be skipped, and the (i+1)-th frame of the YUV image can be read from the memory and its presence determined. As a possible implementation, when the i-th frame of the YUV image does not include a face, the electronic device can also use other exposure parameter adjustment methods provided by related technologies to adjust the camera's exposure parameters.

[0114] S111, the face detection module obtains face bounding box information from the i-th frame YUV image.

[0115] The aforementioned face bounding box information refers to information about faces in a YUV image. For example, face bounding box information may include the number of faces in the YUV image and the coordinates of each face bounding box. The number of faces in the YUV image can be one or more. If the YUV image includes one face, a face bounding box is determined to enclose the face region; if the YUV image includes multiple faces, multiple face bounding boxes are determined to enclose the face regions. The coordinates of a face bounding box can represent the position and size of a face region in the YUV image. For example, if the face bounding box is rectangular, and a two-dimensional coordinate system is established with a vertex of the YUV image as the origin, the coordinates of a face bounding box include the coordinates of the four vertices of the rectangle.

[0116] S112, the face detection module transmits the i-th frame YUV image and its face bounding box information to the MSAK module. In this way, the MSAK module can obtain the mask data of the real face in the i-th frame YUV image based on the face bounding box information.

[0117] As a first possible implementation, the MSAK module can obtain the mask data of the real face in the i-th frame YUV image based on the face bounding box information of the i-th frame YUV image, as described in S113-S118 below. As a second possible implementation, the MSAK module can obtain the mask data of the real face in the (i+k)-th frame YUV image based on the face bounding box information of the i-th frame YUV image. The specific implementation of the second possible implementation is similar to that of the first possible implementation, and can be referred to the description in S113-S118 below, which will not be repeated here. Here, i and k are positive integers.

[0118] S113, the MSAK module obtains the face region with the largest face bounding box area in the i-th frame YUV image based on the face bounding box information of the i-th frame YUV image.

[0119] When the i-th frame of the YUV image includes a single face bounding box, the "face region with the largest face bounding box area in the i-th frame of the YUV image" refers to the face region enclosed by that single face bounding box. When the i-th frame of the YUV image includes multiple face bounding boxes, the "face region with the largest face bounding box area in the i-th frame of the YUV image" refers to the face region enclosed by the largest face bounding box among the multiple face bounding boxes.

[0120] like Figure 11As shown, a two-dimensional coordinate system xoy is established with the top-left corner of the i-th frame of the YUV image as the origin. The face detection module can determine two rectangular face bounding boxes in the i-th frame of the YUV image based on the face detection algorithm: face bounding box 1 and face bounding box 2. The coordinates of the four vertices of face bounding box 1 are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), and the coordinates of the four vertices of face bounding box 2 are (x5, y5), (x6, y6), (x7, y7), and (x8, y8). The MSAK module can calculate the area of ​​the face region enclosed by face bounding box 1 as S1 = (y7-y5)*(x6-x5) and the area of ​​the face region enclosed by face bounding box 2 as S2 = (y3-y1)*(x2-x1) based on these vertex coordinates. If S1 > S2, then the face region enclosed by face bounding box 1 is the face region with the largest face bounding box area in the i-th frame of the YUV image.

[0121] S114, the MSAK module determines whether the size of the face region with the largest face frame area is greater than or equal to the preset size.

[0122] As an example, the MSAK module determines whether the area of ​​the face region is greater than or equal to a preset area.

[0123] As another example, the MSAK module determines whether the height of the face region is greater than or equal to a preset height, and whether the width of the face region is greater than or equal to a preset width. The preset height and preset width can be equal or unequal. For example, both the preset height and preset width can be 64 pixels.

[0124] It is understood that the size of each YUV image frame is usually fixed. If the face region with the largest face frame in the i-th YUV image is smaller than the preset size, it means that the face occupies a small proportion of the i-th YUV image and has little impact on the overall image brightness. Therefore, it is not the focus of the shot, and the brightness of this face region should not be used as a basis for adjusting exposure parameters; S115 below does not need to be executed. If the face region with the largest face frame in the i-th YUV image is greater than or equal to the preset size, it means that the face occupies a large proportion of the i-th YUV image and may be the focus of the shot; S115 below can continue to be executed.

[0125] S115, the MSAK module crops the image of the face region with the largest face frame area from the i-th frame YUV image (called the first face image).

[0126] S116, the MSAK module converts the image of the face region with the largest face frame area into RGB format.

[0127] Since artificial intelligence (AI) models are typically based on MSAK images obtained from RGB format images, the face region image needs to be converted from the YUV domain to the RGB domain before being input into the AI ​​model.

[0128] S117, the MSAK module adjusts the image of the face region, converted to the RGB domain, to a preset size.

[0129] Taking a preset size of 64 pixels * 64 pixels as an example. Referring to the description of S114 in the above embodiment, since the size of the face region with the largest face frame area may far exceed 64 pixels * 64 pixels, directly inputting it into the AI ​​model may result in a large amount of computation for the AI ​​model. In order to reduce the amount of computation for the AI ​​model, the MSAK module can first adjust the image of the face region to 64 pixels * 64 pixels before inputting it into the AI ​​model.

[0130] It should be noted that the execution order of S116 and S117 is not specifically limited in this embodiment. For example, the MSAK module may execute S117 first and then S116. After S116 and S117, the MSAK module executes S118 as follows.

[0131] S118, the MSAK module calls the AI ​​model to obtain the MSAK image.

[0132] The aforementioned MSAK image includes data corresponding to the exposed skin areas not covered by occlusions, and data corresponding to the areas covered by occlusions and / or the background image. The MASK image is a matrix with the same dimensions as the original image (i.e., the i-th frame YUV image). Elements in the MASK image determine whether to retain, modify, or occlude image pixels at corresponding positions in the i-th frame YUV image. In this embodiment, the MASK image can be used to select real face regions and filter non-face regions (such as background images and areas covered by occlusions). As an example, in a real face MSAK image, a pixel value of 0 indicates that the pixel is located within the real face region, and a pixel value of 1 indicates that the pixel is located within a non-face region. As another example, a pixel value of 1 indicates that the pixel is located within the real face region, and a pixel value of 0 indicates that the pixel is located within a non-face region.

[0133] In some embodiments, to reduce the overall performance and power consumption of the electronic device, some algorithms can be deployed on the CPU, while model inference and some preprocessing algorithms can be placed in the DSP. For example, the DSP may have an NPU, whose AI model can perform model inference. After the MSAK module acquires the image of the face region, the image of the face region can be input into the AI ​​model to obtain the MSAK image carrying real face data from the AI ​​model.

[0134] For example, continue as follows Figure 11 As shown, the area enclosed by face frame 1 is larger than the area enclosed by face frame 2. Therefore, the MSAK module can crop the image of the area enclosed by face frame 1 from the i-th frame YUV image. The image of the area enclosed by face frame 1 includes the real face area (i.e., the exposed skin area not covered by occlusions, such as the forehead, glasses, nose, mouth, cheeks, ears, and chin), the background image, and the area covered by occlusions (such as the face area covered by a mask). Referring to the description of the above embodiment, if the color of the background image or the area covered by occlusions differs significantly from the color of the real face area, the calculated overall brightness of the face area may not be accurate, making the exposure parameters calculated based on the overall brightness of the face area inaccurate. Based on this, the MSAK module can call an AI model to obtain an MSAK image carrying real face data to ignore or delete images of non-face areas.

[0135] In some embodiments, the AI ​​model described above can be a portrait segmentation model. Specifically, after adjusting the image of the face region to a preset RGB size through steps S116 and S117, the MSAK module can input the image into the portrait segmentation model. The portrait segmentation model continues to segment the image, obtaining data of size 2*64*64 after segmentation, where 2 represents the dimension and 64 represents the width and height. Then, the portrait segmentation model uses the argMax() operation to reduce it to a data size of 1*64*64, becoming a 0-1 MSAK image, and then maps the 1*64*64 MSAK image to a data size of 1*16*16. Here, argMax() is a maximum value independent variable point set function, representing the search for the parameter with the maximum score.

[0136] S119, the MSAK module transmits the MSAK image and the i-th frame YUV image to the 3A module.

[0137] Specifically, the MSAK module transmits the MSAK image and the i-th frame YUV image to the AE module of the 3A module so that the AE module can perform AE processing, that is, execute the following S120 and S121.

[0138] The S120,3A module calculates the brightness of the real face based on the MASK image and the i-th frame YUV image.

[0139] The aforementioned "brightness of the real face" refers to the brightness of the region corresponding to the mask image in the i-th frame of the YUV image. It can be understood that since the position of the face region with the largest face bounding box in the i-th frame of the YUV image is known, and the position of the real face region in the mask image is also known, the 3A module can determine the position of the real face region in the i-th frame of the YUV image and obtain the brightness value of each pixel in that real face region. Therefore, it can calculate the brightness of the real face based on the brightness value of each pixel; for example, it can use the average brightness value of all pixels in the real face region as the brightness value of the real face.

[0140] The S121,3A module determines the exposure parameters corresponding to the brightness of the real human face.

[0141] The exposure parameters mentioned above include exposure value and gain value. Exposure value is related to exposure time; the longer the shutter is open, the more light enters, and the brighter the image will appear. Gain value is related to ISO; ISO describes the sensitivity of the image sensor to light. The higher the ISO, the greater the sensitivity. In well-lit conditions, set the ISO to a relatively low value, and in poor lighting conditions, the ISO value can be increased appropriately.

[0142] In this embodiment, the brightness of the face is negatively correlated with the exposure value and gain value. When the brightness of the face is high, the calculated exposure value and gain value are small to reduce the brightness of the subsequently acquired image; when the brightness of the face is low, the calculated exposure value and gain value are large to increase the brightness of the subsequently acquired image; when the brightness of the face is just right, there is no need to adjust the exposure parameters of the camera.

[0143] The S122,3A module sends exposure parameters to the camera driver.

[0144] S123, Camera driver instructs the camera to adjust exposure parameters.

[0145] The camera can acquire a new data stream (i.e., a new RAW image) based on the adjusted exposure parameters.

[0146] In step S124, the camera returns the newly acquired RAW image to the camera driver. The camera driver can then pass the new RAW image to the 3A module to execute a new round of processing. Refer to the description of steps S106-S123 in the above embodiment; further details are omitted here.

[0147] In the above solution, when displaying a preview image based on a preview stream, if a face image is detected in the preview image, the actual face portion can be segmented from the face image first. Then, exposure parameters are calculated based on the average brightness of the actual face portion, and the camera's exposure parameters are adjusted accordingly. By segmenting the actual face portion from the face image, the influence of facial obstructions such as masks, glasses, beards, or hair, as well as the background image, on the overall brightness calculation can be reduced. This makes the exposure parameters calculated based on the average brightness of the actual face portion more accurate, resulting in a final image brightness that better meets the user's needs. It also avoids the problem of continuous flickering in the captured image, improving the quality of the final image and enhancing the user's photography experience.

[0148] The above embodiment illustrates how to obtain the MSAK image of the i-th frame YUV image based on the face bounding box information, and then obtain the brightness of the real face in the i-th frame YUV image based on the MSAK image. In actual implementation, depending on the scheduling logic from the face detection module to the MASK module, the MSAK image obtained by the MASK module is not necessarily the MSAK image of the preview frame currently displayed on the screen. For example, the preview frame currently displayed on the screen might be the (i+k)-th frame, while the MSAK image obtained by the MASK module is the MSAK image of the i-th frame YUV image. When the image content of the i-th frame YUV image and the (i+k)-th frame image are identical, the above exposure parameter adjustment method can meet the requirement of accurate exposure parameter adjustment. When the image content of the i-th frame YUV image and the (i+k)-th frame image differs (e.g., the subject moves), if the brightness of the real face in the (i+k)-th frame image is obtained based on the MSAK image of the i-th frame YUV image, there may be a sudden exposure problem.

[0149] For example, let's take the currently displayed preview frame as the i+kth frame RGB image (referred to as the first image). Figure 12 As shown, the i-th frame YUV image (referred to as the second image) is converted from the RAW image acquired at time ti. The i+k-th frame YUV image (referred to as the third image) and the i+k-th frame RGB image ( Figure 12 (Not shown) is the result of converting the RAW image acquired at time t(i+k). The face detection module can obtain the face bounding box information of the i-th frame YUV image through a face detection algorithm, such as the coordinates of face bounding box 1. Then, the MASK module crops the face image (called the first face image) from the i-th frame YUV image based on the coordinates of face bounding box 1, and then obtains the MSAK image of the i-th frame YUV image based on the face image of the i-th frame YUV image. Then, the 3A module obtains the face region (e.g., ...) of the i+k frame image based on the MSAK image of the i-th frame YUV image. Figure 12The brightness of the shaded area shown. It can be understood that if the subject is in motion, the specific position of the subject in the i-th frame YUV image and the i+k-th frame YUV image will be different. This will cause the face image cropped based on the face bounding box 1 to not cover the complete face, thus making the MASK image (called the first mask image) inaccurate, which in turn makes the exposure parameters obtained based on the MASK image inaccurate, resulting in the problem of sudden exposure.

[0150] Furthermore, in certain scenarios, when the preview stream includes a distant image, and the distant image is a large-sized face image (such as a face image on a lightbox), if the exposure parameter adjustment method provided in the above embodiments is used, the electronic device may determine that the face image on the lightbox is the face image with the largest face bounding box area, and obtain the exposure parameters based on the face image on the lightbox. Although the face image on the lightbox is large, users are usually more concerned with close-up images than distant images; therefore, the exposure parameters obtained based on the face image on the lightbox may also be inaccurate.

[0151] Based on the above reasons, and building upon the exposure parameter adjustment method provided in the aforementioned embodiments, this application provides an improved exposure parameter adjustment method. When displaying a preview image based on a preview stream, the electronic device can acquire depth data of the face image, and then search for the face region with the largest face frame area (where the depth is less than a preset depth and the face frame area is greater than a preset size). This avoids misidentifying larger face images in distant images as face images used for MASK image extraction, resulting in more accurate final face region selection. Furthermore, the AI ​​model adds temporal processing functionality to the traditional portrait segmentation algorithm. The AI ​​model internally stores multiple frames of historical portrait data, allowing it to predict the MASK image of the (i+k)th frame YUV image based on these multiple frames of historical portrait data and the i-th frame YUV image. This compensates for frame differences during transmission, reduces incorrect exposure operations, and improves the final image quality.

[0152] The following is combined with Figures 13 to 17 An example is provided to illustrate the improved method for adjusting exposure parameters.

[0153] exist Figure 9 Based on the provided module interaction diagram, Figure 13 This shows another module interaction diagram in a portrait shooting scenario.

[0154] and Figure 9 The difference is that, in such Figure 13 The interactive diagram of the module shown includes a depth estimation device. This device uses computer vision technology to estimate the distance information of objects in the depth map, specifically the distance from the object being photographed to the camera.

[0155] In response to the user opening the camera application, the camera application sends a shooting request to the camera driver. In response to this shooting request, the camera driver can issue shooting commands to the camera and the depth estimation device, respectively.

[0156] On one hand, the camera begins acquiring a data stream, and the light emitted or reflected by the object being photographed passes through the lens to generate an optical image that is projected onto the CIS. The CIS converts the optical signal into a RAW format electrical signal, i.e., a RAW image, and then passes the RAW image to the 3A module. The 3A module performs basic 3A processing on the RAW image and then passes it to the ISP. The ISP converts it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard format (such as RGB and YUV formats), such as RGB and YUV images. Refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0157] On the other hand, depth estimation devices can acquire a data stream consisting of depth information maps (also known as depth images or depth maps), with each frame of the depth information map carrying depth information representing the distance from the object being photographed to the camera / camera. For example, Figure 14 A schematic diagram of a depth map acquired by a depth estimation device is shown. (As shown) Figure 14 As shown, the depth map consists of an array of m*n pixels. The larger the depth value of a pixel, the farther the object being photographed is from the depth estimation device. It should be noted that since the depth estimation device is only used to roughly estimate the distance from the object being photographed to the camera, and does not require complex calculations such as human detection, the resolution of the depth map acquired by the depth estimation device is lower than the resolution of the RAW image acquired by the camera. For example, m = 30, n = 40.

[0158] It should be noted that there is a one-to-one mapping between the frames in the data stream captured by the camera and the frames in the data stream captured by the depth estimation device. For example, each frame in both the camera and depth estimation device data streams is marked with a timestamp. Two frames marked with the same timestamp are two different forms of data captured at the same time, used to present the same captured content.

[0159] Taking the currently displayed preview frame as the (i+k)th frame RGB image as an example, the face detection module can read the (i+k)th frame YUV image from the memory and perform detection on it to obtain face bounding box information (such as the number of faces and the coordinates of the face bounding boxes) in the (i+k)th frame YUV image. The MSAK module can obtain the (i+k)th frame YUV image and the face bounding box information of the (i+k)th frame YUV image from the memory / face detection module, as well as the (i+k)th frame depth map from the depth estimation device. The (i+k)th frame depth map is acquired by the depth estimation device at time ti, the (i+k)th frame YUV image is converted from the RAW image acquired at time ti, and the (i+k)th frame YUV image and the (i+k)th frame RGB image are converted from the RAW image acquired at time t(i+k). Based on the face bounding box information of the (i+k)th frame YUV image and the (i+k)th frame depth map, the MSAK module filters out the face regions with the largest face bounding box area (where the depth of the face image is less than a preset depth and the face bounding box area is greater than a preset size). Then, the MSAK module, based on the selected face region and storing multiple frames of historical portrait data within the AI ​​model, obtains mask data, which is then passed to the AE module. The AE module, based on this mask data and the (i+k)th frame of the YUV image, calculates the brightness of the real face, determines the exposure parameters corresponding to the brightness of the real face, and sends these exposure parameters to the camera via the camera driver. The camera can then readjust the exposure parameters and acquire a new data stream based on the adjusted exposure parameters.

[0160] Figure 15 To and Figure 13 The module interaction diagram corresponds to a specific method flow diagram. Figure 10 The difference is that, in such Figure 15 In the illustrated method flow, the electronic device may also include a depth estimation device, and the AI ​​model stores multiple frames of historical portrait data. For example... Figure 15 As shown, the method may include the following steps S201 to S226.

[0161] S201, The camera application receives a user's action of opening the camera application or a request from another application to call the camera application.

[0162] S202, the camera application sends a shooting request to the camera driver, instructing the acquisition of data streams.

[0163] For the specific implementation of S201-S202, please refer to the description of S101-S102 in the above embodiments.

[0164] S203, in response to the shooting request, the camera driver sends a shooting command to the camera and depth estimation device, instructing them to acquire the data stream.

[0165] In response to a user opening the camera application or a request from another application to invoke the camera application, the camera application at the application layer identifies the current shooting scene and determines the shooting parameters corresponding to that scene. The camera application then sends these shooting parameters to the camera driver via the camera service and the camera HAL. The camera driver can issue shooting commands to the camera and depth estimation device, respectively, instructing them to acquire data streams. In response to these shooting commands, the camera begins acquiring a data stream composed of RAW images, while the depth estimation device acquires a data stream composed of depth maps. Each frame of the depth map carries depth information representing the distance from the object being photographed to the camera / sensor.

[0166] S204, the camera returns the acquired data stream, i.e., the RAW image, to the camera driver.

[0167] S205, the camera driver passes the RAW image to the 3A module.

[0168] S206, the 3A module performs basic 3A processing on the RAW image and then transmits the processed RAW image to the ISP.

[0169] S207, the ISP converts the processed RAW image into a digital image signal and transmits the digital image signal to the DSP.

[0170] The S208 DSP converts digital image signals into image signals in standard formats (such as RGB and YUV formats), such as RGB and YUV images.

[0171] For the specific implementation of S204-S208, please refer to the description of S104-S108 in the above embodiments.

[0172] After the DSP converts the digital image signal into RGB and YUV images, the DSP can perform the following processing flow: on the one hand, the DSP sends the RGB format preview stream (i.e., RGB image) to the display; on the other hand, the DSP puts the YUV format preview stream (i.e., YUV image) into the memory. Then, for each frame of the preview stream stored in the memory, the face detection module can execute the following S209 according to the acquisition order of each frame of the image.

[0173] The following example uses the preview image currently displayed on the screen as the i+kth frame RGB image based on the data stream (referred to as the first image). The face detection module can execute the following S209-S211.

[0174] S209, the face detection module reads the i-th frame YUV image from the memory.

[0175] S210, the face detection module determines whether the i-th frame YUV image contains a face.

[0176] The face detection module can employ face detection algorithms, such as template matching, singular value feature-based methods, subspace analysis, local preserving projection, or principal component analysis, to determine whether the i-th frame of the YUV image includes a face. If the i-th frame of the YUV image includes a face, the exposure parameter adjustment method provided in this embodiment can be used, i.e., the following steps S211-S226 are executed. If the i-th frame of the YUV image does not include a face, the exposure parameter adjustment method provided in this embodiment does not need to be executed; instead, the (i+1)-th frame of the YUV image can be read from the memory, and it can be determined whether the (i+1)-th frame of the YUV image includes a face.

[0177] S211, the face detection module obtains face bounding box information from the i-th frame YUV image.

[0178] The aforementioned face bounding box information refers to the information of the face image in the i-th frame of the YUV image. For example, the face bounding box information may include the number of faces in the YUV image and the coordinates of each face bounding box.

[0179] For the specific implementation of S209-S211, please refer to the description of S109-S111 in the above embodiments.

[0180] S212, the MSAK module retrieves the (i+k)th frame YUV image from the memory and the face bounding box information of the (i+k)th frame YUV image from the face detection module. The (i+k)th frame YUV image is converted from the RAW image acquired at time ti (referred to as the second time point), and the (i+k)th frame YUV image (referred to as the third image) and the (i+k)th frame RGB image are converted from the RAW image acquired at time t(i+k) (referred to as the first time point).

[0181] S213, the MSAK module acquires the i-th frame depth map from the depth estimation device. The i-th frame depth map is acquired by the depth estimation device at time ti (called the second time).

[0182] S214, the MSAK module obtains the face region with the largest face area in the i-th frame YUV image based on the face bounding box information of the i-th frame YUV image and the depth map of the i-th frame, which has a depth less than a preset depth and a face bounding box area greater than a preset size.

[0183] For example, when the i-th frame YUV image includes at least one face bounding box, the MSAK module can first filter out face regions with a depth less than a preset depth from the at least one face bounding box, then filter out face regions with a face bounding box area greater than a preset size from the face regions with a depth less than the preset depth, and finally select the face region with the largest face bounding box area. In other embodiments, the MSAK module can also first filter out face regions with a depth less than the preset depth from the at least one face bounding box, then select the face region with the largest face bounding box area from the face regions with a depth less than the preset depth, and finally determine whether the size of the face region with the largest face bounding box area is greater than or equal to a preset size. For the specific implementation of filtering out the face region with the largest face bounding box area greater than the preset size from the face bounding box, please refer to the description of S113 and S114 in the above embodiments, which will not be repeated here.

[0184] Since the depth map and the YUV image of frame i are acquired at the same time, the image content of the two frames is consistent. For example, the relative position of the same face is the same in both frames, meaning there is a mapping relationship between the pixels of the YUV image of frame i and the pixels of the depth map of frame i. Based on this mapping relationship, the depth value of each face bounding box in the YUV image of frame i can be obtained. The depth value of a face bounding box can be the average of the depth values ​​of all pixels within that box. It can be understood that the larger the depth value of a face bounding box, the farther the distance from the object corresponding to that box to the depth estimation device. If the depth value of a face bounding box is greater than or equal to a preset depth, it means that the distance from the object corresponding to that box to the depth estimation device exceeds the preset distance, indicating it is distant content. In this case, the face region of that face bounding box can be ignored or deleted.

[0185] S215, the MSAK module crops the image of the face region from the i-th frame YUV image.

[0186] S216, the MSAK module converts the image of the face region into RGB format.

[0187] Since AI models typically acquire MSAK images based on RGB format images, the face region image needs to be converted from the YUV domain to the RGB domain before inputting it into the AI ​​model.

[0188] S217, the MSAK module adjusts the image of the face region converted to the RGB domain to a preset size, and obtains the RGB domain data of the face region.

[0189] Taking a preset size of 64 pixels * 64 pixels as an example. Since the size of the face region image after conversion to the RGB domain may be much larger than 64 pixels * 64 pixels, directly inputting it into the AI ​​model may result in a large amount of computation for the AI ​​model. In order to reduce the computational load of the AI ​​model, the MSAK module can first adjust the face region image to 64 pixels * 64 pixels, thereby obtaining the RGB domain data of the face region.

[0190] S218, the MSAK module adjusts the face region of the i-th frame depth map to a preset size and obtains the depth information data of the face region.

[0191] The face region in the aforementioned "face region of the depth map of frame i" corresponds to the same face region cropped from the YUV image of frame i. The aforementioned "depth information data of the face region" includes the depth value of each pixel in the face region.

[0192] It should be noted that the preset sizes of S217 and S218 are the same, such as 64 pixels * 64 pixels. By adjusting the face region of the depth map of the i-th frame to the preset size, the face region in the "face region of the depth map of the i-th frame" can be made to have the same size as the face region cropped from the YUV image of the i-th frame, thus facilitating the rearrangement of the depth information data and the RGB domain data of the face region.

[0193] S219, the MSAK module rearranges the depth information data and RGB domain data of the face region to obtain the face depth map.

[0194] For example, such as Figure 16 As shown, the face detection algorithm can detect two face images in the i-th frame YUV image: one face located closer to the camera within face frame 4, and another face located farther from the camera on a lightbox within face frame 3. Since the depth A of face frame 4 is greater than a preset depth, the depth B of face frame 3 is less than a preset depth, and the size of face frame 4 is larger than a preset size, the MSAK module can crop the face image of face frame 4 from the i-th frame YUV image. This face image is then converted to the RGB domain and adjusted to 64 pixels * 64 pixels to obtain the depth information data of the face region. Additionally, the MSAK module adjusts the same region in the i-th frame depth map to 64 pixels * 64 pixels to obtain the depth information data of the face region. Finally, the MSAK module rearranges the depth information data and the RGB domain data of the face region to obtain the face depth map.

[0195] The RGB data of the face region is 3*64*64 pixels, where 3 represents the dimension and 64 represents the width and height. After rearranging the depth information data and the RGB data of the face region, the 3D input data (3*64*64) becomes 4D input data (4*64*64). That is, the face depth map is 4D input data (4*64*64).

[0196] S220, the MSAK module calls the AI ​​model to obtain the MASK image based on the face depth map obtained through S219 and multiple frames of historical portraits stored inside the AI ​​model.

[0197] In some embodiments, the AI ​​model described above can be a human face segmentation model.

[0198] To reduce the overall performance and power consumption of electronic devices, some algorithms can be deployed on the CPU, while model inference and some preprocessing algorithms can be placed in the DSP. For example, the DSP can have an NPU, whose AI model can perform model inference. After the MSAK module obtains the face depth map through S219, the face depth map can be input into the AI ​​model to obtain the MSAK image carrying real face data from the AI ​​model.

[0199] Referring to the description of the above embodiments, the MSAK image obtained by the MASK module is not the MSAK image of the preview frame currently displayed on the screen. For example, the preview frame currently displayed on the screen is the (i+k)th frame image, while the MSAK image obtained by the MASK module is the MSAK image of the (i+k)th frame YUV image. When the image content of the (i+k)th frame YUV image differs from that of the (i+k)th frame image (e.g., the person being photographed has moved), if the MSAK image of the (i+k)th frame YUV image is obtained based on the face bounding box information of the (i+k)th frame YUV image, there may be a problem of sudden exposure changes. To solve this problem, the AI ​​model adds a temporal processing function to the traditional portrait segmentation algorithm, and the AI ​​model internally stores multiple frames of historical portrait data. In this way, the MSAK module can call the AI ​​model to predict the MASK image corresponding to the (i+k)th frame YUV image based on the face depth map obtained through S219 and the multiple frames of historical portrait data stored internally by the AI ​​model.

[0200] For example, consider storing two historical portrait frames within the AI ​​model. Figure 17As shown, the AI ​​model internally stores the face depth maps of the (i-2)th and (i-1)th YUV frames. The MSAK module can input the face depth map of the i-th YUV frame into the AI ​​model. Based on the face depth maps of the (i-2)th, (i-1)th, and i-th YUV frames, the AI ​​model can estimate the movement speed of the subject, and then predict the MASK image corresponding to the (i+k)th YUV frame based on the subject's movement speed. Then, the portrait segmentation model uses the argMax() operation to reduce the data to a size of 1*64*64, transforming it into a 0-1 MSAK image, and then mapping the 1*64*64 MSAK image to a size of 1*16*16. Here, argMax() is a function that finds the parameter with the maximum score.

[0201] S221, the MSAK module transmits the MSAK image and the i+kth frame YUV image obtained through S220 to the 3A module.

[0202] Specifically, the MSAK module transmits the MSAK image and the i+kth frame YUV image to the AE module of the 3A module so that the AE module can perform AE processing, that is, execute the following S222 and S223.

[0203] The S222,3A module calculates the brightness of the real face in the i+k frame YUV image based on the MASK image.

[0204] The aforementioned "brightness of the real human face" refers to the brightness of the region corresponding to the MASK image in the i+k frame YUV image.

[0205] The S223,3A module determines the exposure parameters corresponding to the brightness of the real human face.

[0206] The exposure parameters mentioned above include the exposure value and the gain value.

[0207] The S224,3A module sends exposure parameters to the camera driver.

[0208] S225, the camera driver instructs the camera to adjust exposure parameters.

[0209] The camera can acquire a new data stream, i.e., a new RAW image, based on the adjusted exposure parameters.

[0210] S226, the camera returns the newly acquired RAW image to the camera driver. The camera driver can then pass the new RAW image to the 3A module to execute a new round of processing.

[0211] For the specific implementation of S221-S226, please refer to the description of S119-S124 in the above embodiments.

[0212] In the above scheme, when displaying the preview image based on the preview stream, the electronic device can acquire the depth data of the face image, and then find the face region with the largest face frame area that is less than a preset depth and larger than a preset size. This avoids misidentifying large face images in distant images as face images used to extract the mask image, making the final face region selection more accurate. In addition, the AI ​​model adds a temporal processing function to the traditional portrait segmentation algorithm. The AI ​​model internally stores multiple frames of historical portrait data. In this way, the AI ​​model can predict the mask image of the (i+k)th frame of YUV image based on multiple frames of historical portrait data and the i-th frame of YUV image, thereby compensating for frame differences in the transmission path, reducing false exposure operations, and improving the quality of the final image.

[0213] This application also provides an electronic device, including a processor coupled to a memory, the processor being used to execute computer programs or instructions stored in the memory, so that the electronic device implements the methods in the above embodiments.

[0214] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the method described above. The computer instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or can include one or more data storage devices such as servers or data centers that can be integrated with the medium. Available media can be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media, or semiconductor media (e.g., solid-state disks (SSDs)).

[0215] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0216] This application also provides a chip coupled to a memory. This chip is used to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments. The chip can be a general-purpose processor or a special-purpose processor. It should be noted that the chip can be implemented using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0217] The electronic device, computer-readable storage medium, computer program product, and chip provided in the embodiments of this application are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0218] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0219] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0220] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0222] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0223] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An exposure parameter adjustment method characterized by, The method includes: The preview interface displays the first image based on the first data stream; Based on the first depth map, a first face image is cropped from the second image based on the first data stream. The depth of the first face image is less than a preset depth. The first depth map is acquired at the second time. Arrange the RGB domain data of the first face image and the depth information data of the first depth map to obtain the first face depth map; The first face depth map is input into the artificial intelligence model, which saves multiple frames of historical portraits. The multiple frames of historical portraits are face depth maps obtained based on RAW images collected before the second time. The artificial intelligence model is invoked to obtain a first mask image corresponding to the third image based on the first face depth map and the multi-frame historical portrait. The first mask image includes at least the data of the face area not covered by the occluder. Based on the data of the face region that is not covered by the obstruction, the first face region is determined in the third image; Based on the brightness of the first face region, obtain the first exposure parameters; The camera is controlled to acquire a second data stream based on the first exposure parameters; Wherein, the first image is an RGB image converted from a RAW image acquired at the first time point, the second image is a YUV image converted from a RAW image acquired at the second time point, and the third image is a YUV image converted from a RAW image acquired at the first time point; the second time point is a time prior to the first time point, or the second time point and the first time point are the same time point.

2. The method of claim 1, wherein, The step of cropping the first face image from the second image based on the first data stream based on the first depth map includes: The face detection algorithm is used to obtain face bounding box information, which includes the number of faces in the second image and the coordinates of each face bounding box, with each face bounding box corresponding to one face. Based on the face frame information and the first depth map, a second face region is determined. The second face region is the face frame region in the second image whose depth is less than a preset depth, whose face frame area is greater than a preset size, and whose face frame area is the largest. If the size of the second face region is greater than or equal to a preset size, the first face image is cropped from the second image, and the first face image is the image of the second face region.

3. The method of claim 2, wherein, After cropping the first face image from the second image, the method further includes: Convert the first face image to RGB format; The first face image, converted to RGB format, is adjusted to the preset size.

4. The method of claim 1, wherein, Before displaying the first image based on the first data stream in the preview interface, the method further includes: The first depth map is acquired using a depth estimation device.

5. The method according to any one of claims 1 to 4, characterized in that, The determination of the first face region in the third image based on the data of the face region not covered by the obstruction includes: Based on the coordinates of the pixels in the face region that is not covered by the obstruction, a first face region is determined in the third image, and the coordinates of the pixels in the first face region correspond to the coordinates of the pixels in the face region that is not covered by the obstruction.

6. The method according to any one of claims 1 to 4, characterized in that, The second time point is a time point prior to the first time point, and the second image and the third image are different YUV images; or, the second time point and the first time point are the same time point, and the second image and the third image are the same YUV image.

7. The method according to any one of claims 1 to 4, characterized in that, The first mask image further includes at least one of the data of the face region covered by the occluder and the data of the background image.

8. An electronic device, comprising: The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7.

Images