Image processing methods and electronic devices

Abstract

This application provides an image processing method and an electronic device. The method includes: displaying a camera application's photo preview interface, the photo preview interface including a first thumbnail, and the camera application's HDR function being enabled; receiving a user's first operation, the first operation instructing continuous shooting of multiple frames of images; responding to the first operation, executing the first process once at preset intervals until a second operation is received from the user, the second operation instructing stopping shooting; the first process includes: acquiring multiple frames of raw images through an image sensor; performing HDR fusion, conversion, and encoding on the multiple frames of raw images through an ISP to obtain a first target image; saving the first target image, and refreshing the first thumbnail to be the thumbnail of the first target image. This method enables HDR continuous shooting.

Description

Technical Field

[0001] This application relates to the field of electronic technology, specifically to an image processing method and an electronic device. Background Technology

[0002] Currently, photography has become an important function of electronic devices. However, the inventors discovered that in some scenarios, such as high dynamic range (HDR) scenarios, electronic devices cannot achieve continuous shooting (i.e., burst shooting), which affects the user experience. Summary of the Invention

[0003] This application provides an image processing method and an electronic device that enables continuous shooting in HDR scenes.

[0004] In a first aspect, this application provides an image processing method executed by an electronic device, the electronic device including an image sensor and an image signal processor (ISP). The method includes: displaying a camera application's photo preview interface, the photo preview interface including a first thumbnail, and the camera application's HDR function being enabled; receiving a user's first operation, the first operation being used to instruct the continuous capture of multiple frames of images; responding to the first operation, executing the first process once at preset intervals until a second operation is received from the user, the second operation being used to instruct the capture to stop; the first process includes: acquiring multiple frames of original images through the image sensor; performing HDR fusion, conversion, and encoding on the multiple frames of original images through the ISP to obtain a first target image; saving the first target image, and refreshing the first thumbnail to be a thumbnail of the first target image.

[0005] The image processing method provided in the first aspect can be the method shown in Embodiment 2 of the specific embodiments. The thumbnail of the first target image can be an HDR JPEG real image in the specific embodiments.

[0006] Optionally, the original image can be a RAW format image. Optionally, the preset duration can be the frame interval of the electronic device, i.e., 1 / frame rate.

[0007] The image processing method provided in the first aspect, in HDR scenarios, uses an ISP to perform HDR fusion, conversion, and encoding on multiple frames of original images to achieve the capture of an HDR image (i.e., the first target image). This allows for several advantages: firstly, for capturing any two adjacent HDR images, the acquisition of the multiple original images corresponding to the later HDR image and the HDR fusion, conversion, and encoding of the earlier HDR image can reuse time, thus shortening the capture time for two HDR images; secondly, the hardware chip ISP can process multiple original images more quickly without consuming software system resources, reducing system load, shortening HDR image processing time, and ultimately enabling HDR continuous shooting, thus improving the user experience.

[0008] In one possible implementation, the original image is a RAW format image. The first target image is obtained by performing HDR fusion, conversion and encoding on multiple frames of the original image through an ISP. This includes: performing an HDR fusion algorithm on multiple frames of the original image through an ISP to obtain a first fused image; performing a RAW domain to YUV domain conversion algorithm on the first fused image through an ISP to obtain a first YUV image; and performing an encoding algorithm on the first YUV image through an ISP to obtain the first target image.

[0009] In this implementation, HDR fusion, conversion, and encoding are achieved by executing HDR fusion algorithms, RAW-to-YUV domain conversion algorithms, and encoding algorithms to obtain an image in the target format. The target format can be, for example, PNG or JPEG. Furthermore, in this implementation, the HDR fusion algorithm, RAW-to-YUV domain conversion algorithm, and encoding algorithm are all implemented using an ISP (Internet Service Provider Interface), enabling faster processing of multiple frames of original images without consuming software system resources, thus reducing system load and further shortening the processing time for each HDR image frame.

[0010] In one possible implementation, the electronic device includes a front-end algorithm module and an ISP resource call module. The ISP performs an HDR fusion algorithm on multiple frames of original images to obtain a first fused image. The process includes: the front-end algorithm module calling the ISP resource call module to execute the HDR fusion algorithm on the multiple frames of original images; the ISP resource call module performing data conversion and encapsulation on the multiple frames of original images to obtain multi-frame converted images; the ISP resource call module calling the ISP to execute the HDR fusion algorithm on the multi-frame converted images; the ISP performing the HDR fusion algorithm on the multi-frame converted images to obtain an original fused image; the ISP returning the original fused image to the ISP resource call module; the ISP resource call module performing data conversion on the original fused image to obtain the first fused image; and the ISP resource call module returning the first fused image to the front-end algorithm module.

[0011] In other words, the HDR fusion algorithm is executed by calling the ISP resource invocation module through the front-end algorithm module, and then calling the ISP through the ISP resource invocation module. Optionally, the ISP resource invocation module may include an interface layer for communication with the front-end algorithm module. Additionally, the ISP resource invocation module can communicate with the ISP through an ISP-encapsulated interface.

[0012] In this implementation, the ISP resource call module converts and encapsulates multiple frames of original images to obtain multiple frames of converted images. These converted images can be recognized and processed by the ISP, thereby executing the HDR fusion algorithm. The resulting original fused image, after conversion by the ISP resource call module, yields a first fused image that can be recognized by the front-end algorithm module and the camera application. In other words, the ISP resource call module enables the conversion and transmission of input and output data for the HDR fusion algorithm, thereby allowing the front-end algorithm module to call the ISP.

[0013] In one possible implementation, the electronic device includes a front-end algorithm module and an ISP resource call module. The ISP performs a RAW-to-YUV domain conversion algorithm on the first fused image to obtain a first YUV image. This includes: the front-end algorithm module calling the ISP resource call module to perform the RAW-to-YUV domain conversion algorithm on the first fused image; the ISP resource call module performing data conversion and encapsulation on the first fused image to obtain a fused converted image; the ISP resource call module calling the ISP to perform the RAW-to-YUV domain conversion algorithm on the fused converted image; the ISP performing the RAW-to-YUV domain conversion algorithm on the fused converted image to obtain a first original YUV image; the ISP returning the first original YUV image to the ISP resource call module; the ISP resource call module performing data conversion on the first original YUV image to obtain a first YUV image; and the ISP resource call module returning the first YUV image to the front-end algorithm module.

[0014] In this implementation, the first fused image is converted and encapsulated using the ISP resource call module to obtain a fused converted image. The fused converted image can be recognized and processed by the ISP, thereby executing the RAW-to-YUV domain algorithm. The resulting first original YUV image, after conversion by the ISP resource call module, can be recognized by the front-end algorithm module and the camera application. In other words, the ISP resource call module realizes the conversion and transmission of input and output data for the RAW-to-YUV domain algorithm, thereby enabling the front-end algorithm module to call the ISP.

[0015] In one possible implementation, the electronic device includes a front-end algorithm module and an ISP resource call module. The ISP performs an encoding algorithm on a first YUV image to obtain a first target image. This includes: the front-end algorithm module calling the ISP resource call module to execute an encoding algorithm on the first YUV image; the ISP resource call module performing data conversion and encapsulation on the first YUV image to obtain a first YUV converted image; the ISP resource call module calling the ISP to execute an encoding algorithm on the first YUV converted image; the ISP performing an encoding algorithm on the first YUV converted image to obtain a first original target image; the ISP returning the first original target image to the ISP resource call module; the ISP resource call module performing data conversion on the first original target image to obtain the first target image; and the ISP resource call module returning the first target image to the front-end algorithm module.

[0016] In this implementation, the first YUV image is converted and encapsulated by the ISP resource call module to obtain a first YUV converted image. This first YUV converted image can be recognized and processed by the ISP, thereby executing the encoding algorithm. The encoded first original target image, after conversion by the ISP resource call module, becomes a first target image that can be recognized by the front-end algorithm module and the camera application. In other words, the ISP resource call module realizes the conversion and transmission of input and output data for the encoding algorithm, thereby enabling the front-end algorithm module to call the ISP.

[0017] In one possible implementation, the encoding algorithm is the JPEG encoding algorithm, and the first target image is a JPEG format image.

[0018] In one possible implementation, the multiple original images include a first original image and a second original image, wherein the first original image is an overexposed image and the second original image is an underexposed image.

[0019] In other words, multiple original images can be long and short RAW images.

[0020] In one possible implementation, the size of the first target image is the same as the size of the original image.

[0021] In other words, the original image was not cropped during image processing, thus preserving the integrity of the original image information, thereby improving the image quality of the obtained first target image and enhancing the user experience.

[0022] In one possible implementation, the photo preview interface includes a first control, the first operation is to long press the first control, and the second operation is to stop long pressing the first control.

[0023] The first control can be a camera control. That is, long-pressing the camera control triggers continuous shooting, and releasing the long press (i.e., raising the hand) stops continuous shooting.

[0024] Secondly, this application provides an image processing method executed by an electronic device. The electronic device includes an application layer, a hardware abstraction layer, and a hardware layer. The application layer includes a gallery application and an AIDL interface. The hardware abstraction layer includes an image processing engine. The hardware layer includes an image signal processor (ISP). The method includes: displaying a first interface of the gallery application, the first interface including a first RAW image and a second control; responding to a user's selection operation on the second control, the gallery application sends the first RAW image to the image processing engine through the AIDL interface; the image processing engine calls the ISP to perform format conversion on the first RAW image to obtain a second target image.

[0025] The image processing method provided in the second aspect can be the method shown in Embodiment 3 of the specific embodiments. The first RAW image can be the RAW image to be converted in the specific embodiments, and the second target image can be the converted JPEG image in the specific embodiments.

[0026] The second aspect provides an image processing method that can also call the ISP to perform secondary processing on RAW images within the gallery application interface, enabling format conversion of RAW images offline (not in photo-taking situations), meeting users' needs for image formats and improving user experience.

[0027] In one possible implementation, the image processing engine calls the ISP to perform format conversion on the first RAW image to obtain the second target image, including: the image processing engine performing a RAW domain algorithm on the first RAW image to obtain the second RAW image; the image processing engine calling the ISP to perform a RAW domain to YUV domain algorithm on the second RAW image to obtain the second YUV image; the image processing engine performing a YUV domain algorithm on the second YUV image to obtain the third YUV image; and the image processing engine calling the ISP to perform an encoding algorithm on the third YUV image to obtain the second target image.

[0028] In this implementation, while performing format conversion (RAW domain to YUV domain and encoding), the RAW domain algorithm is executed on the first RAW image, and the YUV domain algorithm is executed on the converted second YUV image to improve image quality and thus enhance user experience.

[0029] In one possible implementation, the image processing engine includes a background algorithm module and an ISP resource invocation module; the image library application sends a first RAW image to the image processing engine via an AIDL interface, including: the image library application sends the first RAW image to the background algorithm module via an AIDL interface; the image processing engine performs a RAW domain algorithm on the first RAW image to obtain a second RAW image, including: the background algorithm module performs a RAW domain algorithm on the first RAW image to obtain a second RAW image; the image processing engine performs a YUV domain algorithm on the second YUV image to obtain a third YUV image, including: the background algorithm module performs a YUV domain algorithm on the second YUV image to obtain a third YUV image.

[0030] In this implementation, the AIDL interface of the application layer enables communication between the camera application and the image processing engine of the hardware abstraction layer, thereby realizing image processing in offline situations (non-photography situations).

[0031] In one possible implementation, the image processing engine calls the ISP to perform a RAW-to-YUV domain conversion algorithm on the second RAW image to obtain a second YUV image. This includes: the background algorithm module calling the ISP resource call module to perform the RAW-to-YUV domain conversion algorithm on the second RAW image; the ISP resource call module performing data conversion and encapsulation on the second RAW image to obtain a second converted RAW image; the ISP resource call module calling the ISP to perform the RAW-to-YUV domain conversion algorithm on the second converted RAW image; the ISP performing the RAW-to-YUV domain conversion algorithm on the second converted RAW image to obtain a second original YUV image; the ISP returning the second original YUV image to the ISP resource call module; the ISP resource call module performing data conversion on the second original YUV image to obtain a second YUV image; and the ISP resource call module returning the second YUV image to the background algorithm module.

[0032] Optionally, the ISP resource invocation module may include an interface layer for communicating with the backend algorithm module. Additionally, the ISP resource invocation module can communicate with the ISP through an ISP-encapsulated interface.

[0033] In this implementation, the ISP resource call module is used to realize the conversion and transmission of input and output data for the RAW domain to YUV domain algorithm, thereby enabling the backend algorithm module to call the ISP.

[0034] In one possible implementation, the image processing engine calls the ISP to perform an encoding algorithm on the third YUV image to obtain the second target image. This includes: the background algorithm module calling the ISP resource call module to perform an encoding algorithm on the third YUV image; the ISP resource call module performing data conversion and encapsulation on the third YUV image to obtain a third YUV converted image; the ISP resource call module calling the ISP to perform an encoding algorithm on the third YUV converted image; the ISP performing an encoding algorithm on the third YUV converted image to obtain the second original target image; the ISP returning the second original target image to the ISP resource call module; the ISP resource call module performing data conversion on the second original target image to obtain the second target image; and the ISP resource call module returning the second target image to the background algorithm module.

[0035] In this implementation, the ISP resource call module is used to realize the conversion and transmission of input and output data of the encoding algorithm, thereby enabling the backend algorithm module to call the ISP.

[0036] In one possible implementation, the second target image is a JPEG or PNG image.

[0037] Thirdly, this application provides an image processing method executed by an electronic device, the electronic device including an image sensor and an image signal processor (ISP). The method includes: displaying a camera application's photo preview interface, the photo preview interface including a first thumbnail; receiving a third operation from a user, the third operation being used to instruct the capture of a frame image; in response to the third operation, acquiring a third RAW image through the image sensor; performing a first processing on the third RAW image to obtain a second thumbnail; saving the second thumbnail and refreshing the first thumbnail to the second thumbnail; performing a second processing on the third RAW image to obtain a third target image, wherein the duration of the first processing is less than the duration of the second processing, the second processing including executing a RAW-to-YUV domain conversion algorithm and an encoding algorithm through the ISP; and replacing the second thumbnail with the third target image.

[0038] The image processing method provided in the third aspect can be the method shown in Embodiment 1 of the specific embodiments. The second thumbnail can be an RGB thumbnail in the specific embodiments. The third target image can be a JPEG real image in the specific embodiments. The first processing can be the front-end algorithm flow in the specific embodiments, and the second processing can be the back-end algorithm flow in the specific embodiments.

[0039] The image processing method provided in the third aspect has a shorter processing time for the first step than for the second step. Therefore, the first step enables rapid processing of the third RAW image, allowing for quick entry into the next frame's capture process upon receiving a single-shot request, thus improving the capture response speed. Simultaneously, the second step ensures comprehensive and high-quality image processing. The second step includes executing RAW-to-YUV domain conversion and encoding algorithms through the ISP (Internet Service Provider), which can be implemented via the ISP hardware chip. This achieves asynchronous offline processing of the second step, without consuming software system resources, reducing system load, further improving capture response speed, and enhancing algorithm efficiency.

[0040] In one possible implementation, the second thumbnail is an RGB image. The first processing of the third RAW image to obtain the second thumbnail includes: performing a first RAW domain algorithm on the third RAW image to obtain a fourth RAW image; and performing a RAW domain to RGB domain conversion algorithm on the fourth RAW image to obtain the second thumbnail.

[0041] In other words, the first processing step includes a RAW domain algorithm and a RAW-to-RGB domain conversion algorithm. As can be seen, the first processing algorithm is simpler and saves time, thus quickly providing the second thumbnail and rapidly ending the single-shot request response for the current frame, improving the shooting response speed.

[0042] In one possible implementation, the number of third RAW images is one or more frames. The first RAW domain algorithm is performed on the third RAW images to obtain the fourth RAW image, which includes: performing the first RAW domain algorithm on the target RAW image to obtain the fourth RAW image, wherein the target RAW image is one frame of one or more third RAW images.

[0043] In other words, the RAW domain algorithm in the first processing is a single-input, single-output RAW domain algorithm. This further simplifies the first processing, saves time, improves the generation speed of the second thumbnail, and thus further improves the shooting response speed.

[0044] In one possible implementation, the electronic device includes an image processing engine that performs a second processing on a third RAW image to obtain a third target image. This includes: the image processing engine performing a second RAW domain algorithm on the third RAW image to obtain a fifth RAW image; the image processing engine calling an ISP to perform a RAW-to-YUV domain algorithm on the fifth RAW image to obtain a fourth YUV image; the image processing engine performing a YUV domain algorithm on the fourth YUV image to obtain a fifth YUV image; and the image processing engine calling an ISP to perform an encoding algorithm on the fifth YUV image to obtain the third target image.

[0045] In this implementation, the image processing engine calls the ISP to implement the RAW domain to YUV domain conversion algorithm and encoding algorithm. That is, it is processed offline by the ISP, which does not occupy software system resources, reduces system load, shortens the second processing time, and improves user experience.

[0046] In one possible implementation, the image processing engine includes a background algorithm module and an ISP resource call module. The image processing engine calls the ISP to perform a RAW-to-YUV domain conversion algorithm on the fifth RAW image to obtain a fourth YUV image. This includes: the background algorithm module calling the ISP resource call module to perform a RAW-to-YUV domain conversion algorithm on the fifth RAW image; the ISP resource call module performing data conversion and encapsulation on the fifth RAW image to obtain a fifth converted RAW image; the ISP resource call module calling the ISP to perform a RAW-to-YUV domain conversion algorithm on the fifth converted RAW image; the ISP performing a RAW-to-YUV domain conversion algorithm on the fifth converted RAW image to obtain a fourth original YUV image; the ISP returning the fourth original YUV image to the ISP resource call module; the ISP resource call module performing data conversion on the fourth original YUV image to obtain a fourth YUV image; and the ISP resource call module returning the fourth YUV image to the background algorithm module.

[0047] Optionally, the ISP resource invocation module may include an interface layer for communicating with the backend algorithm module. Additionally, the ISP resource invocation module can communicate with the ISP through an ISP-encapsulated interface.

[0048] In this implementation, the ISP resource call module is used to realize the conversion and transmission of input and output data for the RAW domain to YUV domain algorithm, thereby enabling the backend algorithm module to call the ISP.

[0049] In one possible implementation, the image processing engine calls the ISP to perform an encoding algorithm on the fifth YUV image to obtain the third target image. This includes: the background algorithm module calling the ISP resource call module to perform an encoding algorithm on the fifth YUV image; the ISP resource call module performing data conversion and encapsulation on the fifth YUV image to obtain a fifth YUV converted image; the ISP resource call module calling the ISP to perform an encoding algorithm on the fifth YUV converted image; the ISP performing an encoding algorithm on the fifth YUV converted image to obtain the third original target image; the ISP returning the third original target image to the ISP resource call module; the ISP resource call module performing data conversion on the third original target image to obtain the third target image; and the ISP resource call module returning the third target image to the background algorithm module.

[0050] In this implementation, the ISP resource call module is used to realize the conversion and transmission of input and output data of the encoding algorithm, thereby enabling the backend algorithm module to call the ISP.

[0051] In one possible implementation, the electronic device further includes an application layer, where a camera application is located. The application layer also includes an AIDL interface for replacing the second thumbnail with a third target image, including: an image processing engine sending the third target image to the camera application via the AIDL interface; and the camera application replacing the second thumbnail with the third target image.

[0052] In this implementation, the AIDL interface at the application layer enables communication between the camera application and the image processing engine at the hardware abstraction layer, thereby achieving image processing in offline conditions.

[0053] In one possible implementation, the number of third RAW images is one or more frames. The image processing engine performs a second RAW domain algorithm on the third RAW images to obtain a fifth RAW image. This includes: the image processing engine performs a second RAW domain algorithm on all third RAW images to obtain a fifth RAW image.

[0054] In this implementation, the RAW domain algorithm is performed on all RAW images in the second processing step, which improves the image processing effect and thus enhances the user experience.

[0055] In one possible implementation, the encoding algorithm is the JPEG encoding algorithm, and the third target image is a JPEG format image.

[0056] Fourthly, this application provides an apparatus included in an electronic device, which has the function of implementing the electronic device behavior described in the first aspect and possible implementations of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-described functions. For example, a receiving module or unit, a processing module or unit, etc.

[0057] Fifthly, this application provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform any one of the technical solutions of the first aspect, the second aspect or the third aspect.

[0058] Sixthly, this application provides a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to execute any one of the technical solutions of the first, second, or third aspects.

[0059] Optionally, the chip system may include an application processor (AP) and an ISP, etc.

[0060] In a seventh aspect, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform any one of the technical solutions of the first aspect, the second aspect, or the third aspect.

[0061] Eighthly, this application provides a computer program product comprising: computer program code, which, when executed on an electronic device, causes the electronic device to perform any one of the technical solutions of the first aspect, the second aspect, or the third aspect. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of an interface of an example image processing method provided in an embodiment of this application;

[0065] Figure 4 This is a schematic flowchart of an example image processing method provided in an embodiment of this application;

[0066] Figure 5 This is another example of a software architecture diagram provided in the embodiments of this application;

[0067] Figure 6 This is a schematic flowchart of another image processing method provided in an embodiment of this application;

[0068] Figure 7 This is a schematic diagram of an example of interface changes provided in an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of an HDR burst shooting interface in related technologies;

[0070] Figure 9 This is yet another example of a software architecture diagram provided in the embodiments of this application;

[0071] Figure 10 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0072] Figure 11 This is a schematic diagram illustrating the interface changes of an example of HDR burst shooting provided in an embodiment of this application;

[0073] Figure 12 This is another example of an interface diagram provided in the embodiments of this application;

[0074] Figure 13 This is yet another example of an interface diagram provided in the embodiments of this application;

[0075] Figure 14 This is yet another example of an interface diagram provided in the embodiments of this application;

[0076] Figure 15 This is yet another example of an interface diagram provided in the embodiments of this application;

[0077] Figure 16 This is a schematic diagram illustrating an example of interface changes for image format conversion provided in an embodiment of this application;

[0078] Figure 17 This is yet another example of a software architecture diagram provided in the embodiments of this application;

[0079] Figure 18 This is a flowchart illustrating another example of an image processing method provided in this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0081] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0082] References to "one embodiment" or "some embodiments" as described in this application specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0083] To better understand the embodiments of this application, the terms or concepts that may be involved in the embodiments are explained below.

[0084] 1. RAW chart

[0085] Images in RAW format are called RAW images. RAW images are unprocessed and uncompressed images captured by the image sensors of electronic devices (such as mobile phones, digital cameras, scanners, or film scanners). Optionally, RAW images can have file extensions such as .dng.

[0086] Generally, RAW images record the original information of the image sensor (such as shutter speed, aperture value, white balance, etc.) without missing information, so their file size is relatively large.

[0087] 2. YUV and RGB images

[0088] YUV is a color encoding method. "Y" represents luminance (or luma), which is the grayscale value; "U" and "V" represent chrominance (or chroma), used to describe the color and saturation of an image, and together they specify the color of a pixel. An image encoded using the YUV method is called a YUV image or YUV map.

[0089] Images encoded using the RGB color model are called RGB images or RGB diagrams. The RGB color model represents various colors by mixing different intensities of red, green, and blue. In an RGB image, each pixel's color is determined by the values ​​of these three color channels.

[0090] 3. JPEG and PNG images

[0091] Currently, image encoding formats include JPEG (developed by the Joint Photographic Experts Group), Portable Network Graphics (PNG), and Graphics Interchange Format (GIF).

[0092] JPEG is a lossy image compression format. Images in JPEG format are called JPEG images. JPEG images can have file extensions such as .jpg or .jpeg. JPEG images are smaller in file size compared to RAW images.

[0093] PNG format is a lossless compression format that features transparent backgrounds. Images in PNG format are called PNG images. PNG images have the .png file extension. PNG images are smaller in file size compared to RAW images.

[0094] 4. Real images and thumbnails

[0095] A true image refers to an image at its original output size (i.e., the size supported by the electronic device). It is a complete image that has not been reduced in size or cropped, possessing high resolution and clarity, and capable of displaying rich details and colors. True images are typically used when users need to view detailed content, perform editing, or print. It should be noted that in this embodiment, "true image" is not intended to limit the scope, but only to distinguish it from a thumbnail. Optionally, a true image can be a RAW image, a JPEG image, or a PNG image, etc.

[0096] Thumbnails are smaller versions of the original-size image. Their primary purpose is to allow users to quickly browse and identify the general content of an image without loading the full, high-resolution image. This improves image browsing efficiency, especially in folders, web pages, or applications containing a large number of images. Thumbnails are smaller in size and therefore have smaller file sizes.

[0097] 5. High Dynamic Range (HDR)

[0098] HDR is an image processing technique that enhances the brightness and contrast of an image to create a more realistic, vivid, and layered visual effect.

[0099] HDR images can simultaneously display details in both very bright and very dark areas, whereas traditional images often lose information in these extreme areas. HDR technology makes bright parts of an image brighter and dark parts darker while retaining more detail, making the image closer to what the human eye actually sees.

[0100] The technical problems faced by this application and the electronic devices to which the methods provided in the embodiments of this application are applicable will be described below.

[0101] Images refer to the images of objects presented through optical devices, electronic devices, etc. Photography is an important branch of imaging. Currently, photography has become an essential function of electronic devices. How to make photography faster and the quality of captured images is a major research direction for researchers.

[0102] Generally, improving image quality requires not only using better hardware but also enhancing the quality of image processing algorithms. Therefore, during the photography process, electronic devices load increasingly more complex and demanding algorithms (i.e., the system carries a heavy workload of image processing algorithms). For example, when an electronic device captures a single frame, the RAW image acquired by the image sensor undergoes algorithms such as white balance correction, noise reduction, color correction, sharpening, exposure adjustment, lens correction, format conversion, and compression to obtain the final image. However, the electronic device's system requires that photo capture requests be sent frame by frame; if the request for the previous frame is not received, the request for the next frame cannot be sent. Therefore, a heavy workload of image processing algorithms leads to decreased photography efficiency. From the user's perspective, the photo capture response speed feels slower; that is, after clicking the shutter button, the electronic device takes a long time to complete the capture of a single frame. Furthermore, in some scenarios, such as HDR photography, the excessive load of the system's image processing algorithms prevents the electronic device from performing continuous shooting (burst shooting).

[0103] The image processing method provided in this application aims to solve the above-mentioned technical problems.

[0104] The image processing method provided in this application can be applied to electronic devices with photographic functions, such as mobile phones, tablets, wearable devices, in-vehicle devices, digital cameras, drones, scanners, film scanners, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0105] For example, Figure 1This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0106] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0107] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0108] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0109] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0110] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0111] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0112] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0113] The ISP is used to process data fed back from the camera 193. For example, the ISP can perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0114] Camera 193 is used to capture still images or videos. When taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element (also called an image sensor). The photosensitive element converts the light signal into an electrical signal. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. Optionally, an analog-to-digital converter (ADC) module can be integrated into the photosensitive element. The ADC module further converts the electrical signal into a digital image signal, which is stored in a specific format and data structure to form a RAW image. The photosensitive element transmits the RAW image to an ISP and / or a DSP. The ISP and / or DSP process the RAW image and convert the processed image into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0115] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0116] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0117] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0118] The hardware system of electronic device 100 has been described in detail above. The software system of electronic device 100 is described below. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes the layered architecture Android system as an example to exemplarily describe the software system of electronic device 100.

[0119] like Figure 2 As shown, a layered architecture software system is divided into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into four layers, from top to bottom: application layer 210, application framework layer 220, hardware abstraction layer 230, and driver layer 240. Furthermore, for ease of understanding, Figure 2 The hardware layer 250 of the electronic device 100 is also shown. Each layer will be described below.

[0120] Application layer 210 may include camera and gallery applications, as well as applications such as calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS (not shown in the figure).

[0121] The camera application (or simply camera app) responds to the user's photo-taking action by sending a photo-taking request to the application framework layer. This photo-taking request is used to request the capture of an image.

[0122] Optionally, the application layer 210 may have an Android Interface Definition Language (AIDL) interface. Applications in the application layer 210 can communicate with modules in other layers through the AIDL interface. For example, camera applications and gallery applications can communicate with modules in the hardware abstraction layer through the AIDL interface.

[0123] Application framework layer 220 provides application access interfaces and programming frameworks for applications in application layer 210. For example, application framework layer 220 includes a camera access interface for providing camera shooting services through camera management and camera devices.

[0124] The camera management section in application framework layer 220 is used to manage cameras. Camera management can obtain camera parameters, such as determining the camera's working status.

[0125] The camera devices in the application framework layer 220 are used to provide data access interfaces between different camera devices and camera management.

[0126] The hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer 230 may include a camera hardware abstraction layer and other hardware device abstraction layers. In this embodiment, the camera hardware abstraction layer may include a front-end image output module and a back-end processing module. The front-end image output module is used to send shooting parameters to the lower layer to trigger the image sensor in the camera to acquire RAW images. The back-end processing module is used to process the RAW images acquired by the image sensor. Optionally, the camera hardware abstraction layer can be connected to a camera algorithm library, and each module in the camera hardware abstraction layer can call the algorithm modules in the camera algorithm library.

[0127] In this embodiment, the camera algorithm library includes an image processing engine. The image processing engine may include a front-end algorithm module (or front-end capture module), a back-end algorithm module (or back-end capture module), and an ISP resource call module. The front-end algorithm module processes RAW images based on the front-end algorithm. The back-end algorithm module processes RAW images based on the back-end algorithm. The back-end algorithm module executes the back-end algorithm on the RAW image based on the back-end algorithm. It should be understood that, in this embodiment, for a given frame, the front-end algorithm can be understood as the processing algorithm for that frame executed during the capture request response process. The back-end algorithm can be understood as the processing algorithm for that frame executed after the capture request response process. The capture request response process for a given frame refers to the process of acquiring and processing an image in response to a capture request sent by the camera application, and initially applying the relevant data (e.g., thumbnail, JPEG image, etc.) of that frame to the camera. In other words, the time period involved in the photo request response process is the time between the moment the camera application sends a request to capture a frame of an image and the moment the relevant data for that frame of an image is first fed back to the camera application.

[0128] The front-end and back-end algorithm modules can call the ISP resource call module to process images through the hardware-layer ISP. Optionally, the ISP resource call module is used to implement data conversion and matching between the front-end or back-end algorithm modules and the ISP, so that the data input to the ISP can be recognized by it, and the data fed back to the front-end or back-end algorithm modules can be recognized by it. Specifically, the ISP resource call module can receive calls from the front-end or back-end algorithm modules, convert and encapsulate the image data, and then input it into the ISP. The ISP returns the processed image data to the ISP resource call module. The ISP resource call module then converts the processed image data and returns it to the front-end or back-end algorithm modules.

[0129] Optionally, the hardware abstraction layer may also include an ISP encapsulation interface. The ISP encapsulation interface is used to enable communication between the ISP resource access module and the ISP hardware.

[0130] Optionally, the algorithms included in the front-end and back-end algorithm modules can be loaded according to the actual shooting scene, shooting mode, etc. For details, please refer to [link / reference]. Figure 2 In one embodiment, the image processing engine further includes a decision module. The decision module determines the shooting parameters and algorithm type based on the shooting mode, configuration parameters, and environmental parameters in the shooting request. The shooting parameters may include the frame sequence, exposure parameters, and image size. The algorithm type may include a front-end algorithm type and a back-end algorithm type. The decision module can send the front-end algorithm type to the front-end algorithm module, which loads the corresponding front-end algorithm based on the front-end algorithm type. The decision module can also send the back-end algorithm type to the back-end algorithm module, which loads the corresponding back-end algorithm based on the back-end algorithm type.

[0131] Optionally, the image processed by the front-end algorithm module can be returned to the back-end processing module, which then passes it to the camera application in the application layer 210 via the camera access interface. The image processed by the back-end algorithm module can be passed to the camera application and / or the gallery application via the AIDL interface.

[0132] It should be noted that the names of modules, data, or operations in the embodiments of this application are for distinction only and not for limitation. For example, the front-end algorithm module and the back-end algorithm module are used to distinguish the two modules, and are not used to specifically limit whether the module runs in the front-end or the back-end.

[0133] Driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include camera drivers and ISP drivers.

[0134] Hardware layer 250 may include a camera, an ISP, and other hardware devices. The camera includes an image sensor (i.e., a photosensitive element). There may be one or more cameras. The camera may be a wide-angle camera, a telephoto camera, a time-of-flight (TOF) depth sensor, or a multispectral camera, etc., and this embodiment does not limit the specific type of camera.

[0135] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 1 and Figure 2 Taking the electronic device with the structure shown as an example, the image processing method provided in the embodiments of this application will be specifically described in conjunction with the accompanying drawings and application scenarios.

[0136] First, the methods of taking pictures with electronic devices can be divided into single-shot (single-shot) and continuous-shot (burst-shot). Single-shot refers to the electronic device responding to the user's single-shot operation by capturing only one frame of image. Burst-shot refers to the electronic device responding to the user's continuous-shot operation by capturing multiple frames of image. Optionally, the single-shot operation can be clicking the camera control (also called the first control) in the photo preview interface or pressing the power button (i.e., the power button); when the smile capture function is enabled, the single-shot operation can also be making a smiling expression facing the camera; when the gesture photography function is enabled, the single-shot operation can also be performing a preset gesture (such as a gesture of the palm facing the phone camera); when the voice-controlled photography function is enabled, the single-shot operation can also be voice input of a preset photography keyword (such as "take a picture"). The burst-shot operation can be long-pressing the camera control in the photo preview interface, pressing the camera control and dragging it to the right, or pressing the volume button, etc. This application embodiment does not specifically limit the single-shot operation and burst-shot operation. The following mainly uses clicking the camera control as the single-shot operation and long-pressing the camera control as the burst-shot operation as an example for explanation.

[0137] This application relates to image processing methods for single-shot scenes and HDR continuous shooting scenes, which will be described below in conjunction with the accompanying drawings.

[0138] Example 1: Single shot scene.

[0139] Figure 3 This is a schematic diagram of an image processing method provided in an embodiment of this application. Taking a mobile phone as an example, as shown... Figure 3 As shown in Figure (a), the phone displays desktop 301, which includes an icon 3011 for the camera application. In response to a user's selection of the camera application icon 3011 (e.g., a click), the phone launches the camera application and enters default photo mode, displaying... Figure 3 Figure (b) shows the photo preview interface 302. The photo preview interface 302 includes the image captured and processed by the mobile phone. The photo preview interface also includes a photo control 3021 and an image viewing control 3022. The image viewing control 3022 includes a thumbnail of the image previously captured by the mobile phone.

[0140] In response to the user's click operation on the camera control 3021, the phone captures a RAW image frame and performs a series of processing steps on the RAW image to obtain a JPEG or PNG image (in this embodiment, a JPEG image is used as an example for explanation), and saves the JPEG image to the gallery application. Simultaneously, the phone refreshes the interface, displaying a thumbnail of the captured JPEG image at the image viewing control 3022, such as... Figure 3 As shown in Figure (c).

[0141] As described in the above embodiments, during the processing of RAW images, the electronic device carries a heavy-load image-taking algorithm, resulting in slow processing speed and thus hindering the processing of RAW images. Figure 3 Figure (b) in the middle Figure 3 The process in diagram (c) takes a long time and results in a poor user experience.

[0142] To address this, this embodiment provides an image processing method that, after acquiring a RAW image, processes it through a front-end algorithm flow and a back-end algorithm flow. The front-end algorithm flow generates a thumbnail, while the back-end algorithm flow generates a JPEG image. The front-end algorithm flow is simpler than the back-end algorithm flow. Thus, after the front-end algorithm flow quickly processes the RAW image and generates a thumbnail, it can first return the thumbnail to the camera application, completing the response to the current frame's image capture request. Upon receiving the thumbnail, the camera application can then send a capture request for the next frame, and the system executes the capture of the next frame. Simultaneously, the back-end algorithm flow can continue until the JPEG image is generated. In other words, the method provided in this embodiment quickly concludes the current frame's image capture process through a simpler front-end algorithm flow, while the more complex back-end algorithm flow can be performed asynchronously offline. This allows the electronic device to process the next frame's image capture request as early as possible, thereby improving capture response speed, capturing efficiency, and ultimately, the user experience.

[0143] The method of this application embodiment will now be described with reference to the flowchart.

[0144] For example, Figure 4 A flowchart illustrating an example image processing method provided in this application embodiment is shown below. Figure 4 As shown, the method includes:

[0145] S101. In response to the user's click operation on the camera control in shooting mode, the camera application in the application layer sends a single-shot request to the camera access interface in the application framework layer.

[0146] Specifically, the camera application generates different shooting requests based on different shooting modes and user operations. Shooting modes can include large aperture mode, portrait mode, still image mode, video mode, night scene mode, and professional mode. Shooting requests, also known as capture requests, can include still image capture requests (also called video capture requests), video capture requests, and preview requests. Still image capture requests include single-shot requests and burst capture requests. For example, if the shooting mode is determined to be video mode and the user's touch operation is a click on the video control, the camera application generates a video capture request. As another example, if the shooting mode is determined to be non-video mode and the user's operation is a long press on the still image control, the camera application generates a burst capture request. In this embodiment, the camera application determines the current shooting mode is still image mode, and the user's operation is a single click on the still image control; therefore, the camera application generates a single-shot request.

[0147] A single-shot request is used to request the capture of a single frame. Optionally, the single-shot request may include the shooting mode, configuration parameters, and frame image information. The shooting mode can be determined based on the user's current selection. For example, Figure 3 In the image, the user selects "Photo Mode". Configuration parameters refer to the shooting-related settings configured by the user, including resolution, image aspect ratio, whether to add a watermark, whether to record geolocation information, and whether to add filters.

[0148] The frame image information refers to the frame image requested to be captured in the current capture request. Optionally, the frame image information may include a frame number, image name, etc. The frame number is used to represent the number of the frame image currently requested to be captured. Optionally, the frame number can be generated based on a preset numbering rule or randomly generated. The image name is the name of the frame image currently requested to be captured. If the frame image is in JPEG format, the image name is also called the JPEG name. Optionally, the image name can be generated based on the current time information and a preset numbering rule. For example, if the current time is 15:08:12 on July 18, 2024, then the image name can be IMG_20240718_150812_001. Here, "001" is a number generated based on a preset numbering rule.

[0149] In one specific embodiment, a capture request can be defined using the CaptureRequest class. The CaptureRequest class can include a CONTROL_CAPTURE_INTENT parameter, which defines the request type. When the CONTROL_CAPTURE_INTENT parameter is set to VIDEO_CAPTURE, it indicates a video recording request. When the CONTROL_CAPTURE_INTENT parameter is set to PREVIEW_CAPTURE, it indicates a preview request. When the CONTROL_CAPTURE_INTENT parameter is set to STILL_CAPTURE, it indicates a capture request. Additionally, if the capture request is a burst request, the CaptureRequest class can also include a burst identifier. The burst identifier indicates that the capture request is a burst request. A burst request is, for example, BURST MOOD. In this embodiment, the capture request is a single-shot request, the CONTROL_CAPTURE_INTENT parameter can be set to STILL_CAPTURE, and the CaptureRequest class does not include a burst identifier.

[0150] CaptureRequest can also include the FRAME_NUMBER and JPEG_NAME parameters. The FRAME_NUMBER parameter is used to define the frame number, and the JPEG_NAME parameter is used to define the image name.

[0151] It should be noted that this step is illustrated using a single-shot operation performed by the user in photo mode. In reality, the camera app also generates a single-shot request when the user performs a single-shot operation in other shooting modes besides video mode. The difference lies in the shooting mode, configuration parameters, etc., carried in the generated single-shot request.

[0152] Optionally, after generating a single-shot request, the camera application can send the single-shot request to the camera device in the camera access interface.

[0153] S102, the camera access interface of the frame layer sends the single-shot request to the front-end image output module in the camera hardware abstraction layer.

[0154] S103, The front-end image output module calls the decision module in the image processing engine to make a photo-taking decision.

[0155] Optionally, when the front-end output module calls the decision module, it can carry a single-shot request, including various parameters carried in the single-shot request.

[0156] S104. The decision module responds to the call from the front-end image output module and obtains the shooting parameters 1, front-end algorithm type 1, and back-end algorithm type 1 based on the single-shot request.

[0157] Specifically, the decision-making module can make a shooting decision based on the shooting mode and configuration parameters carried in the single-shot request, combined with the current environmental parameters. The environmental parameters are relevant parameters of the current shooting environment, such as ambient light intensity and light source type.

[0158] The decision-making module determines the shooting strategy, which may include shooting parameters and algorithm type. Shooting parameters may include frame sequence, exposure parameters, and image size. Frame sequence refers to the sequence information of x RAW images needed to capture the current frame. x can be 1 or an integer greater than 1. For example, in some shooting modes, to obtain a good frame, multiple RAW images can be captured consecutively, and the final frame image is obtained after processing these multiple RAW images; in this case, x is an integer greater than 1. Exposure parameters may include exposure time and exposure value. Image size refers to the number of pixels in the horizontal and vertical directions of the RAW image.

[0159] Algorithm type refers to the type of algorithm required to process the current RAW image frame. It can be understood that different shooting modes, configuration parameters, and environmental parameters will necessitate different algorithm types. In this embodiment, the algorithm includes a foreground algorithm and a background algorithm; correspondingly, the algorithm type can include a foreground algorithm type and a background algorithm type. The decision module can determine the type of foreground algorithm and the type of background algorithm required for processing the current frame image based on the shooting mode, configuration parameters carried in the single-shot request, and the current environmental parameters. The foreground algorithm is less complex than the background algorithm. Lower algorithm complexity means fewer algorithm types and / or fewer processing steps. In other words, the time required to execute the foreground algorithm is less than the time required to execute the background algorithm.

[0160] In one specific embodiment, the foreground algorithm is used to process the RAW image to generate a thumbnail. Optionally, the thumbnail can be an RGB thumbnail. Optionally, the foreground algorithm may include a RAW domain algorithm and a RAW-to-RGB domain algorithm (referred to as the RAW2RGB algorithm). The RAW domain algorithm refers to an algorithm that processes the RAW image. The image obtained after processing the RAW image using the RAW domain algorithm remains a RAW image. The RAW domain algorithm may include, but is not limited to, black level compensation algorithms, lens correction algorithms, noise reduction algorithms, and automatic white balance algorithms. The RAW2RGB algorithm is used to convert the RAW image to an RGB image. For ease of distinction, the RAW domain algorithm in the foreground algorithm is referred to as the foreground RAW domain algorithm.

[0161] The background algorithm is used to process RAW images to generate a true image, which can optionally be a JPEG true image. Optionally, the background algorithm may include RAW domain algorithms, RAW-to-YUV domain algorithms (abbreviated as RAW2YUV algorithms), YUV domain algorithms, and JPEG encoding algorithms. For ease of distinction, the RAW domain algorithm in the background algorithm is referred to as the background RAW domain algorithm. The RAW2YUV algorithm is used to convert RAW images to YUV images. YUV domain algorithms refer to algorithms that process YUV images. The image obtained after processing a YUV image using a YUV domain algorithm remains a YUV image. YUV domain algorithms may include, but are not limited to, luminance denoising algorithms, color denoising algorithms, edge enhancement algorithms, hue / saturation control algorithms, and contrast / luminance control algorithms. The JPEG encoding algorithm is used to perform JPEG encoding on the YUV image to generate a JPEG image.

[0162] In this embodiment, the algorithm type in the foreground RAW domain algorithm can be the same as or different from the algorithm type in the background RAW domain algorithm. As one possible implementation, the number of algorithm types in the foreground RAW domain algorithm can be fewer than the number of algorithm types in the background RAW domain algorithm; and / or, the processing flow of the foreground RAW domain algorithm can be less than that of the background RAW domain algorithm. For example, when a frame sequence indicates the capture of multiple RAW frames, the foreground RAW domain algorithm can be used to process only one frame (e.g., a reference frame) in the RAW domain, while the background RAW domain algorithm can be used to process all RAW frames. That is, the foreground RAW domain algorithm can be a single-input single-output algorithm, while the background RAW domain algorithm can be a multiple-input multiple-output (MIMO) or multiple-input single-output (MIMO) algorithm.

[0163] In summary, the number of front-end algorithm types is less than the number of back-end algorithm types, and / or the processing flow of front-end algorithms is less than that of back-end RAW domain algorithms, thus making the execution time of front-end algorithms shorter than that of back-end algorithms.

[0164] In addition, to facilitate differentiation from the parameters in subsequent embodiments, in this embodiment, the shooting parameters obtained based on the single-shot request decision in step S101 are denoted as shooting parameters 1, the foreground algorithm type obtained from the decision is denoted as foreground algorithm type 1, and the background algorithm type obtained from the decision is denoted as background algorithm type 1. The algorithm corresponding to foreground algorithm type 1 is denoted as foreground algorithm 1, and the algorithm corresponding to background algorithm type 1 is denoted as background algorithm 1.

[0165] S105, The decision module sends front-end algorithm type 1 to the front-end algorithm module in the image processing engine.

[0166] S106. The front-end algorithm module loads front-end algorithm 1 according to the front-end algorithm type.

[0167] For example, Figure 5 This is another example of a software architecture diagram provided for an embodiment of this application. For example... Figure 5 As shown, after loading front-end algorithm 1, the front-end algorithm module can include a front-end RAW domain algorithm unit and a RAW2RGB algorithm unit. The front-end RAW domain algorithm unit is used to execute the front-end RAW domain algorithm. The RAW2RGB algorithm unit is used to execute the RAW2RGB algorithm.

[0168] S107. The decision module sends background algorithm type 1 to the background algorithm module in the image processing engine.

[0169] S108. The background algorithm module loads background algorithm 1 according to the background algorithm type.

[0170] See also Figure 5 After loading background algorithm 1, the background algorithm module can include a background RAW field algorithm unit, a RAW2YUV algorithm unit, a YUV field algorithm unit, and a JPEG encoding algorithm unit. The background RAW field algorithm unit executes the background RAW field algorithm. The RAW2YUV algorithm unit executes the RAW2YUV algorithm. The YUV field algorithm unit executes the YUV field algorithm. The JPEG encoding algorithm unit executes the JPEG encoding algorithm.

[0171] S109. The decision module sends the photo parameter 1 to the front-end image output module.

[0172] S110, the front-end image output module sends the image capture parameter 1 to the camera in the hardware layer.

[0173] Specifically, the front-end image output module can send image capture parameter 1 to the camera driver in the hardware driver layer, and the camera driver will then send image capture parameter 1 to the camera. Furthermore, in electronic devices that include multiple cameras, image capture parameter 1 can further include camera information, and the front-end image output module will send image capture parameter 1 to the corresponding camera based on this information.

[0174] S111, The camera captures a RAW image (also known as a third RAW image) according to the shooting parameter 1.

[0175] Specifically, the camera controls the image sensor to capture RAW images based on the shooting parameters 1. For example, the camera can adjust the exposure based on the exposure parameters in shooting parameters 1, and control the number of pixels in the RAW image based on the image size in the shooting parameters. In addition, the camera controls the number of RAW images captured based on the frame sequence.

[0176] S112, The camera returns the RAW image to the front-end output module.

[0177] S113. The front-end output module sends the RAW image to the back-end processing module.

[0178] After receiving the RAW image from the front-end image output module, the back-end processing module can call the front-end algorithm module to execute the front-end algorithm process, and call the back-end algorithm module to execute the back-end algorithm process. The front-end and back-end algorithm processes can be executed sequentially or simultaneously; this application does not impose any restrictions on this. These will be explained separately below.

[0179] 1) Front-end algorithm flow (also known as first processing).

[0180] See also Figure 4 The front-end algorithm process may include:

[0181] S114. The backend processing module calls the frontend algorithm module to execute frontend algorithm 1.

[0182] Optionally, when the back-end processing module calls the front-end algorithm module to execute front-end algorithm 1, it can carry a RAW image.

[0183] S115. The front-end algorithm module responds to the call of the back-end processing module by executing front-end algorithm 1 to generate an RGB thumbnail (also known as the second thumbnail).

[0184] For more details, please refer to [link / reference]. Figure 5 The front-end algorithm module can perform front-end RAW domain algorithms on the RAW graph through the front-end RAW domain algorithm unit to obtain the processed RAW. Figure 1 The front-end RAW domain algorithm unit will process the RAW data. Figure 1 Input to the RAW2RGB algorithm unit. The RAW2RGB algorithm unit processes the RAW... Figure 1 Execute the RAW2RGB algorithm to obtain RGB thumbnails.

[0185] S116. The front-end algorithm module returns the RGB thumbnail to the back-end processing module.

[0186] S117. The backend processing module sends the RGB thumbnail to the camera access interface.

[0187] S118, The camera access interface sends RGB thumbnails to the camera application.

[0188] S119: The camera application refreshes the photo preview interface based on the RGB thumbnail.

[0189] Specifically, the camera app can replace the thumbnail (also known as the first thumbnail) in the image viewing control of the photo preview interface with the generated RGB thumbnail (i.e., the second thumbnail).

[0190] S120, the camera app saves RGB thumbnails to the gallery.

[0191] It can be understood that the image name of the RGB thumbnail generated by the above front-end algorithm process is the same as the image name carried in the single-shot request in step S101. The camera application can send the RGB thumbnail to the gallery application, and the gallery application will save the RGB thumbnail to a preset storage path.

[0192] 2) Backend algorithm process (also known as second processing).

[0193] For example, Figure 6 Please refer to the schematic flowchart of another image processing method provided in this application embodiment. Figure 5 and Figure 6 The background algorithm process may include:

[0194] S121. The backend processing module calls the backend algorithm module to execute backend algorithm 1.

[0195] Optionally, when the backend processing module calls the backend algorithm flow to execute backend algorithm 1, it can carry a RAW graph.

[0196] S122. The background algorithm module responds to the call from the backend processing module by performing the background RAW domain algorithm on the RAW graph to obtain the processed RAW. Figure 2 .

[0197] Please see also Figure 5 The background RAW domain algorithm unit in the background algorithm module performs the background RAW domain algorithm on the RAW graph to obtain the processed RAW. Figure 2 .

[0198] S123, The background algorithm module calls the ISP resource call module to execute the RAW2YUV algorithm.

[0199] Optionally, when the background algorithm module calls the ISP resource call module to execute the RAW2YUV algorithm, it can carry the processed RAW file. Figure 2 .

[0200] S124. The ISP resource call module responds to the call from the background algorithm module by calling the hardware layer ISP to execute the RAW2YUV algorithm.

[0201] S125, In response to the call from the ISP resource retrieval module, the ISP processes the RAW file. Figure 2 The RAW2YUV algorithm is executed to obtain the YUV image.

[0202] S126. The ISP returns the YUV diagram to the ISP resource retrieval module.

[0203] S127, the ISP resource call module returns the YUV image to the background algorithm module.

[0204] Specifically, such as Figure 5 As shown, the ISP resource retrieval module may include an interface layer, a RAW2YUV conversion unit, and a JPEG encoding conversion unit. The interface layer is used for communication with various units in the backend algorithm module. The RAW2YUV conversion unit converts the input and output data of the RAW2YUV algorithm. The JPEG encoding conversion unit converts the input and output data of the JPEG encoding algorithm. It should be noted that, depending on the algorithm required to interface with the ISP resource retrieval module, the ISP resource retrieval module may also include other units, such as an HDR fusion conversion unit and a RAW2YUV+JPEG conversion unit. The HDR fusion conversion unit converts the input and output data of the HDR fusion algorithm. The RAW2YUV+JPEG conversion unit converts the input and output data of the RAW2YUV+JPEG algorithm. Specifically, the RAW2YUV+JPEG algorithm converts the RAW image to a YUV image and then performs JPEG encoding on the YUV image to generate a JPEG image.

[0205] Combination Figure 5 The specific execution process of steps S123 to S127 may include:

[0206] ① The RAW2YUV algorithm unit in the background algorithm module sends an instruction to invoke the RAW2YUV algorithm to the interface layer of the ISP resource call module.

[0207] ②After receiving the call instruction from the backend algorithm module through the interface layer, the ISP resource retrieval module responds to the call instruction by converting the processed RAW to YUV data through the RAW2YUV conversion unit. Figure 2 The data is converted to a format that the ISP platform can recognize and then encapsulated. The resulting data is called a platform-formatted RAW image.

[0208] ③ The RAW2YUV conversion unit sends the RAW image in platform format to the ISP driver through the ISP encapsulation interface.

[0209] ④ The ISP driver sends the platform-formatted RAW image to the hardware-layer ISP.

[0210] ⑤ The ISP performs the RAW2YUV algorithm on the platform-formatted RAW image to obtain the platform-formatted YUV image.

[0211] ⑥ The ISP returns the platform-formatted YUV diagram to the RAW2YUV conversion unit in the ISP resource call module through the ISP driver and ISP encapsulation interface.

[0212] ⑦ The RAW2YUV conversion unit converts the YUV image in the platform format to obtain a YUV image that the system can recognize.

[0213] ⑧ The RAW2YUV conversion unit returns the YUV image to the RAW2YUV algorithm unit in the background algorithm module through the interface layer.

[0214] Then, the RAW2YUV algorithm unit inputs the YUV image into the YUV domain algorithm unit and executes the YUV domain algorithm, that is, it executes the following step S128.

[0215] S128. The background algorithm module performs the YUV domain algorithm on the YUV image to obtain the processed YUV image.

[0216] Afterwards, the YUV domain algorithm unit inputs the processed YUV image into the JPEG encoding algorithm unit to execute the JPEG encoding algorithm, as detailed in S129 to S133 below.

[0217] S129. The background algorithm module calls the ISP resource call module to execute the JPEG encoding algorithm.

[0218] Optionally, when the background algorithm module calls the ISP resource call module to execute the JPEG encoding algorithm, it can carry the processed YUV image.

[0219] S130, the ISP resource call module responds to the call from the background algorithm module by calling the hardware layer ISP to execute the JPEG encoding algorithm.

[0220] S131. In response to the call from the ISP resource call module, the ISP performs the JPEG encoding algorithm on the processed YUV image to obtain a JPEG true image (also known as a third target image).

[0221] S132, the ISP returns the JPEG image to the ISP resource retrieval module.

[0222] S133, the ISP resource retrieval module returns the JPEG image to the background algorithm module.

[0223] Combination Figure 5 The specific execution process of steps S129 to S133 may include:

[0224] ① The JPEG encoding algorithm unit in the background algorithm module sends an instruction to the interface layer of the ISP resource call module to call the JPEG encoding algorithm.

[0225] ②After receiving the call instruction from the backend algorithm module through the interface layer, the ISP resource call module responds to the call instruction by converting the processed YUV image into a format that the ISP platform can recognize through the JPEG encoding conversion unit, and then encapsulates the data. The resulting data is called the platform format YUV image.

[0226] ③ The JPEG encoding conversion unit sends the platform-formatted YUV image to the ISP driver through the ISP encapsulation interface.

[0227] ④ The ISP driver sends the platform-formatted YUV diagram to the hardware-layer ISP.

[0228] ⑤ The ISP performs JPEG encoding on the YUV image in the platform format to obtain a JPEG image in the platform format.

[0229] ⑥ The ISP returns the JPEG image in platform format to the JPEG encoding conversion unit in the ISP resource call module through the ISP driver and ISP encapsulation interface.

[0230] ⑦ The JPEG encoding conversion unit converts the JPEG image in the platform format to obtain a true JPEG image that the system can recognize.

[0231] ⑧ The JPEG encoding conversion unit returns the JPEG image to the JPEG encoding algorithm unit in the background algorithm module through the interface layer.

[0232] Then, perform the following step S134.

[0233] S134. The background algorithm module sends the JPEG real image to the AIDL interface of the application layer.

[0234] The S135 and AIDL interfaces send JPEG images to the camera application.

[0235] S136. The camera application replaces RGB thumbnails in the gallery with JPEG real images.

[0236] It is understood that the image name of the JPEG image generated by the aforementioned background algorithm process is also the same as the image name carried in the single-shot request in step S101. In other words, the image name of the JPEG image generated by the background algorithm process is the same as the image name of the RGB thumbnail generated by the front-end algorithm process. The camera application can send the JPEG image to the gallery application, which then locates the storage path of the RGB thumbnail with the same image name. Afterward, the gallery application replaces the RGB thumbnail in that storage path with the JPEG image.

[0237] In this embodiment, a foreground algorithm process enables rapid processing of RAW images, allowing for quick entry into the next frame capture process upon receiving a single-shot request for the next frame, thus improving capture response speed. Simultaneously, a background algorithm process ensures comprehensive and high-quality image processing. This background algorithm process can be implemented by calling the ISP hardware chip, thus achieving asynchronous offline processing without consuming software system resources, reducing system load, further improving capture response speed, and enhancing algorithm efficiency.

[0238] It's understandable that after the front-end algorithm process completes, the back-end algorithm process may still be in progress. Therefore, when a user views a large image, the electronic device can display a message indicating that the image is being processed, promptly informing the user and improving the user experience. Once the back-end algorithm process is complete, the processed JPEG image is then displayed, improving the image display quality and further enhancing the user experience. The following section combines... Figure 7 Please provide an explanation.

[0239] For example, Figure 7 This is a schematic diagram illustrating an example of interface changes provided in an embodiment of this application. For example... Figure 7 As shown in Figure (a), the mobile phone displays a camera preview interface 701. The user triggers the above steps S101 to S136 by clicking the camera control 3021. After the foreground algorithm process shown in steps S114 to S120 is completed, the mobile phone refreshes interface 701, resulting in a camera preview interface 702, as shown... Figure 7 As shown in Figure (b), it can be seen that the thumbnail in the image viewing control 3022 of the photo preview interface 702 has been replaced with the thumbnail of the current frame image (RGB thumbnail).

[0240] In response to a user's click on the image viewing control 3022, if the background algorithm process for the current image frame has not been completed, the phone can display as follows: Figure 7 The prompt interface 703 shown in Figure (c) is an example. The prompt interface 703 may include the message "Image processing in progress, please wait...". After the background algorithm process is complete, the interface can display the processed JPEG image, such as... Figure 7 Interface 704 is shown in Figure (d).

[0241] Example 2: HDR burst shooting scene.

[0242] During photography, for HDR scenes with strong contrast and requiring rich colors and detail, HDR technology can achieve better shooting results. Examples of HDR scenes include: shooting brightly lit openings, windows, lights, or scenes of sunrise and sunset.

[0243] Optionally, during HDR shooting, images can be generated using a long / short frame method. For example, single-frame high dynamic range (SHDR), dual analog gain (DAG), or dual conversion gain (DCG) methods can be used. When using this method, two RAW images need to be generated within one frame: one long RAW image and one short RAW image. The long RAW image refers to a long-exposure RAW image, i.e., an overexposed image; the short RAW image refers to a short-exposure RAW image, i.e., an underexposed image. The two RAW images are then fused to obtain the HDR image. The long / short frame RAW images can also be represented as L / S images.

[0244] However, the fusion of long and short RAW frames is typically achieved through modules in the camera's algorithm library, i.e., through the software system of the electronic device. These fusion algorithms are generally complex, resulting in a heavy system load and slow processing speed. Therefore, related technologies cannot achieve continuous shooting in HDR scenes (referred to as HDR burst shooting).

[0245] For example, Figure 8 This is a schematic diagram of an example HDR burst shooting interface in related technologies. For example... Figure 8 As shown in Figure (a), the mobile phone displays a camera preview interface 801. The camera preview interface 801 includes an HDR switch control 8011. In response to a user's selection operation (e.g., a click) on the HDR switch control 8011, the mobile phone activates the HDR function and displays the image shown. Figure 8 Figure (b) shows the photo preview interface 802. It can be seen that the HDR switch control 8011 in the photo preview interface 802 is turned on. Additionally, the interface can display a prompt 8021 indicating that the HDR function is enabled, such as "Auto HDR mode is enabled." Afterwards, the user points the camera at a strong light source for continuous shooting. In response to the user long-pressing the photo control 3021 in photo mode, the phone recognizes the HDR scene and the user's long-press operation, and can display a prompt indicating that HDR continuous shooting is not possible, such as... Figure 8 The prompt 8031 ​​in interface 803 of Figure (c) is shown. Optionally, the phone can treat the user's long press on the camera control 3021 as a single-shot operation, initiate HDR shooting, obtain a frame of HDR image, and update the thumbnail at the image viewing control 3022, as shown. Figure 8 As shown in (c) in the figure.

[0246] The inability to perform continuous shooting in HDR scenes causes inconvenience to users. Therefore, in this embodiment, the ISP is asynchronously invoked through the front-end algorithm module and the ISP resource call module to achieve the fusion of long and short frame RAW images and subsequent processing. ISP processing can quickly perform image processing, reduce system load, and shorten the processing time of one HDR image frame, thus enabling HDR continuous shooting and improving the user experience.

[0247] For example, Figure 9 This is another example of a software architecture diagram provided in the embodiments of this application, such as... Figure 9 As shown in this embodiment, the image processing engine may include a front-end algorithm module and an ISP resource retrieval module. The front-end algorithm module may include an HDR fusion algorithm unit, a RAW2YUV algorithm unit, and a JPEG encoding algorithm unit. The ISP resource retrieval module may include an HDR fusion conversion unit, a RAW2YUV conversion unit, and a JPEG encoding conversion unit. The HDR fusion algorithm unit performs the HDR fusion algorithm on long and short frame RAW images. The HDR fusion conversion unit converts the input and output data of the HDR fusion algorithm. The functions of the RAW2YUV algorithm unit, JPEG encoding algorithm unit, RAW2YUV conversion unit, and JPEG encoding conversion unit are similar to those described in Embodiment 1 above, and will not be repeated here.

[0248] For example, Figure 10 This is a schematic flowchart illustrating yet another example of an image processing method provided in this application. Please refer to [the original text here]. Figure 9 and Figure 10 In this embodiment, the image processing method includes:

[0249] S201. When the HDR function is enabled, the camera application at the application layer responds to the user's operation of long-pressing the shutter control in shooting mode by sending a burst shooting request to the camera access interface at the application framework layer.

[0250] Specifically, the camera app determines that it is currently in photo mode and that the user has performed a long press on the camera control, thus generating a burst request. The burst request is used to request the continuous capture of multiple frames. The CONTROL_CAPTURE_INTENT parameter in the CaptureRequest of the burst request is set to STILL_CAPTURE, and the CaptureRequest includes a burst identifier (e.g., BURST MOOD).

[0251] Optionally, the burst shooting request may include shooting mode, configuration parameters, and starting frame image information. In this embodiment, the shooting mode is "photo mode," and the configuration parameters include an HDR identifier. The HDR identifier indicates that the HDR switch is on. The starting frame image information refers to the information of the first frame image among the multiple frames requested by the burst shooting request. The starting frame image information may include the frame number and name of the starting frame image.

[0252] Similar to Example 1, in shooting modes other than video recording, when the user performs a long press, the camera application can also generate a burst shooting request. The difference is that the shooting mode and configuration parameters carried in the generated burst shooting request can be different.

[0253] S202. The camera access interface of the application framework layer responds to the continuous shooting request, generates a single shooting request 1 based on the continuous shooting request, and sends the single shooting request 1 to the front-end image output module in the camera hardware abstraction layer.

[0254] Specifically, after receiving a burst shooting request, the camera access interface can send a single-shot request to the front-end image output module at preset intervals until the user raises their hand. Each single-shot request requests to capture one frame of image. Optionally, the preset interval can be equal to the frame interval of the electronic device (i.e., 1 / frame rate). Similar to the single-shot request in Embodiment 1, each single-shot request in this embodiment can also carry shooting mode, configuration parameters, and frame image information. Unlike Embodiment 1, each single-shot request in this embodiment also carries a burst shooting identifier.

[0255] For ease of distinction, the first single-shot request sent by the camera access interface is denoted as single-shot request 1, the second single-shot request as single-shot request 2, and so on. Among them, single-shot request 1 is the first single-shot request sent in response to the continuous shooting request, and the frame image information it carries is the starting frame image information in the continuous shooting request.

[0256] S203. The front-end image output module responds to single-shot request 1 by calling the decision module in the image processing engine to make a shooting decision.

[0257] S204. The decision module determines the shooting parameters 2 and the front-end algorithm type 2 based on the single-shot request 1.

[0258] It should be understood that after receiving the first single-shot request 1, the decision module can determine that the current scene is an HDR scene based on the HDR identifier in the single-shot request 1, and can determine that the current scene is a continuous shooting scene based on the continuous shooting identifier in the single-shot request 1, thus determining that the current scene is an HDR continuous shooting scene. Based on this, the decision module can decide to generate the corresponding shooting parameters 2 for the HDR continuous shooting scene, and decide the corresponding algorithm type for the HDR continuous shooting scene. Among them, the shooting parameters 2 include the HDR image output method. In this embodiment, the SHDR image output method is used as an example for explanation. The shooting parameters 2 may include an SHDR identifier. The SHDR identifier is used to indicate that the image is output using the SHDR method.

[0259] In this embodiment, the algorithm type corresponding to the HDR burst shooting scene is denoted as front-end algorithm type 2, and the algorithm corresponding to front-end algorithm type 2 is denoted as front-end algorithm 2. Optionally, front-end algorithm 2 may include an HDR fusion algorithm, a RAW2YUV algorithm, and a JPEG encoding algorithm.

[0260] S205, The decision module sends front-end algorithm type 2 to the front-end algorithm module.

[0261] S206. The front-end algorithm module loads front-end algorithm 2 according to front-end algorithm type 2.

[0262] like Figure 9 As shown, after loading front-end algorithm 2, the front-end algorithm module can include an HDR fusion algorithm unit, a RAW2YUV algorithm unit, and a JPEG encoding algorithm unit. The HDR fusion algorithm unit is used to execute the HDR fusion algorithm. The RAW2YUV algorithm unit is used to execute the RAW2YUV algorithm. The JPEG encoding algorithm unit is used to execute the JPEG encoding algorithm.

[0263] S207. The decision module sends the photo parameter 2 to the front-end image output module.

[0264] S208, The front-end image output module sends the image capture parameter 2 to the camera in the hardware layer.

[0265] S209, The camera captures the first set of long and short frame RAW images according to the shooting parameters 2.

[0266] S210, the camera returns the first set of long and short frame RAW images to the front-end output module.

[0267] S211. The front-end output module sends the first set of long and short frame RAW images to the back-end processing module.

[0268] S212, The backend processing module calls the frontend algorithm module to execute frontend algorithm 2.

[0269] Optionally, when the backend processing module calls the frontend algorithm module to execute frontend algorithm 2, it can carry the first set of long and short frame RAW images.

[0270] S213. The front-end algorithm module responds to the call of the back-end processing module by calling the ISP resource call module to execute the HDR fusion algorithm.

[0271] Optionally, when the background algorithm module calls the ISP resource call module to execute the HDR fusion algorithm, it can carry the first set of long and short frame RAW images.

[0272] S214. The ISP resource call module responds to the call from the background algorithm module by calling the hardware layer ISP to execute the HDR fusion algorithm.

[0273] S215, ISP performs an HDR fusion algorithm on the first group of long and short frame RAW images to obtain fused RAW. Figure 1 (Also known as the first fused image).

[0274] S216 and ISP will integrate RAW Figure 1 Return to the ISP resource call module.

[0275] S217, the ISP resource access module will integrate RAW Figure 1 Return to the foreground algorithm module.

[0276] For details, see Figure 9 The specific execution process of steps S213 to S217 may include:

[0277] ① The HDR fusion algorithm unit in the front-end algorithm module sends an instruction to the interface layer of the ISP resource call module to call the HDR fusion algorithm.

[0278] ②After receiving the call instruction from the front-end algorithm module through the interface layer, the ISP resource call module responds to the call instruction by converting the first set of long and short frame RAW images into a format that the ISP platform can recognize through the HDR fusion conversion unit, and encapsulates the data. The resulting data is called the platform format long and short frame RAW image (also known as multi-frame converted image).

[0279] ③ The HDR fusion conversion unit sends the long and short frame RAW images in the platform format to the ISP driver through the ISP encapsulation interface.

[0280] ④ The ISP driver sends long and short frame RAW images in platform format to the hardware layer ISP.

[0281] ⑤ The ISP performs an HDR fusion algorithm on the long and short frame RAW images in the platform format to obtain the fused RAW in the platform format. Figure 1 (Also known as the original fused image).

[0282] ⑥ISP will integrate platform formats into RAW Figure 1The ISP driver and ISP encapsulation interface are used to return the data to the HDR fusion and conversion unit in the ISP resource call module.

[0283] ⑦ The HDR fusion conversion unit fuses RAW files from different platform formats. Figure 1 The conversion is performed to obtain the blended RAW. Figure 1 (Also known as the first fused image).

[0284] ⑧ The RAW2YUV conversion unit will integrate RAW... Figure 1 The interface layer returns the data to the HDR fusion algorithm unit in the front-end algorithm module.

[0285] Next, the HDR fusion algorithm unit will fuse the RAW files. Figure 1 Input is fed into the RAW2YUV algorithm unit to execute the RAW2YUV algorithm, that is, to execute the following step S218.

[0286] S218, The front-end algorithm module calls the ISP resource call module to execute the RAW2YUV algorithm.

[0287] Optionally, when the front-end algorithm module calls the ISP resource call module to execute the RAW2YUV algorithm, it can carry the fused RAW. Figure 1 .

[0288] S219. The ISP resource call module responds to the call from the front-end algorithm module by calling the hardware layer ISP to execute the RAW2YUV algorithm.

[0289] Optionally, when the ISP resource retrieval module calls the ISP to execute the RAW2YUV algorithm, it can carry the fused RAW. Figure 1 .

[0290] S220, ISP for converged RAW Figure 1 Execute the RAW2YUV algorithm to obtain fused YUV Figure 1 (Also known as the first YUV diagram).

[0291] S221 and ISP will integrate YUV Figure 1 Return to the ISP resource call module.

[0292] S222 and the ISP resource retrieval module will integrate YUV. Figure 1 Return to the foreground algorithm module.

[0293] The steps S218 to S222 described above are similar to steps S123 to S127 in Embodiment 1, and will not be repeated here.

[0294] S223, The front-end algorithm module calls the ISP resource call module to execute the JPEG encoding algorithm.

[0295] Optionally, when the front-end algorithm module calls the ISP resource call module to execute the JPEG encoding algorithm, it can carry the fused YUV... Figure 1 .

[0296] S224. The ISP resource call module responds to the call from the front-end algorithm module by calling the hardware layer ISP to execute the JPEG encoding algorithm.

[0297] Optionally, when the ISP resource retrieval module calls the ISP to execute the JPEG encoding algorithm, it can carry the fused YUV... Figure 1 .

[0298] S225, ISP for YUV fusion Figure 1 Execute the JPEG encoding algorithm to obtain HDR JPEG true image. Figure 1 (Also known as the first target image).

[0299] S226 and ISP will integrate HDR JPEG true Figure 1 Return to the ISP resource call module.

[0300] S227, the ISP resource retrieval module will use HDR JPEG true Figure 1 Return to the foreground algorithm module.

[0301] The steps S223 to S227 described above are similar to steps S129 to S133 in Embodiment 1, and will not be repeated here.

[0302] S228, the front-end algorithm module will use HDR JPEG true... Figure 1 Send to the backend processing module.

[0303] S229, The back-end processing module will process HDR JPEG true... Figure 1 Send to the camera access interface.

[0304] S230, Camera Access Interface will support HDR JPEG true Figure 1 Send to the camera app.

[0305] S231, Camera application based on HDR JPEG true Figure 1 Refresh the photo preview interface.

[0306] S232, Camera application will support HDR JPEG true Figure 1 Save to gallery.

[0307] Steps S213 to S232 described above constitute the process of generating an HDR JPEG true image from a RAW image. It can be seen that in this embodiment, the image processing engine can call the ISP to process long and short frame RAW images (multiple frames) to obtain an HDR JPEG (one frame). This means that calling the ISP capability enables multi-input single-output processing, improving image processing speed.

[0308] Furthermore, steps S202 to S207 can be understood as the process of determining image capture parameters and algorithms. Steps S208 to S232 can be understood as the process of capturing a frame of HDR JPEG true image (also known as the first process).

[0309] After a preset time following the sending of photo request 1, the camera access interface can execute step S233 to trigger the second frame of HDR JPEG true image capture. Figure 2 The filming.

[0310] S233. At a preset time interval from the photo request 1, the camera access interface sends a single-shot request 2 to the front-end image output module.

[0311] Single-shot request 2 is similar to single-shot request 1, and will not be described in detail again.

[0312] In response to single-shot request 2, the electronic device repeats steps S208 to S232 to complete the second frame HDR JPEG true image. Figure 2 The electronic device sequentially sends single-shot requests to capture each frame of the HDR JPEG image until the user raises their hand, at which point the following step S234 is executed:

[0313] S234. In response to the user's raise-hand operation, the camera application sends a stop continuous shooting request to the camera access interface.

[0314] The Stop Continuous Shooting Request is used to request that continuous shooting be stopped.

[0315] The camera access interface responds to the stop continuous shooting request by stopping sending single-shot requests to the front-end image output module and stopping shooting.

[0316] It is understood that during the processing of the nth frame HDR JPEG true image n through steps S213 to S227, the acquisition of long and short frame RAW images of the (n+1)th frame HDR JPEG true image n+1 can be performed simultaneously through steps S208 to S212, and the two processes do not affect each other. Here, n is an integer greater than or equal to 1. In other words, the image processing method provided in this embodiment, in HDR scenes, asynchronously calls the ISP capability through the ISP resource call module to fuse long and short frame RAW images to obtain an HDR JPEG true image. Thus, on the one hand, for any two adjacent HDR image frames, the acquisition of the RAW image corresponding to the later HDR image and the processing of the previous HDR image can reuse time, thereby shortening the shooting time of two HDR images; on the other hand, the hardware chip ISP can more quickly process long and short frame RAW images without occupying software system resources, reducing system load and further shortening the processing time of each HDR image frame. By shortening the shooting time of HDR images through the above two aspects, electronic devices can complete the continuous shooting of multiple HDR images in a timely manner, that is, achieve HDR burst shooting and improve the user experience.

[0317] The following is an explanation of the interface corresponding to HDR burst shooting. For example, Figure 11 This is a schematic diagram illustrating the interface changes for an example of HDR burst shooting provided in an embodiment of this application. For example... Figure 11 As shown in Figure (a), with HDR enabled, the phone displays a photo preview interface 1101. In response to a long press operation by the user on the camera control 3021, the phone begins taking photos according to the above process. After each frame of the HDR JPEG image is captured, the thumbnail in the image viewing control 3022 is refreshed, and the number of frames of the currently captured HDR JPEG image is displayed on the interface. Figure 11 As shown in Figure (b), after capturing the 15th frame of the HDR JPEG image, the photo preview interface can be displayed as shown in interface 1102. It can be seen that the currently displayed frame number 1121 in interface 1102 is "15", and the thumbnail in the image viewing control 3022 is refreshed to the thumbnail corresponding to the 15th frame of the HDR JPEG image.

[0318] After a period of time, in response to the user raising their hand, the phone stops recording and displays the following: Figure 11 The interface 1103 is shown in Figure (c). The thumbnail in the image viewing control 3022 is the thumbnail corresponding to the last frame of the HDR JPEG image. In response to the user's selection operation on the image viewing control 3022, the phone displays the last frame of the HDR JPEG image, as shown below. Figure 11 The interface 1104 in Figure (d) is shown. In response to a user's click action on interface 1104, the phone displays... Figure 11The interface 1105 is shown in Figure (e). Besides the last frame of the HDR JPEG image, interface 1105 also includes relevant information about the image, including a burst image illustration 1151. The burst image illustration 1151 indicates that the current image is a frame (generally the last frame) in a burst of images. Therefore, in this embodiment, burst shooting can be achieved in HDR scenes.

[0319] In addition, each frame of the HDR JPEG image is saved to the gallery during continuous shooting, so the images captured in the continuous shooting can also be viewed through the gallery application. For example, Figure 12 Another example of an interface diagram provided for an embodiment of this application, such as... Figure 12 As shown in Figure (a), the phone displays desktop 301. Desktop 301 includes an icon 3012 for the Gallery app. In response to the user's selection of the Gallery app icon 3012, the phone displays the Gallery app interface 1201, as shown... Figure 12 As shown in Figure (b). The gallery application interface 1201 includes a camera album 1211. In response to the user's selection of the camera album 1211, the phone displays the camera album interface 1203, as shown. Figure 12 As shown in Figure (c). The camera album interface 1202 includes... Figure 11 The thumbnail 1221 corresponds to the HDRJPEG image captured in the image. A burst mode icon 1222 is displayed on the thumbnail 1221. The burst mode icon 1222 indicates that the current image is a frame (generally the last frame) in a burst of images. Therefore, the method provided in this embodiment can achieve burst shooting in HDR scenes.

[0320] Furthermore, as can be seen from the above steps, the image processing method provided in this embodiment does not crop the long and short frame RAW images during the processing, thus maintaining the integrity of the information in the long and short frame RAW images, thereby improving the image quality of the obtained HDR JPEG true image and enhancing the user experience. The following description is in conjunction with the accompanying drawings.

[0321] For example, Figure 13 This is another example of interface change provided in an embodiment of this application. For example... Figure 13 As shown in Figure (a), the mobile phone displays the camera album interface 1202. The camera album interface 1202 includes thumbnails 1221 corresponding to the HDR JPEG images captured in HDR burst mode. In response to the user's selection of thumbnail 1221, the mobile phone displays the last HDR JPEG image and related information, such as... Figure 13 The interface 1105 in Figure (b) is shown. Interface 1105 includes a burst view icon 1151 and a details control 1152. In response to the user's selection of the details control 1152, the phone displays interface 1301, as shown... Figure 13 As shown in Figure (c), interface 1301 includes a detailed information card 1311 for the current frame's HDR JPEG image, which contains the image size information 1312. It can be seen that the size of the current frame's HDR JPEG image is 3072×4096. Therefore, the method provided in this embodiment does not crop the long or short frame RAW images, maintaining the original image size, i.e., the same as the RAW image size, thus ensuring the integrity of the image information.

[0322] It should be noted that Example 2 uses continuous shooting in an HDR scene as an example. In other examples, when taking a single shot in an HDR scene, the process of Example 2 can also be referred to to call the ISP to execute the HDR fusion algorithm, RAW2YUV algorithm, and JPEG encoding algorithm to improve image processing speed, thereby improving shooting response and user experience.

[0323] Example 3: Converting RAW images to JPEG images in non-photography scenarios.

[0324] The image processing method provided in this application can be applied not only to the two shooting scenarios described in Embodiments 1 and 2 above, but also to non-shooting scenarios. Specifically, this method can be used to convert RAW images into other image formats in non-shooting scenarios, that is, it supports RAW image format conversion offline. Optionally, this method can convert RAW images into JPEG images, PNG images, etc. In this embodiment, JPEG images are used as an example for explanation.

[0325] It is understandable that in some scenarios, electronic devices acquire images in RAW format, such as images taken in professional mode. For example, Figure 14 This is another example of interface change provided in an embodiment of this application. For example... Figure 14 As shown in Figure (a), the mobile phone displays a photo preview interface 1401 in photo mode. In response to the user swiping left in the mode selection area of ​​the photo preview interface 1401, the shooting mode can be switched to professional mode, as shown in Figure (a). Figure 14 The interface 1402 is shown in Figure (b). Interface 1402 includes "JPEG" format information 1421, indicating that the currently selected image format is JPEG. In response to the user's selection of "JPEG format information 1421," the phone displays as shown... Figure 14 Interface 1403 is shown in Figure (c). Unlike interface 1402, interface 1403 displays multiple format options, including a "RAW" format option 1431. In response to the user's selection of the "RAW" format option 1431, the phone switches the image format to RAW format and displays as shown in Figure (c). Figure 14The interface 1404 is shown in Figure (d). Interface 1404 includes “RAW” format information 1441, indicating that the currently selected image format is RAW format.

[0326] On interface 1404, in response to the user's click on the camera control 3021, the phone takes a picture in professional mode, capturing and saving a RAW format image. RAW format images record the raw information of the image sensor, therefore the file size is relatively large. For example, Figure 15 This is yet another example of an interface diagram provided for an embodiment of this application. The user is based on... Figure 14 The image and detailed information obtained by performing the photo-taking operation on the interface 1404 shown in Figure (d) can be viewed as follows: Figure 15 The interface 1501 is shown in the image. Interface 1501 includes a details card 1511, which contains information such as image name 1512, storage path 1513, and image size information 1514. The image name 1512 shows the image extension as .dng, indicating it is a RAW format image. Similarly, the storage path 1513, "Internal Storage / DCIM / Camera / RAW / ", also confirms the RAW format. The image size information 1514 shows that the image occupies 25.20MB of storage space, indicating a relatively large image size.

[0327] RAW format images are relatively large in size. In some scenarios, users need smaller images for easier transmission, downloading, and viewing. However, current technologies cannot convert RAW format images, making them inconvenient for users.

[0328] In view of this, the image processing method provided in this embodiment supports format conversion of RAW images.

[0329] For example, Figure 16 This is a schematic diagram illustrating an example of interface changes during image format conversion, provided as an embodiment of this application. For example... Figure 16 As shown in Figure (a), the mobile phone displays a graphical interface 1601. The graphical interface 1601 includes a RAW image 1611 and more controls 1612. In response to the user's selection of the more controls 1611, the mobile phone displays... Figure 16 The interface 1602 (also referred to as the first interface) shown in Figure (b) is compared to the image interface 1601. The interface 1602 includes multiple operation items for the image, such as the "move" operation item, the "copy" operation item, and the "convert to JPEG" operation item 1621 (also referred to as the second control).

[0330] In response to the user's selection of the "Convert to JPEG" operation item 1621, the mobile phone initiates format conversion processing on the RAW image 1611. During this process, the mobile phone may display as follows: Figure 16 The interface 1603 shown in Figure (c) prompts the user that an image is currently being processed. After the format conversion is complete, the phone can display as shown... Figure 16 The image interface 1604 is shown in Figure (d). This interface includes the converted JPEG image 1641. The image interface 1604 also includes a details control 1152. In response to a user's selection of the details control 1152, the phone displays the JPEG image 1641 and a details card 1651, such as... Figure 16 The interface 1605 in Figure (e) is shown. The detailed information card 1651 includes image name 1652, storage path 1653, and image size information 1654. In image name 1652, the image extension is .jpg, indicating that the image format is JPEG. Additionally, the storage path 1653... Figure 15 The storage path 1513 has changed, indicating that the image format is no longer RAW. Image size information 1654 shows that the image occupies 1.96MB of storage space, which is significantly smaller than... Figure 15 The image size is 25.20MB, while the image size is smaller in JPEG format.

[0331] The implementation process of the image processing method provided in this embodiment will be described below.

[0332] For example, Figure 17 This is another example of a software architecture diagram provided for an embodiment of this application. In this embodiment, the image processing engine includes a background algorithm module and an ISP resource retrieval module. The structures of the background algorithm module and the ISP resource retrieval module are the same as those in Embodiment 1. Figure 5 The structure of the back-end algorithm module and the ISP resource call module can be the same, so it will not be described in detail here.

[0333] For example, Figure 18 This is a schematic flowchart illustrating yet another example of an image processing method provided in this application. Please refer to [the original text here]. Figure 17 and Figure 18 In this embodiment, the image processing method may include:

[0334] S301, In response to the user's operation to convert the RAW image to JPEG format (also known as the first RAW image), the library application sends the RAW image to be converted to the AIDL interface.

[0335] For example, the RAW image to be converted is Figure 16RAW image 1611. The conversion to JPEG format is, for example, as follows: Figure 16 The selection operation (such as a click operation, also known as the second operation) of the "Convert to JPEG" operation item 1621 is shown.

[0336] The S302 and AIDL interfaces send the RAW image to be converted to the backend algorithm module.

[0337] S303. The background algorithm module executes the background RAW domain algorithm on the RAW image to be converted, and obtains the processed RAW image to be converted.

[0338] S304. The background algorithm module calls the ISP resource call module to execute the RAW2YUV algorithm.

[0339] Optionally, when the background algorithm module calls the ISP resource call module to execute the RAW2YUV algorithm, it can carry the processed RAW image to be converted.

[0340] S305, the ISP resource call module responds to the call from the background algorithm module by calling the hardware layer ISP to execute the RAW2YUV algorithm.

[0341] S306. In response to the call from the ISP resource call module, the ISP executes the RAW2YUV algorithm on the processed RAW image to be converted to obtain the YUV image to be converted.

[0342] S307. The ISP returns the YUV image to be converted to the ISP resource call module.

[0343] S308, the ISP resource call module returns the YUV image to be converted to the background algorithm module.

[0344] S309. The background algorithm module executes the YUV domain algorithm on the YUV image to be converted, and obtains the processed YUV image to be converted.

[0345] S310, the background algorithm module calls the ISP resource call module to execute the JPEG encoding algorithm.

[0346] Optionally, when the background algorithm module calls the ISP resource call module to execute the JPEG encoding algorithm, it can carry the processed YUV image to be converted.

[0347] S311, the ISP resource call module responds to the call from the background algorithm module by calling the hardware layer ISP to execute the JPEG encoding algorithm.

[0348] S312. In response to the call from the ISP resource call module, the ISP performs a JPEG encoding algorithm on the processed YUV image to be converted to obtain the converted JPEG true image (also known as the second target image).

[0349] S313, the ISP returns the converted JPEG image to the ISP resource retrieval module.

[0350] S314, the ISP resource retrieval module returns the converted JPEG image to the background algorithm module.

[0351] The steps S303 to S314 described above are similar to steps S122 to S133 in Embodiment 1, and will not be repeated here.

[0352] S315, the background algorithm module sends the converted JPEG image to the AIDL interface.

[0353] The S316 and AIDL interfaces will send the converted JPEG images to the photo album application.

[0354] S317, Save the converted JPEG image using the photo album application.

[0355] The image processing method provided in this embodiment can also call the ISP to perform secondary processing on the RAW image after the photo is taken, realizing offline format conversion of the RAW image to meet the user's needs for image format and improve the user experience.

[0356] It should be noted that Embodiments 1, 2, and 3 above describe image processing methods for three different scenarios. However, these three embodiments can be combined where the solutions do not conflict. For example, after completing the shooting according to the process shown in Embodiment 1 in a single-shot scenario, HDR continuous shooting can be performed according to the process shown in Embodiment 2, or the RAW image can be converted to a different format according to the process shown in Embodiment 3. This application does not impose specific limitations on these methods.

[0357] The foregoing has detailed examples of image processing methods provided in the embodiments of this application. It is understood that, in order to achieve the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0358] This application embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0359] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0360] The electronic device provided in this embodiment is used to execute the above-described image processing method, and therefore can achieve the same effect as the above-described implementation method.

[0361] When using integrated units, the electronic device may further include a processing module, a storage module, and a communication module. The processing module is used to control and manage the operation of the electronic device. The storage module supports the execution of stored program code and data. The communication module supports communication between the electronic device and other devices.

[0362] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a digital signal processor (DSP), and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0363] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment can be a device having... Figure 1 The device with the structure shown.

[0364] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the image processing method of any of the above embodiments.

[0365] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0366] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the image processing methods in the above-described method embodiments.

[0367] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0368] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0369] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0370] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0371] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0372] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0373] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, said method being executed by an electronic device, characterized in that, The electronic device includes an image sensor and an image signal processor (ISP), and the method includes: The camera application displays a photo preview interface, which includes a first thumbnail, and the camera application's HDR function is enabled. Receive a first operation from the user, the first operation being used to instruct continuous shooting and saving of multiple frames of target images; In response to the first operation, the first process is executed once every preset time interval until a second operation from the user is received, the second operation being used to instruct to stop shooting and save the target image; The first process includes: Multiple frames of raw images are acquired using the image sensor; the raw images are in RAW format. The first fused image is obtained by performing an HDR fusion algorithm on the multiple original images using the ISP. The first fused image is obtained by performing a RAW-to-YUV domain conversion algorithm on the ISP. The first target image is obtained by performing an encoding algorithm on the first YUV image using the ISP; the first target image is one frame among the multiple target images. Save the first target image and refresh the first thumbnail to be a thumbnail of the first target image.

2. The method according to claim 1, characterized in that, The electronic device includes a front-end algorithm module and an ISP resource call module. The step of performing an HDR fusion algorithm on the multiple original images through the ISP to obtain a first fused image includes: The front-end algorithm module calls the ISP resource call module to execute the HDR fusion algorithm on the multiple frames of original images; The ISP resource call module performs data conversion and encapsulation on the multiple original images to obtain multiple converted images. The ISP resource call module calls the ISP to execute the HDR fusion algorithm on the multi-frame converted image; The ISP performs the HDR fusion algorithm on the multi-frame converted images to obtain the original fused image; The ISP returns the original fused image to the ISP resource call module; The ISP resource call module performs data conversion on the original fused image to obtain the first fused image; The ISP resource call module returns the first fused image to the front-end algorithm module.

3. The method according to claim 1 or 2, characterized in that, The electronic device includes a front-end algorithm module and an ISP resource call module. The step of performing a RAW-to-YUV domain conversion algorithm on the first fused image through the ISP to obtain a first YUV image includes: The front-end algorithm module calls the ISP resource call module to execute the RAW domain to YUV domain algorithm on the first fused image; The ISP resource call module performs data conversion and encapsulation on the first fused image to obtain a fused converted image; The ISP resource call module calls the ISP to execute the RAW domain to YUV domain algorithm for the fused and converted image; The ISP performs the RAW-to-YUV domain conversion algorithm on the fused and converted image to obtain the first original YUV image; The ISP returns the first original YUV image to the ISP resource invocation module; The ISP resource call module performs data conversion on the first original YUV image to obtain the first YUV image; The ISP resource call module returns the first YUV image to the front-end algorithm module.

4. The method according to any one of claims 1 to 3, characterized in that, The electronic device includes a front-end algorithm module and an ISP resource retrieval module. The step of performing an encoding algorithm on the first YUV image through the ISP to obtain the first target image includes: The front-end algorithm module calls the ISP resource call module to execute the encoding algorithm for the first YUV image; The ISP resource call module performs data conversion and encapsulation on the first YUV map to obtain the first YUV conversion map; The ISP resource invocation module invokes the ISP to execute the encoding algorithm for the first YUV conversion graph; The ISP executes the encoding algorithm on the first YUV conversion map to obtain the first original target image; The ISP returns the first original target image to the ISP resource invocation module; The ISP resource retrieval module performs data conversion on the first original target image to obtain the first target image; The ISP resource retrieval module returns the first target image to the front-end algorithm module.

5. The method according to any one of claims 1 to 4, characterized in that, The encoding algorithm is the JPEG encoding algorithm, and the first target image is a JPEG format image.

6. The method according to any one of claims 1 to 5, characterized in that, The multi-frame original images include a first original image and a second original image, wherein the first original image is an overexposed image and the second original image is an underexposed image.

7. The method according to any one of claims 1 to 6, characterized in that, The size of the first target image is the same as the size of the original image.

8. The method according to any one of claims 1 to 7, characterized in that, The photo preview interface includes a first control, the first operation is to long press the first control, and the second operation is to stop long pressing the first control.

9. An image processing method, executed by an electronic device, characterized in that, The electronic device includes an application layer, a hardware abstraction layer, and a hardware layer. The application layer includes a gallery application and an AIDL interface. The hardware abstraction layer includes an image processing engine. The hardware layer includes an image signal processor (ISP). The method includes: The first interface of the gallery application is displayed, and the first interface includes a first RAW image and a second control; In response to the user's selection operation on the second control, the gallery application sends the first RAW image to the image processing engine through the AIDL interface; The image processing engine calls the ISP to perform format conversion on the first RAW image to obtain the second target image; The image processing engine sends the second target image to the gallery application via the AIDL interface.

10. The method according to claim 9, characterized in that, The image processing engine calls the ISP to perform format conversion on the first RAW image to obtain the second target image, including: The image processing engine performs a RAW domain algorithm on the first RAW image to obtain a second RAW image; The image processing engine calls the ISP to perform a RAW-to-YUV domain conversion algorithm on the second RAW image to obtain the second YUV image; The image processing engine performs a YUV domain algorithm on the second YUV image to obtain a third YUV image; The image processing engine calls the ISP to perform an encoding algorithm on the third YUV image to obtain the second target image.

11. The method according to claim 10, characterized in that, The image processing engine includes a background algorithm module and an ISP resource call module; The image library application sends the first RAW image to the image processing engine through the AIDL interface, including: The image library application sends the first RAW image to the backend algorithm module through the AIDL interface; The image processing engine performs a RAW domain algorithm on the first RAW image to obtain a second RAW image, including: The background algorithm module performs the RAW domain algorithm on the first RAW graph to obtain the second RAW graph; The image processing engine performs a YUV domain algorithm on the second YUV image to obtain a third YUV image, including: The background algorithm module performs a YUV domain algorithm on the second YUV image to obtain the third YUV image.

12. The method according to claim 11, characterized in that, The image processing engine calls the ISP to perform a RAW-to-YUV domain conversion algorithm on the second RAW image to obtain a second YUV image, including: The background algorithm module calls the ISP resource call module to execute the RAW domain to YUV domain conversion algorithm for the second RAW graph; The ISP resource call module performs data transformation and encapsulation on the second RAW graph to obtain the second RAW transformed graph; The ISP resource call module calls the ISP to execute the RAW domain to YUV domain algorithm for the second RAW conversion graph; The ISP performs the RAW-to-YUV domain conversion algorithm on the second RAW conversion graph to obtain the second original YUV graph; The ISP returns the second original YUV map to the ISP resource invocation module; The ISP resource retrieval module performs data conversion on the second original YUV image to obtain the second YUV image; The ISP resource call module returns the second YUV image to the background algorithm module.

13. The method according to claim 11 or 12, characterized in that, The image processing engine calls the ISP to perform an encoding algorithm on the third YUV image to obtain the second target image, including: The background algorithm module calls the ISP resource call module to execute the encoding algorithm for the third YUV image; The ISP resource call module performs data conversion and encapsulation on the third YUV map to obtain the third YUV conversion map; The ISP resource invocation module invokes the ISP to execute the encoding algorithm for the third YUV conversion map; The ISP performs the encoding algorithm on the third YUV conversion image to obtain the second original target image; The ISP returns the second original target image to the ISP resource invocation module; The ISP resource retrieval module performs data conversion on the second original target image to obtain the second target image; The ISP resource retrieval module returns the second target image to the background algorithm module.

14. The method according to any one of claims 9 to 13, characterized in that, The second target image is a JPEG or PNG format image.

15. An image processing method, said method being performed by an electronic device, characterized in that, The electronic device includes an image sensor and an image signal processor (ISP), and the method includes: The camera application displays a photo preview interface, which includes a first thumbnail. Receive a third operation from the user, the third operation being used to instruct the capture of an image frame; In response to the third operation, a third RAW image is acquired through the image sensor; The third RAW image is processed in the first way to obtain the second thumbnail; Save the second thumbnail and refresh the first thumbnail to the second thumbnail; The third RAW image is subjected to a second processing to obtain a third target image. The duration of the first processing is less than the duration of the second processing. The first processing includes: executing a first RAW domain algorithm to obtain a fourth RAW image, and executing a RAW domain to RGB domain algorithm on the fourth RAW image to obtain a second thumbnail. The second processing includes executing a RAW domain to YUV domain algorithm and an encoding algorithm through the ISP. Replace the second thumbnail with the third target image.

16. The method according to claim 15, characterized in that, The number of third RAW images is one frame or more. The step of performing a first RAW domain algorithm on the third RAW images to obtain a fourth RAW image includes: The first RAW domain algorithm is performed on the target RAW image to obtain the fourth RAW image, wherein the target RAW image is one frame of the third RAW image or one of the three frames.

17. The method according to claim 15 or 16, characterized in that, The electronic device includes an image processing engine, and the second processing of the third RAW image to obtain a third target image includes: The image processing engine performs a second RAW domain algorithm on the third RAW image to obtain a fifth RAW image; The image processing engine calls the ISP to perform a RAW-to-YUV domain conversion algorithm on the fifth RAW image to obtain the fourth YUV image; The image processing engine performs a YUV domain algorithm on the fourth YUV image to obtain a fifth YUV image; The image processing engine calls the ISP to perform an encoding algorithm on the fifth YUV image to obtain the third target image.

18. The method according to claim 17, characterized in that, The electronic device further includes an application layer, the camera application is located in the application layer, and the application layer also includes an AIDL interface. The step of replacing the second thumbnail with the third target image includes: The image processing engine sends the third target image to the camera application via the AIDL interface; The camera application replaces the second thumbnail with the third target image.

19. The method according to claim 17 or 18, characterized in that, The number of third RAW images is one frame or more. The image processing engine performs a second RAW domain algorithm on the third RAW images to obtain a fifth RAW image, including: The image processing engine performs the second RAW domain algorithm on all the third RAW images to obtain the fifth RAW image.

20. The method according to any one of claims 17 to 19, characterized in that, The encoding algorithm is the JPEG encoding algorithm, and the third target image is a JPEG format image.

21. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 20.

22. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 20.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 20.

Images