Image processing method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and electronic equipment, and the method comprises the steps: displaying a photographing preview interface of a camera application, the photographing preview interface comprises a first thumbnail, and the HDR function of the camera application is in an on state; receiving a first operation of a user, wherein the first operation is used for indicating continuous shooting of multiple frames of images; in response to the first operation, the first process is executed once every preset time until a second operation of the user is received, and the second operation is used for indicating to stop shooting; the first process comprises the following steps: acquiring multiple frames of original images through an image sensor; hDR fusion, conversion and coding are carried out on the multiple frames of original images through ISP, and a first target image is obtained; and storing the first target image, and refreshing the first thumbnail into a thumbnail of the first target image. According to the method, HDR continuous shooting can be realized.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] Currently, taking photos has become an important function of electronic devices. However, the inventors have found that in some scenarios, such as high dynamic range (HDR) scenarios, electronic devices cannot achieve continuous shooting (i.e., burst shooting), which affects the user experience. Summary of the Invention

[0003] The present application provides an image processing method and an electronic device that can achieve continuous shooting in an HDR scene.

[0004] In a first aspect, the present application provides an image processing method, which is performed by an electronic device, the electronic device including an image sensor and an image signal processor (ISP), the method including: displaying a photo preview interface of a camera application, the photo preview interface including a first thumbnail, and the HDR function of the camera application being turned on; receiving a first operation of the user, the first operation being used to instruct continuous shooting of multiple frames of images; in response to the first operation, executing a first process once every preset time interval until receiving a second operation of the user, the second operation being used to instruct to stop shooting; the first process including: acquiring multiple frames of original images through the image sensor; performing HDR fusion, conversion and encoding of the multiple frames of original images through the ISP to obtain a first target image; saving the first target image, and refreshing the first thumbnail to a thumbnail of the first target image.

[0005] The image processing method provided in the first aspect may be the method shown in Example 2 of the specific embodiment. The thumbnail of the first target image may be the HDR JPEG true image in the specific embodiment.

[0006] Optionally, the original image may be an image in RAW format. Optionally, the preset duration may be a frame interval of the electronic device, that is, 1 / frame rate.

[0007] The image processing method provided in the first aspect, in an HDR scenario, performs HDR fusion, conversion, and encoding of multiple frames of original images through the ISP to achieve the capture of HDR images (i.e., the first target image). In this way, on the one hand, for the capture of any two adjacent HDR images, the acquisition of multiple frames of original images corresponding to the latter HDR image can reuse the time of HDR fusion, conversion, and encoding of the previous HDR image, thereby shortening the capture time of the two HDR images; on the other hand, the hardware chip ISP can process multiple frames of original images more quickly without occupying software system resources, which can reduce the system load, shorten the processing time of HDR images, and thus achieve HDR continuous shooting and improve user experience.

[0008] In one possible implementation, the original image is an image in RAW format, and HDR fusion, conversion, and encoding are performed on multiple frames of original images through the ISP to obtain a first target image, including: executing an HDR fusion algorithm on the multiple frames of original images through the ISP to obtain a first fused image; executing a RAW domain to YUV domain algorithm on the first fused image through the ISP to obtain a first YUV image; and executing an encoding algorithm on the first YUV image through the ISP to obtain the first target image.

[0009] In this implementation, HDR fusion, conversion, and encoding are performed by executing an HDR fusion algorithm, a RAW domain to YUV domain conversion algorithm, and an encoding algorithm to obtain an image in the target format. The target format is, for example, PNG or JPEG. Furthermore, in this implementation, the HDR fusion algorithm, the RAW domain to YUV domain conversion algorithm, and the encoding algorithm are all implemented via the ISP, enabling faster processing of multiple frames of raw images without occupying software system resources, reducing system load, and further shortening the processing time for each HDR image frame.

[0010] In one possible implementation, the electronic device includes a foreground algorithm module and an ISP resource calling module, and executes an HDR fusion algorithm on multiple frames of original images through the ISP to obtain a first fused image, including: the foreground algorithm module calls the ISP resource calling module to execute the HDR fusion algorithm on the multiple frames of original images; the ISP resource calling module performs data conversion and encapsulation on the multiple frames of original images to obtain multiple frames of converted images; the ISP resource calling module calls the ISP to execute the HDR fusion algorithm on the multiple frames of converted images; the ISP executes the HDR fusion algorithm on the multiple frames of converted images to obtain the original fused image; the ISP returns the original fused image to the ISP resource calling module; the ISP resource calling module performs data conversion on the original fused image to obtain the first fused image; and the ISP resource calling module returns the first fused image to the foreground algorithm module.

[0011] That is, the ISP resource calling module is called by the foreground algorithm module, and then the ISP resource calling module calls the ISP to implement the HDR fusion algorithm. Optionally, the ISP resource calling module may include an interface layer for communicating with the foreground algorithm module. In addition, the ISP resource calling module may communicate with the ISP via the ISP encapsulation interface.

[0012] In this implementation, the ISP resource call module converts and encapsulates multiple frames of original images to produce multiple frames of converted images. These multiple frames of converted images can be recognized and processed by the ISP, which then executes the HDR fusion algorithm. The resulting original fused image is then converted by the ISP resource call module to produce a first fused image that can be recognized by the foreground algorithm module and the camera application. In other words, the ISP resource call module converts and transmits the input and output data of the HDR fusion algorithm, enabling the foreground algorithm module to call the ISP.

[0013] In one possible implementation, the electronic device includes a foreground algorithm module and an ISP resource calling module, and the ISP executes a RAW domain to YUV domain algorithm on the first fused image to obtain a first YUV image, including: the foreground algorithm module calls the ISP resource calling module to execute the RAW domain to YUV domain algorithm on the first fused image; the ISP resource calling module performs data conversion and encapsulation on the first fused image to obtain a fused converted image; the ISP resource calling module calls the ISP to execute the RAW domain to YUV domain algorithm on the fused converted image; the ISP executes the RAW domain to YUV domain algorithm on the fused converted image to obtain a first original YUV image; the ISP returns the first original YUV image to the ISP resource calling module; the ISP resource calling module performs data conversion on the first original YUV image to obtain a first YUV image; and the ISP resource calling module returns the first YUV image to the foreground algorithm module.

[0014] In this implementation, the ISP resource call module converts and encapsulates the first fused image to obtain a fused converted image. The fused converted image can be recognized and processed by the ISP, and then the RAW domain to YUV domain algorithm is executed. The converted first original YUV image is converted by the ISP resource call module, and the resulting first YUV image can be recognized by the foreground algorithm module and the camera application. In other words, the ISP resource call module realizes the conversion and transmission of the input and output data of the RAW domain to YUV domain algorithm, thereby enabling the foreground algorithm module to call the ISP.

[0015] In one possible implementation, an electronic device includes a foreground algorithm module and an ISP resource calling module, and executes an encoding algorithm on a first YUV image through the ISP to obtain a first target image, including: the foreground algorithm module calls the ISP resource calling module to execute the encoding algorithm on the first YUV image; the ISP resource calling module performs data conversion and encapsulation on the first YUV image to obtain a first YUV conversion image; the ISP resource calling module calls the ISP to execute the encoding algorithm on the first YUV conversion image; the ISP executes the encoding algorithm on the first YUV conversion image to obtain a first original target image; the ISP returns the first original target image to the ISP resource calling module; the ISP resource calling module performs data conversion on the first original target image to obtain a first target image; and the ISP resource calling module returns the first target image to the foreground algorithm module.

[0016] In this implementation, the ISP resource call module converts and encapsulates the first YUV image to obtain a first YUV conversion image. The first YUV conversion image can be recognized and processed by the ISP, and the encoding algorithm can then be executed. The encoded first original target image is converted by the ISP resource call module, and the resulting first target image can be recognized by the foreground algorithm module and the camera application. In other words, the ISP resource call module realizes the conversion and transmission of the encoding algorithm input and output data, thereby enabling the foreground algorithm module to call the ISP.

[0017] In a possible implementation, the encoding algorithm is a JPEG encoding algorithm, and the first target image is an image in JPEG format.

[0018] In a possible implementation, the multiple frames of original images include a first original image and a second original image, the first original image is an overexposed image, and the second original image is an underexposed image.

[0019] That is, the multiple frames of original images may be long-frame and short-frame RAW images.

[0020] In a possible implementation, the size of the first target image is the same as the size of the original image.

[0021] That is to say, during the image processing process, the original image is not cropped, so the integrity of the original image information is maintained, thereby improving the image effect of the obtained first target image and enhancing the user experience.

[0022] In a possible implementation, the photo preview interface includes a first control, the first operation is an operation of long pressing the first control, and the second operation is an operation of stopping long pressing the first control.

[0023] The first control may be a photo control. That is, long pressing the photo control triggers continuous shooting, and stopping the long press (i.e., lifting the hand) triggers the stop of continuous shooting.

[0024] In second aspect, the present application provides an image processing method, which is executed by an electronic device, the electronic device includes an application layer, a hardware abstraction layer and a hardware layer, the application layer includes a gallery application and an AIDL interface, the hardware abstraction layer includes an image processing engine, and the hardware layer includes an image signal processor ISP. The method includes: displaying a first interface of the gallery application, the first interface including a first RAW image and a second control; in response to a user's selection operation of the second control, the gallery application sends the first RAW image to the image processing engine through the AIDL interface; the image processing engine calls the ISP to convert the format of the first RAW image to obtain a second target image.

[0025] The image processing method provided in the second aspect may be the method shown in Example 3 of the specific embodiment. The first RAW image may be the RAW image to be converted in the specific embodiment, and the second target image may be the converted JPEG true image in the specific embodiment.

[0026] The image processing method provided in the second aspect can also call ISP to perform secondary processing on the RAW image in the interface of the gallery application, and realize the format conversion of the RAW image in an offline state (not taking pictures), meeting the user's demand for image format and improving the user experience.

[0027] In one possible implementation, the image processing engine calls the ISP to perform format conversion on the first RAW image to obtain a second target image, including: the image processing engine executes a RAW domain algorithm on the first RAW image to obtain a second RAW image; the image processing engine calls the ISP to execute a RAW domain to YUV domain algorithm on the second RAW image to obtain a second YUV image; the image processing engine executes a YUV domain algorithm on the second YUV image to obtain a third YUV image; the image processing engine calls the ISP to execute an encoding algorithm on the third YUV image to obtain the second target image.

[0028] In this implementation, while converting the formats (RAW domain to YUV domain and encoding), the RAW domain algorithm is executed on the first RAW image, and the YUV domain algorithm is executed on the converted second YUV image, thereby improving the image quality and thus improving the user experience.

[0029] In one possible implementation, the image processing engine includes a background algorithm module and an ISP resource calling module; the gallery application sends the first RAW image to the image processing engine through the AIDL interface, including: the gallery application sends the first RAW image to the background algorithm module through the AIDL interface; the image processing engine executes a RAW domain algorithm on the first RAW image to obtain a second RAW image, including: the background algorithm module executes a RAW domain algorithm on the first RAW image to obtain a second RAW image; the image processing engine executes a YUV domain algorithm on the second YUV image to obtain a third YUV image, including: the background algorithm module executes a YUV domain algorithm on the second YUV image to obtain a third YUV image.

[0030] In this implementation, the AIDL interface of the application layer is used to realize the communication between the camera application and the image processing engine of the hardware abstraction layer, thereby realizing image processing in an offline state (non-photographing state).

[0031] In one possible implementation, the image processing engine calls the ISP to execute the RAW domain to YUV domain algorithm on the second RAW image to obtain the second YUV image, including: the background algorithm module calls the ISP resource calling module to execute the RAW domain to YUV domain algorithm on the second RAW image; the ISP resource calling module performs data conversion and encapsulation on the second RAW image to obtain the second RAW conversion image; the ISP resource calling module calls the ISP to execute the RAW domain to YUV domain algorithm on the second RAW conversion image; the ISP executes the RAW domain to YUV domain algorithm on the second RAW conversion image to obtain the second original YUV image; the ISP returns the second original YUV image to the ISP resource calling module; the ISP resource calling module performs data conversion on the second original YUV image to obtain the second YUV image; and the ISP resource calling module returns the second YUV image to the background algorithm module.

[0032] Optionally, the ISP resource calling module may include an interface layer for communicating with the background algorithm module. In addition, the ISP resource calling module may communicate with the ISP through the ISP encapsulation interface.

[0033] In this implementation, the ISP resource calling module is used to realize the conversion and transmission of the input and output data of the RAW domain to YUV domain algorithm, thereby realizing the call of the background algorithm module to the ISP.

[0034] In one possible implementation, the image processing engine calls the ISP to execute an encoding algorithm on the third YUV image to obtain a second target image, including: the background algorithm module calls the ISP resource calling module to execute the encoding algorithm on the third YUV image; the ISP resource calling module performs data conversion and encapsulation on the third YUV image to obtain a third YUV conversion image; the ISP resource calling module calls the ISP to execute the encoding algorithm on the third YUV conversion image; the ISP executes the encoding algorithm on the third YUV conversion image to obtain a second original target image; the ISP returns the second original target image to the ISP resource calling module; the ISP resource calling module performs data conversion on the second original target image to obtain a second target image; and the ISP resource calling module returns the second target image to the background algorithm module.

[0035] In this implementation, the ISP resource calling module is used to realize the conversion and transmission of the input and output data of the encoding algorithm, thereby realizing the calling of the background algorithm module to the ISP.

[0036] In a possible implementation, the second target image is an image in JPEG format or PNG format.

[0037] In a third aspect, the present application provides an image processing method, which is executed by an electronic device, the electronic device including an image sensor and an image signal processor (ISP), the method including: displaying a photo preview interface of a camera application, the photo preview interface including a first thumbnail; receiving a third operation of the user, the third operation being used to instruct the taking of a frame of image; in response to the third operation, obtaining a third RAW image through the image sensor; performing a first processing on the third RAW image to obtain a second thumbnail; saving the second thumbnail and refreshing the first thumbnail to the second thumbnail; performing a second processing on the third RAW image to obtain a third target image, wherein the duration of the first processing is less than the duration of the second processing, and the second processing includes executing a RAW domain to YUV domain algorithm and an encoding algorithm through the ISP; replacing the second thumbnail with the third target image.

[0038] The image processing method provided in the third aspect may be the method described in Example 1 of the specific embodiment. The second thumbnail may be the RGB thumbnail described in the specific embodiment. The third target image may be the JPEG true image described in the specific embodiment. The first processing may be the foreground algorithm flow described in the specific embodiment, and the second processing may be the background algorithm flow described in the specific implementation.

[0039] The image processing method provided by the third aspect has a first processing time that is shorter than a second processing time. Thus, the first processing can realize rapid processing of the third RAW image, so that when a single-shot request for the next frame of image is received, the next frame of image shooting process can be quickly entered, thereby improving the response speed of shooting. At the same time, the second processing ensures comprehensive and high-quality processing of the image. The second processing includes executing the RAW domain to YUV domain algorithm and encoding algorithm through the ISP, that is, it can be implemented through the ISP hardware chip, thereby realizing asynchronous offline processing of the second processing, not occupying software system resources, reducing system load, further improving the response speed of shooting, and improving the efficiency of algorithm operation.

[0040] In one possible implementation, the second thumbnail is an RGB image, and the first processing is performed on the third RAW image to obtain the second thumbnail, including: executing a first RAW domain algorithm on the third RAW image to obtain a fourth RAW image; and executing a RAW domain to RGB domain algorithm on the fourth RAW image to obtain the second thumbnail.

[0041] In other words, the first processing includes the RAW domain algorithm and the RAW domain to RGB domain algorithm. As can be seen, the first processing algorithm is simpler and saves time, thus quickly providing the second thumbnail and quickly completing the response to the single-shot request for the current frame, improving the photo response speed.

[0042] In one possible implementation, the number of the third RAW images is one or more frames, and the first RAW domain algorithm is executed on the third RAW image to obtain the fourth RAW image, including: executing the first RAW domain algorithm on the target RAW image to obtain the fourth RAW image, where the target RAW image is one frame of the one or more third RAW images.

[0043] That is to say, the RAW domain algorithm in the first processing is a single-input and single-output RAW domain algorithm. This further simplifies the first processing, further saves time, increases the speed of generating the second thumbnail, and further increases the photo response speed.

[0044] In one possible implementation, the electronic device includes an image processing engine, which performs a second processing on the third RAW image to obtain a third target image, including: the image processing engine executes a second RAW domain algorithm on the third RAW image to obtain a fifth RAW image; the image processing engine calls the ISP to execute a RAW domain to YUV domain algorithm on the fifth RAW image to obtain a fourth YUV image; the image processing engine executes a YUV domain algorithm on the fourth YUV image to obtain a fifth YUV image; the image processing engine calls the ISP to execute an encoding algorithm on the fifth YUV image to obtain the third target image.

[0045] In this implementation, the image processing engine calls the ISP to implement the RAW domain to YUV domain algorithm and encoding algorithm, that is, through ISP offline processing, it does not occupy software system resources, can reduce the system load, shorten the second processing time, and improve the user experience.

[0046] In one possible implementation, the image processing engine includes a background algorithm module and an ISP resource calling module, and the image processing engine calls the ISP to execute a RAW domain-to-YUV domain algorithm on the fifth RAW image to obtain a fourth YUV image, including: the background algorithm module calls the ISP resource calling module to execute the RAW domain-to-YUV domain algorithm on the fifth RAW image; the ISP resource calling module performs data conversion and encapsulation on the fifth RAW image to obtain a fifth RAW conversion image; the ISP resource calling module calls the ISP to execute the RAW domain-to-YUV domain algorithm on the fifth RAW conversion image; the ISP executes the RAW domain-to-YUV domain algorithm on the fifth RAW conversion image to obtain a fourth original YUV image; the ISP returns the fourth original YUV image to the ISP resource calling module; the ISP resource calling module performs data conversion on the fourth original YUV image to obtain a fourth YUV image; and the ISP resource calling module returns the fourth YUV image to the background algorithm module.

[0047] Optionally, the ISP resource calling module may include an interface layer for communicating with the background algorithm module. In addition, the ISP resource calling module may communicate with the ISP through the ISP encapsulation interface.

[0048] In this implementation, the ISP resource calling module is used to realize the conversion and transmission of the input and output data of the RAW domain to YUV domain algorithm, thereby realizing the call of the background algorithm module to the ISP.

[0049] In one possible implementation, the image processing engine calls the ISP to execute an encoding algorithm on the fifth YUV image to obtain a third target image, including: the background algorithm module calls the ISP resource calling module to execute the encoding algorithm on the fifth YUV image; the ISP resource calling module performs data conversion and encapsulation on the fifth YUV image to obtain a fifth YUV conversion image; the ISP resource calling module calls the ISP to execute the encoding algorithm on the fifth YUV conversion image; the ISP executes the encoding algorithm on the fifth YUV conversion image to obtain a third original target image; the ISP returns the third original target image to the ISP resource calling module; the ISP resource calling module performs data conversion on the third original target image to obtain a third target image; and the ISP resource calling module returns the third target image to the background algorithm module.

[0050] In this implementation, the ISP resource calling module is used to realize the conversion and transmission of the input and output data of the encoding algorithm, thereby realizing the calling of the background algorithm module to the ISP.

[0051] In one possible implementation, the electronic device also includes an application layer, the camera application is located in the application layer, and the application layer also includes an AIDL interface. Replacing the second thumbnail with the third target image includes: the image processing engine sends the third target image to the camera application through the AIDL interface; the camera application replaces the second thumbnail with the third target image.

[0052] In this implementation, the AIDL interface of the application layer is used to enable communication between the camera application and the image processing engine of the hardware abstraction layer, thereby realizing image processing in an offline state.

[0053] In one possible implementation, the number of third RAW images is one or more frames, and the image processing engine executes the second RAW domain algorithm on the third RAW images to obtain the fifth RAW image, including: the image processing engine executes the second RAW domain algorithm on all third RAW images to obtain the fifth RAW image.

[0054] In this implementation, in the second processing, the RAW domain algorithm is performed on all RAW images, which improves the image processing effect and further improves the user experience.

[0055] In a possible implementation, the encoding algorithm is a JPEG encoding algorithm, and the third target image is an image in JPEG format.

[0056] In a fourth aspect, the present application provides a device, which is included in an electronic device and has the function of implementing the electronic device behavior described in the first aspect and the possible implementations of the first aspect. The function can be implemented through hardware or through hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, etc.

[0057] In a fifth aspect, the present application provides an electronic device, which includes: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute any one of the methods in the technical solutions of the first aspect, the second aspect or the third aspect.

[0058] In a sixth aspect, the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute any one of the methods in the technical solutions of the first aspect, the second aspect or the third aspect.

[0059] Optionally, the chip system may include an application processor (AP) and an ISP, etc.

[0060] In a seventh aspect, the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes any one of the methods in the technical solutions of the first aspect, the second aspect or the third aspect.

[0061] In an eighth aspect, the present application provides a computer program product, which includes: a computer program code, which, when the computer program code runs on an electronic device, enables the electronic device to execute any one of the methods in the technical solutions of the first aspect, the second aspect or the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0063] Figure 2 This is a schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application;

[0064] Figure 3 This is a schematic diagram of an interface of an image processing method provided in an embodiment of the present application;

[0065] Figure 4 This is a flowchart of an image processing method provided in an embodiment of the present application;

[0066] Figure 5 This is another software architecture diagram provided in an embodiment of the present application;

[0067] Figure 6 is a flowchart of another image processing method provided in an embodiment of the present application;

[0068] Figure 7 This is a schematic diagram of an interface change provided in an embodiment of the present application;

[0069] Figure 8 This is a schematic diagram of an HDR continuous shooting interface in the related art;

[0070] Figure 9 This is another software architecture diagram provided in an embodiment of the present application;

[0071] Figure 10 This is a flowchart of another image processing method provided in an embodiment of the present application;

[0072] Figure 11 This is a schematic diagram of the interface changes of an HDR continuous shooting example provided in an embodiment of the present application;

[0073] Figure 12 This is another example interface diagram provided in the embodiment of the present application;

[0074] Figure 13 This is another interface diagram provided in an embodiment of the present application;

[0075] Figure 14 This is another interface diagram provided in an embodiment of the present application;

[0076] Figure 15 This is another interface diagram provided in an embodiment of the present application;

[0077] Figure 16 This is a schematic diagram of an interface change for an image format conversion provided in an embodiment of the present application;

[0078] Figure 17 This is another software architecture diagram provided in an embodiment of the present application;

[0079] Figure 18 This is a flowchart of another image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0081] In the following, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Therefore, a feature specified as "first," "second," or "third" may explicitly or implicitly include one or more of the features.

[0082] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in different places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0083] To better understand the embodiments of the present application, the terms or concepts that may be involved in the embodiments are explained below.

[0084] 1. RAW images

[0085] Images in RAW format are also called RAW images. RAW images are unprocessed and uncompressed images captured by the image sensor of an electronic device (such as a mobile phone, digital camera, scanner, or film scanner). Optionally, RAW images can have a file extension such as .dng.

[0086] Generally, RAW images record the original information of the photosensitive element (such as shutter speed, aperture value, white balance, etc.) without losing any information, so their file size is relatively large.

[0087] 2. YUV image and RGB image

[0088] YUV is a color encoding method. "Y" represents luminance (luma), or grayscale values; "U" and "V" represent chrominance (chroma), which describe the color and saturation of an image. Together, they specify the color of a pixel. Images encoded using the YUV method are called YUV images or YUV maps.

[0089] An image encoded using the RGB color model is called an RGB image or RGB image. The RGB color model represents colors by mixing varying intensities of red, green, and blue. In an RGB image, the color of each pixel is determined by the values ​​of these three color channels.

[0090] 3. JPEG and PNG images

[0091] Currently, image coding formats include the JPEG format developed by the Joint Photographic Experts Group, the Portable Network Graphics (PNG) format, the Graphics Interchange Format (GIF), and the like.

[0092] JPEG is a lossy image compression format. JPEG images are called JPEG images. JPEG images have file extensions such as .jpg or .jpeg. JPEG images are smaller in size than RAW images.

[0093] The PNG format supports lossless compression and has a transparent background. PNG images are called PNG images. The file extension for PNG images is .png. PNG images are smaller in file size than RAW images.

[0094] 4. Real images and thumbnails

[0095] A true image refers to an image of a size (i.e., original size) that an electronic device supports outputting. It is a complete image that has not been reduced or cropped, has a high resolution and clarity, and can show rich details and colors. A true image is usually used when a user needs to view detailed content, edit, or print. It should be noted that in the embodiments of the present application, a true image is not used for limitation and is only used to distinguish it from a thumbnail. Optionally, a true image can be a RAW image, a JPEG image, a PNG image, or the like.

[0096] A thumbnail is a smaller version of the original image. The primary purpose of a thumbnail is to allow users to quickly browse and identify the approximate content of an image without having to load the full, high-resolution image. This helps improve image browsing efficiency, especially in folders, web pages, or applications containing a large number of images. Thumbnails are smaller in size, resulting in smaller file sizes.

[0097] 5. High dynamic range (HDR)

[0098] HDR is an image processing technology that improves the brightness and contrast of images to create a more realistic, vivid and layered visual effect.

[0099] In HDR images, details of both very bright and very dark areas can be displayed simultaneously, while traditional images often lose information in these extreme areas. HDR technology can make the bright parts of the picture brighter and the dark parts darker while retaining more details, making the image closer to what the human eye actually sees.

[0100] The following describes the technical problems faced by this application and the electronic devices to which the methods provided in the embodiments of this application are applicable.

[0101] An image is the appearance of an object through optical or electronic devices. Photography is a key component of imaging. Currently, photography has become a crucial function of electronic devices. Researchers are primarily interested in making photography faster and producing better images.

[0102] Generally, improving image quality requires not only better devices but also enhanced image processing algorithms. Consequently, during the photography process, electronic devices are increasingly overloaded with system-mounted algorithms, increasing in number and complexity (in other words, overloading the system with photography algorithms). For example, when capturing a single frame, the RAW image captured by the image sensor must undergo algorithms such as white balance correction, denoising, color correction, sharpening, exposure adjustment, lens calibration, format conversion, and compression to produce the final image. However, electronic device systems require that photo requests be issued frame by frame. If a request for the previous frame is not responded to, the next request cannot be issued. Consequently, overloading the photography algorithm leads to reduced photography efficiency. From a user's perspective, users experience a slowed-down photography response: after clicking the photo control, the device takes a long time to complete a single frame. Furthermore, in some scenarios, such as HDR photography, the heavy workload of the system-mounted photography algorithms can prevent the device from taking continuous photos (also known as continuous shooting).

[0103] The image processing method provided in the embodiments of the present application is intended to solve the above-mentioned technical problems.

[0104] The image processing method provided in the embodiments of the present application can be applied to electronic devices with camera functions, such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, digital cameras, drones, scanners, film scanners, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0105] For example, Figure 11 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0106] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0107] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0108] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0109] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0110] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0111] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0112] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0113] The ISP processes data fed back by camera 193. For example, the ISP can perform algorithmic optimizations on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be incorporated into camera 193.

[0114] The camera 193 is used to capture still images or videos. When taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element (also called an image sensor), and the camera's photosensitive element converts the light signal into an electrical signal. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. Optionally, an analog-to-digital conversion module can be integrated into the photosensitive element, which further converts the electrical signal into a digital image signal. These digital image signals are stored in a specific format and data structure to form a RAW image. The photosensitive element transmits the RAW image to the ISP and / or DSP. The ISP and / or DSP processes the RAW image and converts the processed image into an image signal in a standard RGB, YUV, or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0115] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0116] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0117] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0118] The hardware system of electronic device 100 is described in detail above. The following describes the software system of electronic device 100. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. In this embodiment of the application, the Android system with a layered architecture is used as an example to exemplify the software system of electronic device 100.

[0119] like Figure 2 As shown, a software system using a layered architecture is divided into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into four layers, from top to bottom, namely, application layer 210, application framework layer 220, hardware abstraction layer 230, and driver layer 240. In addition, for ease of understanding, Figure 2 Also shown is the hardware layer 250 of the electronic device 100. Each layer will be described below.

[0120] The application layer 210 may include camera and gallery applications, as well as calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications (not shown in the figure).

[0121] In response to a user's photo-taking operation, the camera application (hereinafter referred to as the camera application) may send a photo-taking request to the application framework layer. The photo-taking request is used to request to capture an image.

[0122] Optionally, the application layer 210 may have an Android Interface Definition Language (AIDL) interface. Applications in the application layer 210 may communicate with modules in other layers via the AIDL interface. For example, a camera application and a gallery application may communicate with modules in the hardware abstraction layer via the AIDL interface.

[0123] The application framework layer 220 provides an application access interface and a programming framework for the application of the application layer 210. For example, the application framework layer 220 includes a camera access interface, which is used to provide camera shooting services through camera management and camera devices.

[0124] The camera management in the application framework layer 220 is used to manage the camera. The camera management can obtain the parameters of the camera, such as determining the working status of the camera.

[0125] The camera device in the application framework layer 220 is used to provide a data access interface between different camera devices and camera management.

[0126] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer 230 may include a camera hardware abstraction layer and other hardware device abstraction layers. In the embodiment of the present application, the camera hardware abstraction layer may include a front-end image output module and a back-end processing module. The front-end image output module is used to send photo parameters to the bottom layer to trigger the image sensor in the camera to capture a RAW image. The back-end processing module is used to process the RAW image captured by the image sensor. Optionally, the camera hardware abstraction layer can be connected to a camera algorithm library, and each module in the camera hardware abstraction layer can call the algorithm modules in the camera algorithm library.

[0127] In the embodiment of the present application, the camera algorithm library includes an image processing engine. The image processing engine may include a foreground algorithm module (or called a foreground photo module), a background algorithm module (or called a background photo module), and an ISP resource call module. The foreground algorithm module is used to process the RAW image based on the foreground algorithm. The background algorithm module is used to process the RAW image based on the background algorithm. The background algorithm module is used to execute the background algorithm on the RAW image based on the background algorithm. It should be understood that in the embodiment of the present application, for a certain frame of image, the foreground algorithm can be understood as: the processing algorithm for the frame of image executed during the photo request response process of the frame of image. The background algorithm can be understood as: the processing algorithm for the frame of image that can be executed after the photo request response of the frame of image. Among them, the photo request response process of a certain frame of image refers to the process of performing image acquisition and processing in response to the photo request of the frame of image sent by the camera application, and applying the relevant data of the frame of image (such as thumbnail, JPEG image, etc.) to the camera for the first time. In other words, the time period involved in the photo request response process is the time period between the moment a photo shooting request for a frame of image sent by the camera application is received and the first time relevant data of the frame of image is fed back to the camera application.

[0128] The foreground algorithm module and the background algorithm module can call the ISP resource calling module to realize image processing through the ISP at the hardware layer. Optionally, the ISP resource calling module is used to realize data conversion and matching between the foreground algorithm module or the background algorithm module and the ISP, so that the data input to the ISP can be recognized by it, and the data fed back to the foreground algorithm module or the background algorithm module can be recognized by it. Specifically, the ISP resource calling module can receive the call of the foreground algorithm module or the background algorithm module, convert and encapsulate the image data, and then input it into the ISP. The ISP returns the processed image data to the ISP resource calling module. The ISP resource calling module converts the processed image data and returns it to the foreground algorithm module or the background algorithm module.

[0129] Optionally, the hardware abstraction layer may further include an ISP encapsulation interface, which is used to implement docking communication between the ISP resource calling module and the ISP hardware.

[0130] Optionally, the algorithms contained in the foreground algorithm module and the background algorithm module can be loaded according to the actual shooting scene, shooting mode, etc. Figure 2 In one embodiment, the image processing engine further includes a decision module. The decision module is configured to determine the shooting parameters and algorithm type based on the shooting mode, configuration parameters, environmental parameters, etc. in the shooting request. The shooting parameters may include the frame sequence, exposure parameters, and image size, etc. The algorithm type may include a foreground algorithm type and a background algorithm type. The decision module may send the foreground algorithm type to the foreground algorithm module, and the foreground algorithm module loads the corresponding foreground algorithm based on the foreground algorithm type. The decision module may send the background algorithm type to the background algorithm module, and the background algorithm module loads the corresponding background algorithm based on the background algorithm type.

[0131] Optionally, the image processed by the foreground algorithm module can be returned to the backend processing module, which then passes it to the camera application of the application layer 210 via the camera access interface. The image processed by the background algorithm module can be passed to the camera application and / or the gallery application via the AIDL interface.

[0132] It should be noted that the names of modules, data, or operations in the embodiments of the present application are only used to distinguish and are not used to limit. For example, the foreground algorithm module and the background algorithm module are used to distinguish the two modules, but are not used to specifically limit the modules to run in the foreground or the background.

[0133] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver and an ISP driver.

[0134] The hardware layer 250 may include a camera, an ISP, and other hardware devices. The camera includes an image sensor (i.e., a photosensitive element). The number of cameras may be one or more. The camera may be a wide-angle camera, a telephoto camera, a time of flight (TOF) sensor, a multispectral camera, etc., and the present embodiment does not limit this.

[0135] For ease of understanding, the following examples of this application will be described with Figure 1 and Figure 2 Taking the electronic device with the structure shown as an example, the image processing method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0136] First, the camera modes of an electronic device can be divided into single-shot (abbreviated as single-shot) and continuous-shot (abbreviated as continuous-shot). Single-shot refers to the electronic device capturing only one frame of an image in response to a user's single-shot operation. Continuous-shot refers to the electronic device capturing multiple frames of an image in response to a user's continuous-shot operation. Optionally, the single-shot operation can be a single-click on the camera control (also referred to as the first control) in the camera preview interface or by pressing the power button (i.e., the power-on button). If the smile capture function is enabled, the single-shot operation can also be a smiling expression facing the camera. If the gesture camera function is enabled, the single-shot operation can also be a preset gesture (such as a gesture of pointing the palm toward the phone's camera). If the voice-activated camera function is enabled, the single-shot operation can also be a voice input of a preset camera keyword (such as "photo"). Continuous-shot operations can be a long press on the camera control in the camera preview interface, a press and drag to the right, or a press of the volume button. The embodiments of this application do not specifically limit single-shot and continuous-shot operations. The following mainly uses single-shot operations as a single-click on the camera control and continuous-shot operations as a long press on the camera control as examples.

[0137] The embodiments of the present application relate to image processing methods in single-shot scenarios and HDR continuous-shot scenarios, which are described below in conjunction with the accompanying drawings.

[0138] Example 1: Single shot scenario.

[0139] Figure 3 This is a schematic diagram of an interface of an image processing method provided in an embodiment of the present application. Taking a mobile phone as an example, Figure 3 As shown in FIG. 3 (a), the mobile phone displays a desktop 301 including a camera application icon 3011. In response to a user's selection operation (e.g., a single click operation) on the camera application icon 3011, the mobile phone starts the camera application and enters the photo taking mode by default, displaying the following: Figure 3 FIG. 3( b ) shows a photo preview interface 302. The photo preview interface 302 includes images captured and processed by the mobile phone. The photo preview interface also includes a photo control 3021 and a picture viewing control 3022. The picture viewing control 3022 includes a thumbnail of the image captured by the mobile phone last time.

[0140] In response to the user's single-click operation on the photo control 3021, the mobile phone captures a frame of RAW image, and performs a series of processing on the RAW image to obtain a JPEG image or PNG image (in this embodiment, JPEG image is used as an example for explanation), and saves the JPEG image to the gallery application. At the same time, the mobile phone refreshes the interface and displays a thumbnail of the captured JPEG image in the picture viewing control 3022, as shown in FIG. Figure 3 As shown in Figure (c).

[0141] As described in the above embodiment, when the electronic device processes the RAW image, the system mounts a heavy-loaded photo-taking algorithm, resulting in a slow processing speed, which makes the image processing slower. Figure 3 Figure (b) to Figure 3 The process in Figure (c) takes a long time and the user experience is poor.

[0142] To address this issue, the present embodiment provides an image processing method. After acquiring a RAW image, the RAW image is processed using a foreground algorithm process and a background algorithm process, respectively. The foreground algorithm process is used to generate a thumbnail, while the background algorithm process is used to generate a true JPEG image. The foreground algorithm process is simpler than the background algorithm process. Thus, after the foreground algorithm process quickly processes the RAW image to generate a thumbnail, it can first return the thumbnail to the camera application to complete the response to the photo capture request for the current frame image. After receiving the thumbnail, the camera application can issue a photo capture request for the next frame image, and the system executes the capture of the next frame image. Meanwhile, the background algorithm process can continue until the true JPEG image is generated. In other words, the method provided in this embodiment quickly ends the photo capture process for the current frame image using the simpler foreground algorithm process, while the more complex background algorithm process can be performed asynchronously and offline. In this way, the electronic device can process the photo capture request for the next frame image as soon as possible, thereby improving the photo capture response speed, improving photo capture efficiency, and thus improving the user experience.

[0143] The method of the embodiment of the present application is described below with reference to a flowchart.

[0144] For example, Figure 4 A flowchart of an image processing method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the method includes:

[0145] S101: In response to a user single-clicking operation on a photo control in a photo mode, a camera application of the application layer sends a single-shot request to a camera access interface of the application framework layer.

[0146] Specifically, the camera application generates different shooting requests according to different shooting modes, different user operations, etc. Among them, the shooting modes may include large aperture mode, portrait mode, photo mode, video mode, night mode, professional mode, etc. The shooting request is also called a capture request, which may include a photo request (also called a static capture request), a video request, a preview request, etc. The photo request includes a single shot request and a continuous shot request. For example, when it is determined that the shooting mode is a video mode, and the user's touch operation is a click operation on the video control, the camera application generates a video request. For another example, when it is determined that the shooting mode is a non-video mode, and the operation performed by the user is a long press operation on the photo control, the camera application generates a continuous shot request. In this embodiment, the camera application determines that the current shooting mode is a photo mode, and the operation performed by the user is a single-click operation on the photo control, so the camera application generates a single shot request.

[0147] A single shot request is used to request the capture of a single frame of image. Optionally, the single shot request can carry the capture mode, configuration parameters, and frame image information. The capture mode can be determined based on the user's current selection. For example, Figure 3 In the , the user selects the shooting mode as "Photo Mode". Configuration parameters refer to the shooting-related parameters set by the user, including resolution, image ratio, whether to add a watermark, whether to record geographic location information, whether to add filters, etc.

[0148] The frame image information is the information of the frame image requested to be captured by the current shooting request. Optionally, the frame image information may include a frame number (frame number), an image name, etc. The frame number is used to indicate the number of the frame image currently requested to be captured. Optionally, the frame number can be generated based on a preset numbering rule or randomly generated. The image name is the name of the frame image currently requested to be captured. When the frame image is in JPEG format, the image name is also called a JPEG name (JPEGname). Optionally, the image name can be generated based on the current moment information and a preset numbering rule. For example, if the current moment is 15:08:12 on July 18, 2024, the image name can be IMG_20240718_150812_001. Among them, "001" is a number generated based on a preset numbering rule.

[0149] In a specific embodiment, a shooting request can be defined through the CaptureRequest class. The CaptureRequest may include a CONTROL_CAPTURE_INTENT parameter, which is used to define the request type. When the CONTROL_CAPTURE_INTENT parameter is set to VIDEO_CAPTURE, it indicates that the shooting request is a video recording request. When the CONTROL_CAPTURE_INTENT parameter is set to PREVIEW_CAPTURE, it indicates that the shooting request is a preview request. When the CONTROL_CAPTURE_INTENT parameter is set to STILL_CAPTURE, it indicates that the shooting request is a shooting request. In addition, if the shooting request is a continuous shooting request, the CaptureRequest class may also include a continuous shooting flag. The continuous shooting flag is used to indicate that the shooting request is a continuous shooting request. The continuous shooting request is, for example, BURST MOOD. In this embodiment, the shooting request is a single shooting request, the CONTROL_CAPTURE_INTENT parameter can be set to STILL_CAPTURE, and the CaptureRequest does not include the continuous shooting flag.

[0150] The CaptureRequest may also include a FRAME_NUMBER parameter and a JPEG_NAME parameter. The FRAME_NUMBER parameter is used to define the frame number, and the JPEG_NAME parameter is used to define the image name.

[0151] It should be noted that this step is explained using the example of a user performing a single-shot operation in photo mode. In practice, the camera application also generates a single-shot request when a user performs a single-shot operation in other shooting modes besides video recording mode. The difference lies in the fact that the shooting mode, configuration parameters, etc. carried in the generated single-shot request can be different.

[0152] Optionally, after the camera application generates a single-shot request, it may send the single-shot request to a camera device in the camera access interface.

[0153] S102: The camera access interface of the machine framework layer sends the single-shot request to the front-end image output module in the camera hardware abstraction layer.

[0154] S103: The front-end image output module calls the decision module in the image processing engine to make a photo-taking decision.

[0155] Optionally, when the front-end image output module calls the decision module, it can carry a single-shot request, including various parameters carried in the single-shot request.

[0156] S104. The decision module responds to the call of the front-end image output module and obtains the shooting parameter 1, the foreground algorithm type 1 and the background algorithm type 1 according to the single-shot request.

[0157] Specifically, the decision module can make a photo decision based on the shooting mode and configuration parameters carried in the single shot request, combined with the current environmental parameters. Environmental parameters are related parameters of the current shooting environment, such as ambient light brightness, light source type, etc.

[0158] The shooting strategy determined by the decision module may include shooting parameters, algorithm type, etc. Among them, the shooting parameters may include frame sequence, exposure parameters, image size, etc. The frame sequence refers to the sequence information of x frames of RAW images required to complete the shooting of the current frame image. x can be 1 or an integer greater than 1. For example, in some shooting modes, in order to obtain a frame image with better effect, multiple frames of RAW images can be shot continuously, and the final frame image can be obtained after processing the multiple frames of RAW images. In this case, x is an integer greater than 1. Exposure parameters may include exposure time, exposure value, etc. Image size refers to the number of pixels of the RAW image in the horizontal and vertical directions.

[0159] The algorithm type refers to the type of algorithm required to process the RAW image of this frame. It can be understood that the type of algorithm required may be different depending on the shooting mode, configuration parameters and environmental parameters. In this embodiment, the algorithm includes a foreground algorithm and a background algorithm, and correspondingly, the algorithm type may include a foreground algorithm type and a background algorithm type. The decision module can determine the type of foreground algorithm and the type of background algorithm required in the process of processing the current frame image based on the shooting mode, configuration parameters and current environmental parameters carried in the single shot request. Among them, the complexity of the foreground algorithm is lower than that of the background algorithm. Low algorithm complexity means that there are fewer types of algorithms and / or fewer processing procedures for the algorithm. In other words, the time required to execute the foreground algorithm is less than the time required to execute the background algorithm.

[0160] In a specific embodiment, the foreground algorithm is used to process the RAW image to generate a thumbnail. Optionally, the thumbnail can be an RGB thumbnail. Optionally, the foreground algorithm may include a RAW domain algorithm and a RAW domain to RGB domain algorithm (referred to as the RAW2RGB algorithm). Among them, the RAW domain algorithm refers to an algorithm that performs processing based on the RAW image. After the RAW image is processed by the RAW domain algorithm, the image obtained is still a RAW image. The RAW domain algorithm may include but is not limited to a black level compensation algorithm, a lens correction algorithm, a noise reduction algorithm, an automatic white balance algorithm, etc. The RAW2RGB algorithm is used to convert the RAW image into an RGB image. For ease of distinction, the RAW domain algorithm in the foreground algorithm is referred to as the foreground RAW domain algorithm.

[0161] The background algorithm is used to process the RAW image to generate a true image. Optionally, the true image can be a JPEG true image. Optionally, the background algorithm may include a RAW domain algorithm, a RAW domain to YUV domain algorithm (abbreviated as RAW2YUV algorithm), a YUV domain algorithm and a JPEG encoding algorithm. For ease of distinction, the RAW domain algorithm in the background algorithm is called the background RAW domain algorithm. The RAW2YUV algorithm is used to convert the RAW image into a YUV image. The YUV domain algorithm refers to an algorithm that performs processing based on the YUV image. After the YUV image is processed by the YUV domain algorithm, the image obtained is still a YUV image. The YUV domain algorithm may include but is not limited to a brightness noise reduction algorithm, a color noise reduction algorithm, an edge enhancement algorithm, a hue saturation control algorithm and a contrast brightness control algorithm. The JPEG encoding algorithm is used to perform JPEG encoding on the YUV image to generate a JPEG image.

[0162] In this embodiment, the algorithm type in the foreground RAW domain algorithm and the algorithm type in the background RAW domain algorithm may be the same or different. As a possible implementation method, the algorithm type in the foreground RAW domain algorithm may be less than the algorithm type in the background RAW domain algorithm; and / or, the processing flow of the foreground RAW domain algorithm may be less than the processing flow of the background RAW domain algorithm. For example, in the case where the frame sequence indicates that multiple RAW images are taken, the foreground RAW domain algorithm can be used only to perform RAW domain processing on one frame of the multiple RAW images (such as a reference frame), while the background RAW domain algorithm can be used to perform RAW domain processing on all RAW images. That is, the foreground RAW domain algorithm can be a single-input single-output algorithm, while the background RAW domain algorithm can be a multi-input multi-output algorithm, or a multi-input single-output algorithm.

[0163] In summary, the number of foreground algorithm types is less than that of background algorithm types, and / or the processing flow of the foreground algorithm is less than that of the background RAW domain algorithm, so that the execution time of the foreground algorithm is less than that of the background algorithm.

[0164] In addition, to facilitate distinction from parameters in subsequent embodiments, in this embodiment, the capture parameters determined based on the single-shot request in step S101 are recorded as capture parameters 1, the foreground algorithm type determined is recorded as foreground algorithm type 1, and the background algorithm type determined is recorded as background algorithm type 1. The algorithm corresponding to foreground algorithm type 1 is recorded as foreground algorithm 1, and the algorithm corresponding to background algorithm type 1 is recorded as background algorithm 1.

[0165] S105 , the decision module sends the foreground algorithm type 1 to the foreground algorithm module in the image processing engine.

[0166] S106. The foreground algorithm module loads the foreground algorithm 1 according to the foreground algorithm type.

[0167] For example, Figure 5 This is another software architecture diagram provided in the embodiment of the present application. Figure 5 As shown, after the foreground algorithm module loads the foreground algorithm 1, the module may include a foreground RAW domain algorithm unit and a RAW2RGB algorithm unit. The foreground RAW domain algorithm unit is used to execute the foreground RAW domain algorithm. The RAW2RGB algorithm unit is used to execute the RAW2RGB algorithm.

[0168] S107 , the decision module sends the background algorithm type 1 to the background algorithm module in the image processing engine.

[0169] S108. The background algorithm module loads background algorithm 1 according to the background algorithm type.

[0170] Continue to see Figure 5 After background algorithm 1 is loaded into the background algorithm module, the module may include a background RAW domain algorithm unit, a RAW2YUV algorithm unit, a YUV domain algorithm unit, and a JPEG encoding algorithm unit. The background RAW domain algorithm unit is used to execute the background RAW domain algorithm. The RAW2YUV algorithm unit is used to execute the RAW2YUV algorithm. The YUV domain algorithm unit is used to execute the YUV domain algorithm. The JPEG encoding algorithm unit is used to execute the JPEG encoding algorithm.

[0171] S109, the decision module sends the photographing parameter 1 to the front-end image output module.

[0172] S110. The front-end image output module sends the photographing parameter 1 to the camera of the hardware layer.

[0173] Specifically, the front-end image output module can send the shooting parameter 1 to the camera driver of the hardware driver layer, and the camera driver sends the shooting parameter 1 to the camera. In addition, when the electronic device includes multiple cameras, the shooting parameter 1 can further include camera information. The front-end image output module sends the shooting parameter 1 to the corresponding camera based on the camera information.

[0174] S111 . The camera captures a RAW image (also referred to as a third RAW image) according to photographing parameter 1 .

[0175] Specifically, the camera controls the image sensor to capture a RAW image based on photographic parameter 1. For example, the camera may perform exposure based on the exposure parameter in photographic parameter 1 and control the number of pixels in the RAW image based on the image size in the photographic parameter. Furthermore, the camera controls the number of RAW images captured based on the frame sequence.

[0176] S112: The camera returns the RAW image to the front-end image output module.

[0177] S113: The front-end image output module sends the RAW image to the back-end processing module.

[0178] After receiving the RAW image from the front-end output module, the back-end processing module can call the foreground algorithm module to execute the foreground algorithm process, and call the background algorithm module to execute the background algorithm process. The foreground algorithm process and the background algorithm process can be executed sequentially or simultaneously, which is not limited in this application. They are described below.

[0179] 1) Foreground algorithm process (also called first processing).

[0180] Continue to see Figure 4 , the front-end algorithm process can include:

[0181] S114. The backend processing module calls the foreground algorithm module to execute foreground algorithm 1.

[0182] Optionally, when the post-image processing module calls the foreground algorithm module to execute the foreground algorithm 1, the RAW image can be carried.

[0183] S115 . The foreground algorithm module executes the foreground algorithm 1 in response to the call of the backend processing module to generate an RGB thumbnail (also called a second thumbnail).

[0184] For details, please refer to Figure 5 The foreground algorithm module can perform the foreground RAW domain algorithm on the RAW image through the foreground RAW domain algorithm unit to obtain the processed RAW Figure 1 The foreground RAW domain algorithm unit will process the RAW Figure 1 Input RAW2RGB algorithm unit. RAW2RGB algorithm unit processes the processed RAW Figure 1 Execute the RAW2RGB algorithm to obtain an RGB thumbnail.

[0185] S116. The foreground algorithm module returns the RGB thumbnail to the backend processing module.

[0186] S117. The back-end processing module sends the RGB thumbnail to the camera access interface.

[0187] S118. The camera access interface sends the RGB thumbnail to the camera application.

[0188] S119. The camera application refreshes the photo preview interface according to the RGB thumbnail.

[0189] Specifically, the camera application may replace the thumbnail (also referred to as the first thumbnail) in the picture viewing control in the photo preview interface with the generated RGB thumbnail (also referred to as the second thumbnail).

[0190] S120, the camera app saves the RGB thumbnail to the gallery.

[0191] It is understood that the image name of the RGB thumbnail generated by the foreground algorithm process is the image name carried in the single shot request in step S101. The camera application can send the RGB thumbnail to the gallery application, and the gallery application saves the RGB thumbnail to a preset storage path.

[0192] 2) Background algorithm process (also called second processing).

[0193] For example, Figure 6 For another example of an image processing method provided in this application, please refer to Figure 5 and Figure 6 , the background algorithm process can include:

[0194] S121. The backend processing module calls the background algorithm module to execute background algorithm 1.

[0195] Optionally, when the backend processing module calls the background algorithm flow to execute background algorithm 1, a RAW image can be carried.

[0196] S122, the background algorithm module responds to the call of the back-end processing module and executes the background RAW domain algorithm on the RAW image to obtain the processed RAW image. Figure 2 .

[0197] Please also see Figure 5 The background RAW domain algorithm unit in the background algorithm module performs the background RAW domain algorithm on the RAW image to obtain the processed RAW Figure 2 .

[0198] S123. The background algorithm module calls the ISP resource calling module to execute the RAW2YUV algorithm.

[0199] Optionally, when the background algorithm module calls the ISP resource calling module to execute the RAW2YUV algorithm, it can carry the processed RAW Figure 2 .

[0200] S124. The ISP resource calling module responds to the call of the background algorithm module and calls the ISP of the hardware layer to execute the RAW2YUV algorithm.

[0201] S125, ISP responds to the call of ISP resource calling module and processes the processed RAW Figure 2 Execute the RAW2YUV algorithm to obtain a YUV image.

[0202] S126. The ISP returns the YUV image to the ISP resource calling module.

[0203] S127. The ISP resource calling module returns the YUV image to the background algorithm module.

[0204] Specifically, such as Figure 5 As shown, the ISP resource call module may include an interface layer, a RAW2YUV conversion unit, and a JPEG encoding conversion unit. The interface layer is used to communicate with the various units in the background algorithm module. The RAW2YUV conversion unit is used to convert the input data and output data of the RAW2YUV algorithm. The JPEG encoding conversion unit is used to convert the input data and output data of the JPEG encoding algorithm. It should be noted that, according to the needs of the algorithm that interfaces with the ISP resource call module, the ISP resource call module may also include other units, for example, it may also include an HDR fusion conversion unit, a RAW2YUV+JPEG conversion unit, etc. The HDR fusion conversion unit is used to convert the input data and output data of the HDR fusion algorithm. The RAW2YUV+JPEG conversion unit is used to convert the input data and output data of the RAW2YUV+JPEG algorithm. Among them, the RAW2YUV+JPEG algorithm is used to convert the RAW image into a YUV image, and perform JPEG encoding on the YUV image to generate a JPEG image.

[0205] Combine Figure 5 The specific execution process of steps S123 to S127 may include:

[0206] ① The RAW2YUV algorithm unit in the background algorithm module sends an instruction to call the RAW2YUV algorithm to the interface layer of the ISP resource calling module.

[0207] ② After the ISP resource calling module receives the calling instruction of the background algorithm module through the interface layer, it responds to the calling instruction and converts the processed RAW Figure 2 The data is converted into a format that can be recognized by the ISP platform and encapsulated. The resulting data is called a RAW image in the platform format.

[0208] ③The RAW2YUV conversion unit sends the RAW image in the platform format to the ISP driver through the ISP encapsulation interface.

[0209] ④The ISP driver sends the RAW image in the platform format to the ISP at the hardware layer.

[0210] ⑤ISP performs the RAW2YUV algorithm on the RAW image in the platform format to obtain a YUV image in the platform format.

[0211] ⑥ISP returns the YUV image in the platform format to the RAW2YUV conversion unit in the ISP resource calling module through the ISP driver and ISP encapsulation interface.

[0212] ⑦The RAW2YUV conversion unit converts the YUV image in the platform format to obtain a YUV image that can be recognized by the system.

[0213] ⑧The RAW2YUV conversion unit returns the YUV image to the RAW2YUV algorithm unit in the background algorithm module through the interface layer.

[0214] Afterwards, the RAW2YUV algorithm unit inputs the YUV image into the YUV domain algorithm unit to execute the YUV domain algorithm, that is, execute the following step S128.

[0215] S128. The background algorithm module performs a YUV domain algorithm on the YUV image to obtain a processed YUV image.

[0216] Afterwards, the YUV domain algorithm unit inputs the processed YUV image into the JPEG encoding algorithm unit to execute the JPEG encoding algorithm. For details, see S129 to S133 below.

[0217] S129. The background algorithm module calls the ISP resource calling module to execute the JPEG encoding algorithm.

[0218] Optionally, when the background algorithm module calls the ISP resource calling module to execute the JPEG encoding algorithm, it can carry the processed YUV image.

[0219] S130 , the ISP resource calling module responds to the call of the background algorithm module and calls the ISP of the hardware layer to execute the JPEG encoding algorithm.

[0220] S131. The ISP executes a JPEG encoding algorithm on the processed YUV image in response to the call of the ISP resource calling module to obtain a JPEG true image (also called a third target image).

[0221] S132. The ISP returns the JPEG true image to the ISP resource calling module.

[0222] S133. The ISP resource calling module returns the JPEG true image to the background algorithm module.

[0223] Combine Figure 5 The specific execution process of steps S129 to S133 may include:

[0224] ① The JPEG encoding algorithm unit in the background algorithm module sends an instruction to call the JPEG encoding algorithm to the interface layer of the ISP resource calling module.

[0225] ② After the ISP resource calling module receives the calling instruction of the background algorithm module through the interface layer, it responds to the calling instruction, converts the processed YUV image into a format that can be recognized by the ISP platform through the JPEG encoding conversion unit, and performs data encapsulation. The obtained data is called the platform format YUV image.

[0226] ③The JPEG encoding conversion unit sends the YUV image in the platform format to the ISP driver through the ISP encapsulation interface.

[0227] ④The ISP driver sends the YUV image in the platform format to the ISP at the hardware layer.

[0228] ⑤ISP performs JPEG encoding algorithm on the YUV image in platform format to obtain a JPEG image in platform format.

[0229] ⑥ISP returns the platform format JPEG image to the JPEG encoding conversion unit in the ISP resource calling module through the ISP driver and ISP encapsulation interface.

[0230] ⑦The JPEG encoding conversion unit converts the JPEG image in the platform format to obtain a JPEG true image that the system can recognize.

[0231] ⑧The JPEG encoding conversion unit returns the JPEG true image to the JPEG encoding algorithm unit in the background algorithm module through the interface layer.

[0232] After that, the following step S134 is executed.

[0233] S134. The background algorithm module sends the JPEG true image to the AIDL interface of the application layer.

[0234] S135, AIDL interface sends the JPEG true image to the camera application.

[0235] S136. The camera application replaces the RGB thumbnail in the gallery with the JPEG real image.

[0236] It is understood that the image name of the JPEG true image generated by the background algorithm process is also the image name carried in the single-shot request in step S101. In other words, the image name of the JPEG true image generated by the background algorithm process is the same as the image name of the RGB thumbnail generated by the foreground algorithm process. The camera application can send the JPEG true image to the gallery application. The gallery application uses the image name of the JPEG true image to find the storage path of the RGB thumbnail with the same name. The gallery application then replaces the RGB thumbnail in the storage path with the JPEG true image.

[0237] In this embodiment, the foreground algorithm process enables rapid processing of RAW images. This allows for rapid entry into the next frame capture process upon receiving a request for a single capture, improving capture response speed. Simultaneously, the background algorithm process ensures comprehensive, high-quality image processing. The background algorithm process can be implemented by invoking the ISP hardware chip, enabling asynchronous, offline processing of the background algorithm. This eliminates the need for software system resources, reduces system load, further improves capture response speed, and enhances algorithm operational efficiency.

[0238] It is understandable that after the foreground algorithm process is completed, the background algorithm process may still be in execution. Therefore, when the user views the large image, the electronic device can prompt the user in the interface that the image is being processed, and inform the user in time to improve the user experience. After the background algorithm process is completed, the processed JPEG true image is further displayed to improve the display effect of the image and further improve the user experience. Figure 7 Provide explanation.

[0239] For example, Figure 7 This is a schematic diagram of an interface change provided in the embodiment of this application. Figure 7 As shown in Figure (a), the mobile phone displays the photo preview interface 701, and the user triggers the above steps S101 to S136 by clicking the photo control 3021. After the foreground algorithm process shown in the above steps S114 to S120 is completed, the mobile phone refreshes the interface 701 and obtains the photo preview interface 702, as shown in FIG. Figure 7 As shown in FIG. 3( b ) in FIG. 3( b ) in FIG. 3( c ...

[0240] In response to the user clicking the picture viewing control 3022, if the background algorithm process of the current frame image has not been completed, the mobile phone may display the following Figure 7 The prompt interface 703 shown in Figure (c) of FIG. The prompt interface 703 may include a prompt message "Image processing, please wait..." After the background algorithm process is completed, the interface can display the processed JPEG true image, such as Figure 7 As shown in interface 704 in Figure (d).

[0241] Example 2: HDR continuous shooting scene.

[0242] HDR technology can be used to achieve better results when capturing scenes with strong contrast between light and dark, requiring rich colors and details. Examples of HDR scenes include capturing strong light at cave entrances, outside windows, lights, or at sunrises and sunsets.

[0243] Optionally, during HDR shooting, you can use a long and short frame method to output images, for example, using single-frame high dynamic range (SHDR), dual analog gain (DAG), dual conversion gain (DCG), etc. to output images. When using the long and short frame method to output images, it is necessary to generate two RAW frames within one frame time, one long frame RAW image and one short frame RAW image, where the long frame RAW image refers to a long exposure RAW image, that is, an overexposed image; the short frame RAW image refers to a short exposure RAW image, that is, an underexposed image. Afterwards, the two RAW frames are fused to obtain an HDR image. The long and short frame RAW images can also be expressed as L / S images.

[0244] However, the fusion of long and short RAW frames is typically implemented using modules within the camera's algorithm library, specifically within the electronic device's software system. These fusion algorithms are often complex, resulting in heavy system load and slow processing speeds. Consequently, continuous shooting in HDR scenarios (referred to as HDR continuous shooting) is not possible with related technologies.

[0245] For example, Figure 8 This is a schematic diagram of an HDR continuous shooting interface in the related art. Figure 8 As shown in FIG. 8( a ), the mobile phone displays a photo preview interface 801. The photo preview interface 801 includes an HDR switch control 8011. In response to the user's selection operation (e.g., a click operation) on the HDR switch control 8011, the mobile phone activates the HDR function and displays the following: Figure 8 The photo preview interface 802 is shown in Figure (b). It can be seen that the HDR switch control 8011 in the photo preview interface 802 is in the turned-on state. In addition, the interface can also display a prompt 8021 indicating that the HDR function is turned on, for example: "Auto HDR mode is turned on". After that, the user aims the camera at a strong light and takes continuous shots. In response to the user long pressing the photo control 3021 in the photo mode, the mobile phone recognizes the HDR scene and recognizes the user's long pressing of the photo control 3021. It can display a prompt that HDR continuous shooting is not possible, such as Figure 8 Optionally, the mobile phone can process the user's long press of the photo control 3021 as a single shot operation, start HDR shooting, obtain a frame of HDR image, and update the thumbnail at the picture viewing control 3022, as shown in FIG. Figure 8 As shown in (c) in .

[0246] Continuous shooting is not possible in HDR scenarios, which brings inconvenience to users. In view of this, in this embodiment, through the foreground algorithm module and the ISP resource call module, the ISP is asynchronously called to realize the fusion and subsequent processing of long and short frame RAW images. The ISP processing can quickly realize image processing, reduce the system load, and shorten the processing time of a frame of HDR image, thus realizing HDR continuous shooting and improving the user experience.

[0247] For example, Figure 9 Another software architecture diagram provided in this application embodiment is as follows: Figure 9 As shown, in this embodiment, the image processing engine may include a foreground algorithm module and an ISP resource calling module. Among them, the foreground algorithm module may include an HDR fusion algorithm unit, a RAW2YUV algorithm unit and a JPEG encoding algorithm unit. The ISP resource calling module may include an HDR fusion conversion unit, a RAW2YUV conversion unit and a JPEG encoding conversion unit. Among them, the HDR fusion algorithm unit is used to execute the HDR fusion algorithm on the long and short frame RAW images. The HDR fusion conversion unit is used to convert the input data and output data of the HDR fusion algorithm. The functions of the RAW2YUV algorithm unit, the JPEG encoding algorithm unit, the RAW2YUV conversion unit and the JPEG encoding conversion unit are similar to those described in the above embodiment 1 and will not be repeated here.

[0248] For example, Figure 10 This is a flowchart of another image processing method provided in the embodiment of the present application. Figure 9 and Figure 10 In this embodiment, the image processing method includes:

[0249] S201: When the HDR function is turned on, the camera application of the application layer sends a continuous shooting request to the camera access interface of the application framework layer in response to the user long pressing the photo control in the photo mode.

[0250] Specifically, the camera app determines that the camera is currently in photo mode and that the user has long-pressed the photo control, thus generating a burst request. A burst request is used to request the continuous capture of multiple frames. The CONTROL_CAPTURE_INTENT parameter in the CaptureRequest of the burst request is set to STILL_CAPTURE, and the CaptureRequest includes a burst flag (e.g., BURST MOOD).

[0251] Optionally, the continuous shooting request may include the shooting mode, configuration parameters, and starting frame image information. In this embodiment, the shooting mode is "photo mode" and the configuration parameters include an HDR flag. The HDR flag is used to indicate that the HDR switch is on. The starting frame image information refers to the information of the first frame image among the multiple frames requested in the continuous shooting request. The starting frame image information may include the frame number of the starting frame image and the name of the starting frame image.

[0252] Similar to the first embodiment, in shooting modes other than the video recording mode, the camera application can also generate a continuous shooting request when the user performs a long press operation. The difference is that the shooting mode and configuration parameters carried in the generated continuous shooting request can be different.

[0253] S202. The camera access interface of the application framework layer responds to the continuous shooting request, generates a single shooting request 1 according to the continuous shooting request, and sends the single shooting request 1 to the front-end image output module in the camera hardware abstraction layer.

[0254] Specifically, after the camera access interface receives the continuous shooting request, it can send a single shooting request to the front-end image output module at intervals of a preset duration until the user raises his hand. Each single shooting request is used to request the capture of one frame of image. Optionally, the preset duration can be equal to the frame interval of the electronic device (i.e., 1 / frame rate). Similar to the single shooting request in Example 1, each single shooting request in this embodiment can also carry shooting mode, configuration parameters, frame image information, etc. Different from Example 1, each single shooting request in this embodiment also carries a continuous shooting identifier.

[0255] For ease of distinction, the first single-shot request sent by the camera access interface is denoted as single-shot request 1, the second single-shot request is denoted as single-shot request 2, and so on. Single-shot request 1 is the first single-shot request sent in response to a continuous shooting request, and the frame image information it carries is the starting frame image information in the continuous shooting request.

[0256] S203. The front-end image output module responds to the single-shot request 1 and calls the decision module in the image processing engine to make a photo-taking decision.

[0257] S204. The decision module decides the photographing parameter 2 and the foreground algorithm type 2 according to the single-shot request 1.

[0258] It should be understood that after the decision module receives the first single-shot request 1, it can determine that the current scene is an HDR scene based on the HDR identifier in the single-shot request 1, and can determine that the current scene is a continuous shooting scene based on the continuous shooting identifier in the single-shot request 1, and further determine that the current scene is an HDR continuous shooting scene. Based on this, the decision module can decide to generate the shooting parameters 2 corresponding to the HDR continuous shooting scene, and decide the algorithm type corresponding to the HDR continuous shooting scene. Among them, the shooting parameters 2 include the HDR image output method. In this embodiment, the SHDR image output method is used as an example for explanation, and the shooting parameters 2 may include the SHDR identifier. The SHDR identifier is used to indicate that the image is output in the SHDR mode.

[0259] In this embodiment, the algorithm type corresponding to the HDR continuous shooting scene is recorded as foreground algorithm type 2, and the algorithm corresponding to foreground algorithm type 2 is recorded as foreground algorithm 2. Optionally, foreground algorithm 2 may include an HDR fusion algorithm, a RAW2YUV algorithm, and a JPEG encoding algorithm.

[0260] S205. The decision module sends the foreground algorithm type 2 to the foreground algorithm module.

[0261] S206 , the foreground algorithm module loads foreground algorithm 2 according to foreground algorithm type 2.

[0262] like Figure 9 As shown, after the foreground algorithm module loads foreground algorithm 2, the module may include an HDR fusion algorithm unit, a RAW2YUV algorithm unit, and a JPEG encoding algorithm unit. The HDR fusion algorithm unit is used to execute the HDR fusion algorithm. The RAW2YUV algorithm unit is used to execute the RAW2YUV algorithm. The JPEG encoding algorithm unit is used to execute the JPEG encoding algorithm.

[0263] S207. The decision module sends the photographing parameter 2 to the front-end image output module.

[0264] S208. The front-end image output module sends the photographing parameter 2 to the camera of the hardware layer.

[0265] S209 : The camera captures a first set of long and short frame RAW images according to photographing parameter 2 .

[0266] S210: The camera returns the first set of long and short frame RAW images to the front-end image output module.

[0267] S211 , the front-end image output module sends the first set of long and short frame RAW images to the back-end processing module.

[0268] S212. The backend processing module calls the foreground algorithm module to execute foreground algorithm 2.

[0269] Optionally, when the backend processing module calls the foreground algorithm module to execute the foreground algorithm 2, it can carry the first group of long and short frame RAW images.

[0270] S213. The foreground algorithm module responds to the call of the backend processing module and calls the ISP resource calling module to execute the HDR fusion algorithm.

[0271] Optionally, when the background algorithm module calls the ISP resource calling module to execute the HDR fusion algorithm, it can carry the first set of long and short frame RAW images.

[0272] S214. The ISP resource calling module responds to the call of the background algorithm module and calls the ISP of the hardware layer to execute the HDR fusion algorithm.

[0273] S215, ISP performs HDR fusion algorithm on the first set of long and short frame RAW images to obtain fused RAW Figure 1 (Also called the first fused image).

[0274] S216, ISP will integrate RAW Figure 1 Return to the ISP resource calling module.

[0275] S217, ISP resource call module will integrate RAW Figure 1 Return to the front-end algorithm module.

[0276] For details, see Figure 9 The specific execution process of steps S213 to S217 may include:

[0277] ① The HDR fusion algorithm unit in the foreground algorithm module sends an instruction to call the HDR fusion algorithm to the interface layer of the ISP resource calling module.

[0278] ② After the ISP resource calling module receives the calling instruction from the foreground algorithm module through the interface layer, in response to the calling instruction, it converts the first set of long and short frame RAW images into a format that can be recognized by the ISP platform through the HDR fusion conversion unit, and performs data encapsulation. The obtained data is called long and short frame RAW images in the platform format (also called multi-frame conversion images).

[0279] ③The HDR fusion conversion unit sends the long and short frame RAW images in the platform format to the ISP driver through the ISP encapsulation interface.

[0280] ④The ISP driver sends the long and short frame RAW images in the platform format to the ISP at the hardware layer.

[0281] ⑤ISP performs HDR fusion algorithm on the long and short frame RAW images in the platform format to obtain the fused RAW images in the platform format. Figure 1 (also called the original fused image).

[0282] ⑥ISP integrates RAW format into the platform Figure 1It is returned to the HDR fusion conversion unit in the ISP resource calling module through the ISP driver and ISP encapsulation interface.

[0283] ⑦HDR fusion conversion unit to platform format fusion RAW Figure 1 Convert to Fusion RAW Figure 1 (Also called the first fused image).

[0284] ⑧RAW2YUV conversion unit will fuse RAW Figure 1 Return to the HDR fusion algorithm unit in the foreground algorithm module through the interface layer.

[0285] After that, the HDR fusion algorithm unit will fuse the RAW Figure 1 The data is input to the RAW2YUV algorithm unit to execute the RAW2YUV algorithm, that is, to execute the following step S218.

[0286] S218. The foreground algorithm module calls the ISP resource calling module to execute the RAW2YUV algorithm.

[0287] Optionally, when the foreground algorithm module calls the ISP resource calling module to execute the RAW2YUV algorithm, it can carry the fusion RAW Figure 1 .

[0288] S219. The ISP resource calling module responds to the call of the foreground algorithm module and calls the ISP of the hardware layer to execute the RAW2YUV algorithm.

[0289] Optionally, when the ISP resource calling module calls the ISP to execute the RAW2YUV algorithm, it can carry the fusion RAW Figure 1 .

[0290] S220, ISP to fusion RAW Figure 1 Execute the RAW2YUV algorithm to get the fused YUV Figure 1 (Also called the first YUV map).

[0291] S221, ISP will integrate YUV Figure 1 Return to the ISP resource calling module.

[0292] S222, ISP resource calling module will integrate YUV Figure 1 Return to the front-end algorithm module.

[0293] The above steps S218 to S222 are similar to steps S123 to S127 in the first embodiment and are not described in detail again.

[0294] S223. The foreground algorithm module calls the ISP resource calling module to execute the JPEG encoding algorithm.

[0295] Optionally, when the foreground algorithm module calls the ISP resource calling module to execute the JPEG encoding algorithm, it can carry the fused YUV Figure 1 .

[0296] S224. The ISP resource calling module responds to the call of the foreground algorithm module and calls the ISP of the hardware layer to execute the JPEG encoding algorithm.

[0297] Optionally, when the ISP resource calling module calls the ISP to execute the JPEG encoding algorithm, it can carry the fused YUV Figure 1 .

[0298] S225, ISP for fusion YUV Figure 1 Execute the JPEG encoding algorithm to get the HDR JPEG true Figure 1 (Also called the first target image).

[0299] S226, ISP will integrate HDR JPEG true Figure 1 Return to the ISP resource calling module.

[0300] S227, ISP resource calling module converts HDR JPEG to Figure 1 Return to the front-end algorithm module.

[0301] The above steps S223 to S227 are similar to steps S129 to S133 in the first embodiment and are not described in detail.

[0302] S228, the foreground algorithm module converts HDR JPEG into Figure 1 Send to the back-end processing module.

[0303] S229, the back-end processing module converts HDR JPEG into Figure 1 Sent to the camera access interface.

[0304] S230, camera access interface will HDR JPEG true Figure 1 Send to Camera app.

[0305] S231, camera application based on HDR JPEG true Figure 1 Refresh the photo preview interface.

[0306] S232, Camera application will HDR JPEG true Figure 1 Save to gallery.

[0307] Steps S213 to S232 above describe the process of generating an HDR JPEG true image from a RAW image. As can be seen, in this embodiment, the image processing engine can invoke the ISP to process both long and short frame RAW images (multiple frames) to produce an HDR JPEG (one frame). This means invoking the ISP's capabilities to achieve multi-input, single-output processing, improving image processing speed.

[0308] In addition, the above steps S202 to S207 can be understood as the process of photographing parameters and algorithm decision-making. Steps S208 to S232 can be understood as the process of photographing a frame of HDR JPEG true image (also known as the first process).

[0309] After a preset time period after sending the photo request 1, the camera access interface may execute step S233 to trigger the capture of the second HDR JPEG frame. Figure 2 Shooting.

[0310] S233: At a preset time interval from the photo request 1, the camera access interface sends a single photo request 2 to the front-end image output module.

[0311] Single-shot request 2 is similar to single-shot request 1 and will not be described in detail.

[0312] The electronic device responds to the single shot request 2 and repeats the above steps S208 to S232 to complete the second frame HDR JPEG true Figure 2 By analogy, the electronic device sends single-shot requests in sequence to complete the shooting of each frame of HDR JPEG true image until the user raises his hand and executes the following step S234:

[0313] S234: In response to the user's hand-raising operation, the camera application sends a request to stop continuous shooting to the camera access interface.

[0314] The continuous shooting stop request is used to request to stop continuous shooting.

[0315] The camera access interface responds to the request to stop continuous shooting, stops sending single shooting requests to the front-end image output module, and stops shooting.

[0316] It can be understood that in the process of executing the processing of the nth frame HDR JPEG true image n through the above steps S213 to S227, the acquisition of the long and short frame RAW images of the n+1th frame HDR JPEG true image n+1 can be executed simultaneously through steps S208 to S212, and the two processes do not affect each other. Wherein, n is an integer greater than or equal to 1. That is to say, the image processing method provided by this embodiment, in the HDR scenario, asynchronously calls the ISP capability through the ISP resource calling module to realize the fusion of the long and short frame RAW images to obtain the HDR JPEG true image. In this way, on the one hand, for the shooting of any two adjacent frames of HDR images, the acquisition of the RAW image corresponding to the latter frame HDR image and the processing of the previous frame HDR image can reuse time, thereby shortening the shooting time of the two frames of HDR images; on the other hand, the hardware chip ISP can realize the processing of the long and short frame RAW images more quickly without occupying software system resources, which can reduce the system load and further shorten the processing time of each frame of HDR image. By shortening the shooting time of HDR images through the above two aspects, electronic devices can complete the continuous shooting of multiple frames of HDR images in a timely manner, that is, realize HDR continuous shooting, and improve user experience.

[0317] The following is an explanation of the interface diagram corresponding to HDR continuous shooting. For example, Figure 11 This is a schematic diagram of the interface changes of an HDR continuous shooting example provided in the embodiment of the present application. Figure 11 As shown in Figure (a), when the HDR function is turned on, the phone displays a photo preview interface 1101. In response to the user's long press operation on the photo control 3021, the phone starts taking photos according to the above process. Every time a frame of HDR JPEG true image is taken, the thumbnail in the picture viewing control 3022 is refreshed, and the number of frames of the currently completed HDR JPEG true image is displayed in the interface. Figure 11 As shown in FIG. 11 (b), after the 15th HDR JPEG true image is completed, the photo preview interface may be shown as interface 1102. It can be seen that the currently displayed frame number 1121 in interface 1102 is "15", and the thumbnail in the image viewing control 3022 is refreshed to the thumbnail corresponding to the 15th HDR JPEG true image.

[0318] After a while, in response to the user raising his hand, the phone stops shooting and displays the following Figure 11 The interface 1103 shown in FIG. (c) in FIG. Among them, the thumbnail in the picture viewing control 3022 is the thumbnail corresponding to the last frame of HDR JPEG true image. In response to the user's selection operation on the picture viewing control 3022, the mobile phone displays the last frame of HDR JPEG true image, such as Figure 11 In response to the user's single click operation in the interface 1104, the mobile phone displays Figure 11Interface 1105 is shown in Figure (e) of FIG. In addition to the last HDR JPEG true image, interface 1105 also includes related information about the image, including a continuous shooting icon 1151. Continuous shooting icon 1151 indicates that the current image is a frame (typically the last frame) in a continuous shooting sequence. This shows that continuous shooting is possible in the HDR scene in this embodiment.

[0319] In addition, each HDR JPEG true image during continuous shooting is saved to the gallery, so the continuous shooting images can also be viewed through the gallery application. Figure 12 Another example interface diagram provided in the embodiment of the present application is as follows: Figure 12 As shown in FIG. 1 (a), the mobile phone displays a desktop 301. The desktop 301 includes an icon 3012 of the gallery application. In response to the user's selection operation on the icon 3012 of the gallery application, the mobile phone displays an interface 1201 of the gallery application, as shown in FIG. Figure 12 As shown in FIG. (b) of FIG. 1201, the interface of the gallery application includes a camera album 1211. In response to the user's selection operation on the camera album 1211, the mobile phone displays the camera album interface 1203, as shown in FIG. Figure 12 As shown in Figure (c) of FIG. The camera album interface 1202 includes Figure 11 A thumbnail 1221 corresponding to the HDRJPEG true image captured in the image is shown. A continuous shooting icon 1222 is displayed on the thumbnail 1221. The continuous shooting icon 1222 is used to indicate that the current image is a frame (usually the last frame) in the continuous shooting image. As can be seen, the method provided in this embodiment can achieve continuous shooting in HDR scenes.

[0320] Furthermore, as can be seen from the above steps, the image processing method provided in this embodiment does not crop the long and short frame RAW images during processing, thus maintaining the integrity of the long and short frame RAW image information. This improves the image quality of the resulting HDR JPEG true image and enhances the user experience. This is explained below with reference to the accompanying figures.

[0321] For example, Figure 13 This is another example of interface change diagram provided in the embodiment of the present application. Figure 13 As shown in Figure (a), the mobile phone displays the camera album interface 1202. The camera album interface 1202 includes thumbnails 1221 corresponding to the HDR JPEG true images obtained by HDR continuous shooting. In response to the user's selection operation on the thumbnail 1221, the mobile phone displays the last frame of the HDR JPEG true image and related information of the image, such as Figure 13 The interface 1105 in FIG. (b) includes a continuous shooting icon 1151 and a detailed information control 1152. In response to the user's selection operation on the detailed information control 1152, the mobile phone displays an interface 1301, as shown in FIG. Figure 13 As shown in Figure (c) of the figure, interface 1301 includes a detailed information card 1311 for the current HDR JPEG true image. This detailed information card 1311 includes size information 1312 for the frame. As can be seen, the size of the current HDR JPEG true image is 3072×4096. This shows that the method provided in this embodiment does not crop the long and short RAW images, maintaining the original image size (i.e., the same size as the RAW image), thus ensuring the integrity of the image information.

[0322] It should be noted that Example 2 uses continuous shooting in an HDR scene as an example. In other embodiments, when performing a single shot in an HDR scene, the ISP can also be called to execute the HDR fusion algorithm, RAW2YUV algorithm, and JPEG encoding algorithm according to the process of Example 2 to improve image processing speed, thereby improving photo response and user experience.

[0323] Example 3: Converting RAW images to JPEG images in non-photography scenarios.

[0324] In addition to being applicable to the two photography scenarios described in Examples 1 and 2 above, the image processing method provided in this application can also be applied to non-photography scenarios. Specifically, the method can be used to convert RAW images into images of other formats in non-photography scenarios, i.e., it supports format conversion of RAW images in offline situations. Optionally, the method can also convert RAW images into JPEG images, PNG images, etc. In this embodiment, JPEG images are used as an example for illustration.

[0325] It is understood that in some scenarios, the format of the picture obtained by the electronic device is a RAW picture, for example, a picture taken in professional mode. Figure 14 This is another example of interface change diagram provided in the embodiment of the present application. Figure 14 As shown in FIG. (a), the mobile phone displays a photo preview interface 1401 in the photo mode. In response to the user performing a left swipe operation in the mode selection area in the photo preview interface 1401, the shooting mode can be switched to the professional mode, and the following is displayed: Figure 14 The interface 1402 shown in FIG. (b) in FIG. 1402 includes "JPEG" format information 1421, indicating that the currently selected image format is JPEG format. In response to the user's selection operation of "JPEG format information 1421", the mobile phone displays the following Figure 14 Interface 1403 is shown in FIG. (c) of FIG. 1403. Different from interface 1402, interface 1403 displays multiple format options, including a "RAW" format option 1431. In response to the user's selection of the "RAW" format option 1431, the mobile phone switches the image format to RAW format and displays the following: Figure 14The interface 1404 shown in FIG. 1404 includes “RAW” format information 1441 , indicating that the currently selected image format is the RAW format.

[0326] In the interface 1404, in response to the user's single click operation on the photo control 3021, the mobile phone takes a photo based on the professional mode, captures and saves the image in RAW format. The image in RAW format records the original information of the photosensitive element, so the file size is relatively large. For example, Figure 15 This is another example interface diagram provided in the embodiment of the present application. Figure 14 The image and detailed information obtained by performing the photo-taking operation on the interface 1404 shown in FIG. (d) can be as follows: Figure 15 , as shown in interface 1501. Interface 1501 includes a detailed information card 1511, which includes, among other things, the image name 1512, the storage path 1513, and image volume information 1514. Image name 1512 shows the image extension .dng, indicating that it is in RAW format. Furthermore, storage path 1513 is "Internal Storage / DCIM / Camera / RAW / ," also confirming that it is in RAW format. Image volume information 1514 indicates that the image occupies 25.20 MB of storage space, indicating that the image is quite large.

[0327] RAW format images are large in size, and in some scenarios, users need smaller images for easier transmission, downloading, viewing, etc. However, in related technologies, it is impossible to convert RAW format images, which is inconvenient for users.

[0328] In view of this, the image processing method provided in this embodiment supports format conversion of RAW images.

[0329] For example, Figure 16 This is a schematic diagram of an interface change for an example of image format conversion provided in the embodiment of the present application. Figure 16 As shown in FIG. 1 (a), the mobile phone displays an image interface 1601. The image interface 1601 includes a RAW image 1611 and more controls 1612. In response to the user's selection of the more controls 1611, the mobile phone displays the following Figure 16 Interface 1602 (also referred to as the first interface) shown in FIG. Compared to image interface 1601, interface 1602 includes multiple image operation items, such as a "move" operation item, a "copy" operation item, and a "convert to JPEG" operation item 1621 (also referred to as a second control).

[0330] In response to the user's selection of the "Convert to JPEG" operation item 1621, the mobile phone starts to perform format conversion processing on the RAW image 1611. During this process, the mobile phone can display the following Figure 16 The interface 1603 shown in Figure (c) in the figure is displayed to remind the user that the image is currently being processed. After the format conversion is completed, the mobile phone can display the following Figure 16 The image interface 1604 shown in FIG. (d) includes the converted JPEG image 1641. The image interface 1604 also includes a detailed information control 1152. In response to the user's selection operation on the detailed information control 1152, the mobile phone displays the JPEG image 1641 and the detailed information card 1651. Figure 16 The interface 1605 in (e) of FIG. Detailed information card 1651 includes image name 1652, storage path 1653 and image volume information 1654. Among them, in the image name 1652, the image extension is .jpg, which shows that the image format is JPEG format. In addition, the storage path 1653 is Figure 15 The storage path 1513 in the image has changed, and it can be seen that the image format is no longer RAW. From the image volume information 1654, it can be seen that the image occupies 1.96MB of storage space, which is significantly smaller than that of the image. Figure 15 The size of the medium image is 25.20MB, and the size of the JPEG format image is smaller.

[0331] The implementation process of the image processing method provided by this embodiment is described below.

[0332] For example, Figure 17 This is another software architecture diagram provided in the embodiment of the present application. In this embodiment, the image processing engine includes a background algorithm module and an ISP resource calling module. The structure of the background algorithm module and the ISP resource calling module is the same as that in the embodiment 1. Figure 5 The structures of the mid- and back-end algorithm modules and the ISP resource calling module can be the same and will not be described in detail here.

[0333] For example, Figure 18 This is a flowchart of another image processing method provided in the embodiment of the present application. Figure 17 and Figure 18 In this embodiment, the image processing method may include:

[0334] S301 : In response to a user's operation of converting a RAW image to be converted (also referred to as a first RAW image) to JPEG format, the image library application sends the RAW image to be converted to an AIDL interface.

[0335] For example, the RAW image to be converted is Figure 16The RAW image 1611 in the image is converted to JPEG format. Figure 16 As shown in , a selection operation (such as a click operation, also referred to as a second operation) is performed on the “Convert to JPEG” operation item 1621 .

[0336] S302: The AIDL interface sends the RAW image to be converted to the background algorithm module.

[0337] S303 : The background algorithm module executes the background RAW domain algorithm on the RAW image to be converted to obtain a processed RAW image to be converted.

[0338] S304: The background algorithm module calls the ISP resource calling module to execute the RAW2YUV algorithm.

[0339] Optionally, when the background algorithm module calls the ISP resource calling module to execute the RAW2YUV algorithm, it can carry the processed RAW image to be converted.

[0340] S305 , the ISP resource calling module responds to the call of the background algorithm module and calls the ISP of the hardware layer to execute the RAW2YUV algorithm.

[0341] S306 , the ISP executes a RAW2YUV algorithm on the processed RAW image to be converted in response to the call of the ISP resource calling module, to obtain a YUV image to be converted.

[0342] S307: The ISP returns the YUV image to be converted to the ISP resource calling module.

[0343] S308. The ISP resource calling module returns the YUV image to be converted to the background algorithm module.

[0344] S309 , the background algorithm module executes a YUV domain algorithm on the YUV image to be converted to obtain a processed YUV image to be converted.

[0345] S310: The background algorithm module calls the ISP resource calling module to execute the JPEG encoding algorithm.

[0346] Optionally, when the background algorithm module calls the ISP resource calling module to execute the JPEG encoding algorithm, it can carry the processed YUV image to be converted.

[0347] S311 , the ISP resource calling module responds to the call of the background algorithm module and calls the ISP of the hardware layer to execute the JPEG encoding algorithm.

[0348] S312. The ISP executes a JPEG encoding algorithm on the processed YUV image to be converted in response to the call of the ISP resource calling module, to obtain a converted JPEG true image (also referred to as a second target image).

[0349] S313. The ISP returns the converted JPEG true image to the ISP resource calling module.

[0350] S314. The ISP resource calling module returns the converted JPEG true image to the background algorithm module.

[0351] The above steps S303 to S314 are similar to steps S122 to S133 in the first embodiment and are not described in detail again.

[0352] S315. The background algorithm module sends the converted JPEG true image to the AIDL interface.

[0353] S316. The AIDL interface sends the converted JPEG true image to the photo album application.

[0354] S317, the photo album application saves the converted JPEG true image.

[0355] The image processing method provided in this embodiment can also call ISP to perform secondary processing on the RAW image after taking the photo, thereby realizing format conversion of the RAW image offline, meeting the user's requirements for image format, and improving user experience.

[0356] It should be noted that the above embodiments 1, 2, and 3 respectively describe image processing methods for three scenarios. However, if the solutions do not conflict, the three embodiments can be combined. For example, after completing the shooting according to the process shown in embodiment 1 in a single shot scenario, HDR continuous shooting can be further performed according to the process shown in embodiment 2, or the format of the RAW image can be further converted according to the process shown in embodiment 3. This application does not make specific limitations on this.

[0357] The above describes in detail an example of the image processing method provided by the embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0358] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0359] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0360] The electronic device provided in this embodiment is used to execute the above-mentioned image processing method, and thus can achieve the same effect as the above-mentioned implementation method.

[0361] When integrated, the electronic device may also include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the operation of the electronic device. The storage module may be used to support the execution of program code and data stored in the electronic device. The communication module may be used to support communication between the electronic device and other devices.

[0362] The processing module may be a processor or a controller. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor (DSP) and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0363] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 1 Device with the structure shown.

[0364] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the image processing method of any of the above embodiments.

[0365] An embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0366] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the image processing method in the above-mentioned method embodiments.

[0367] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0368] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0369] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0370] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0371] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0372] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0373] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, the method being executed by an electronic device, characterized in that: The electronic device includes an image sensor and an image signal processor (ISP), and the method includes: Displaying a photo preview interface of a camera application, wherein the photo preview interface includes a first thumbnail, and the HDR function of the camera application is enabled; receiving a first operation from a user, where the first operation is used to instruct to continuously capture multiple frames of images; In response to the first operation, executing the first process once every preset time interval until receiving a second operation from the user, wherein the second operation is used to instruct to stop shooting; The first process includes: Acquire multiple frames of original images through the image sensor; Performing HDR fusion, conversion, and encoding on the multiple frames of original images through the ISP to obtain a first target image; The first target image is saved, and the first thumbnail is refreshed as a thumbnail of the first target image.

2. The method according to claim 1, characterized in that The original image is an image in RAW format, and the ISP performs HDR fusion, conversion, and encoding on the multiple frames of original images to obtain a first target image, including: Executing an HDR fusion algorithm on the multiple frames of original image through the ISP to obtain a first fused image; Performing a RAW domain to YUV domain conversion algorithm on the first fused image through the ISP to obtain a first YUV image; The ISP performs an encoding algorithm on the first YUV image to obtain the first target image.

3. The method according to claim 2, characterized in that The electronic device includes a foreground algorithm module and an ISP resource calling module, and the ISP performs an HDR fusion algorithm on the multiple frames of original images to obtain a first fused image, including: The foreground algorithm module calls the ISP resource calling module to execute the HDR fusion algorithm on the multiple frames of original images; The ISP resource calling module performs data conversion and encapsulation on the multiple frames of original images to obtain multiple frames of converted images; The ISP resource calling module calls the ISP to execute the HDR fusion algorithm on the multi-frame converted images; The ISP performs the HDR fusion algorithm on the multiple frames of converted images to obtain an original fused image; The ISP returns the original fused image to the ISP resource calling module; The ISP resource calling module performs data conversion on the original fused image to obtain the first fused image; The ISP resource calling module returns the first fused image to the foreground algorithm module.

4. The method according to claim 2 or 3, characterized in that The electronic device includes a foreground algorithm module and an ISP resource calling module, and the ISP performs a RAW domain to YUV domain conversion algorithm on the first fused image to obtain a first YUV image, including: The foreground algorithm module calls the ISP resource calling module to execute the RAW domain to YUV domain conversion algorithm for the first fused image; The ISP resource calling module performs data conversion and encapsulation on the first fused image to obtain a fused converted image; The ISP resource calling module calls the ISP to execute the RAW domain to YUV domain algorithm for the fused converted image; The ISP performs the RAW domain to YUV domain conversion algorithm on the fused converted image to obtain a first original YUV image; The ISP returns the first original YUV image to the ISP resource calling module; The ISP resource calling module performs data conversion on the first original YUV image to obtain the first YUV image; The ISP resource calling module returns the first YUV image to the foreground algorithm module.

5. The method according to any one of claims 2 to 4, characterized in that The electronic device includes a foreground algorithm module and an ISP resource calling module, and the encoding algorithm is executed on the first YUV image by the ISP to obtain the first target image, including: The foreground algorithm module calls the ISP resource calling module to execute the encoding algorithm for the first YUV image; The ISP resource calling module performs data conversion and encapsulation on the first YUV image to obtain a first YUV conversion image; The ISP resource calling module calls the ISP to execute the encoding algorithm for the first YUV conversion image; The ISP executes the encoding algorithm on the first YUV conversion image to obtain a first original target image; The ISP returns the first original target image to the ISP resource calling module; The ISP resource calling module performs data conversion on the first original target image to obtain the first target image; The ISP resource calling module returns the first target image to the foreground algorithm module.

6. The method according to any one of claims 2 to 5, characterized in that The encoding algorithm is a JPEG encoding algorithm, and the first target image is an image in JPEG format.

7. The method according to any one of claims 1 to 6, characterized in that The multiple frames of original images include a first original image and a second original image, the first original image is an overexposed image, and the second original image is an underexposed image.

8. The method according to any one of claims 1 to 7, characterized in that The size of the first target image is the same as that of the original image.

9. The method according to any one of claims 1 to 8, characterized in that The photo preview interface includes a first control, the first operation is an operation of long pressing the first control, and the second operation is an operation of stopping long pressing the first control.

10. An image processing method, executed by an electronic device, characterized in that: The electronic device includes an application layer, a hardware abstraction layer, and a hardware layer, the application layer includes a gallery application and an AIDL interface, the hardware abstraction layer includes an image processing engine, and the hardware layer includes an image signal processor (ISP). The method includes: Displaying a first interface of the gallery application, wherein the first interface includes a first RAW image and a second control; In response to a user selecting the second control, the gallery application sends the first RAW image to the image processing engine through the AIDL interface; The image processing engine calls the ISP to perform format conversion on the first RAW image to obtain a second target image.

11. The method according to claim 10, characterized in that The image processing engine calls the ISP to perform format conversion on the first RAW image to obtain a second target image, including: The image processing engine performs a RAW domain algorithm on the first RAW image to obtain a second RAW image; The image processing engine calls the ISP to perform a RAW domain to YUV domain conversion algorithm on the second RAW image to obtain a second YUV image; The image processing engine performs a YUV domain algorithm on the second YUV image to obtain a third YUV image; The image processing engine calls the ISP to execute an encoding algorithm on the third YUV image to obtain the second target image.

12. The method according to claim 11, characterized in that The image processing engine includes a background algorithm module and an ISP resource calling module; The image library application sends the first RAW image to the image processing engine through the AIDL interface, including: The image library application sends the first RAW image to the background algorithm module through the AIDL interface; The image processing engine performs a RAW domain algorithm on the first RAW image to obtain a second RAW image, including: The background algorithm module performs a RAW domain algorithm on the first RAW image to obtain the second RAW image; The image processing engine performs a YUV domain algorithm on the second YUV image to obtain a third YUV image, including: The background algorithm module performs a YUV domain algorithm on the second YUV image to obtain the third YUV image.

13. The method according to claim 12, characterized in that The image processing engine calls the ISP to perform a RAW domain to YUV domain conversion algorithm on the second RAW image to obtain a second YUV image, including: The background algorithm module calls the ISP resource calling module to execute the RAW domain to YUV domain conversion algorithm for the second RAW image; The ISP resource calling module performs data conversion and encapsulation on the second RAW image to obtain a second RAW conversion image; The ISP resource calling module calls the ISP to execute the RAW domain to YUV domain algorithm for the second RAW conversion graph; The ISP performs the RAW domain to YUV domain conversion algorithm on the second RAW conversion image to obtain a second original YUV image; The ISP returns the second original YUV image to the ISP resource calling module; The ISP resource calling module performs data conversion on the second original YUV image to obtain the second YUV image; The ISP resource calling module returns the second YUV image to the background algorithm module.

14. The method according to claim 12 or 13, characterized in that The image processing engine calls the ISP to execute an encoding algorithm on the third YUV image to obtain the second target image, including: The background algorithm module calls the ISP resource calling module to execute the encoding algorithm for the third YUV image; The ISP resource calling module performs data conversion and encapsulation on the third YUV image to obtain a third YUV conversion image; The ISP resource calling module calls the ISP to execute the encoding algorithm for the third YUV conversion image; The ISP executes the encoding algorithm on the third YUV conversion image to obtain a second original target image; The ISP returns the second original target image to the ISP resource calling module; The ISP resource calling module performs data conversion on the second original target image to obtain the second target image; The ISP resource calling module returns the second target image to the background algorithm module.

15. The method according to any one of claims 10 to 14, characterized in that The second target image is an image in JPEG format or PNG format.

16. An image processing method, the method being executed by an electronic device, characterized in that: The electronic device includes an image sensor and an image signal processor (ISP), and the method includes: Displaying a photo preview interface of a camera application, wherein the photo preview interface includes a first thumbnail; receiving a third operation of the user, where the third operation is used to instruct to capture a frame of image; In response to the third operation, acquiring a third RAW image through the image sensor; performing a first processing on the third RAW image to obtain a second thumbnail; Saving the second thumbnail and refreshing the first thumbnail to the second thumbnail; performing a second processing on the third RAW image to obtain a third target image, wherein a duration of the first processing is shorter than a duration of the second processing, and the second processing includes executing a RAW domain-to-YUV domain algorithm and an encoding algorithm through the ISP; The second thumbnail is replaced by the third target image.

17. The method according to claim 16, characterized in that The second thumbnail is an RGB image, and performing the first processing on the third RAW image to obtain the second thumbnail includes: executing the first RAW domain algorithm on the third RAW image to obtain a fourth RAW image; A RAW domain to RGB domain conversion algorithm is performed on the fourth RAW image to obtain the second thumbnail.

18. The method according to claim 17, characterized in that The number of the third RAW image is one or more frames, and performing the first RAW domain algorithm on the third RAW image to obtain the fourth RAW image includes: The first RAW domain algorithm is executed on a target RAW image to obtain the fourth RAW image, where the target RAW image is one frame of the one or more frames of the third RAW image.

19. The method according to any one of claims 16 to 18, characterized in that The electronic device includes an image processing engine, and performing the second processing on the third RAW image to obtain a third target image includes: The image processing engine performs a second RAW domain algorithm on the third RAW image to obtain a fifth RAW image; The image processing engine calls the ISP to perform a RAW domain to YUV domain conversion algorithm on the fifth RAW image to obtain a fourth YUV image; The image processing engine performs a YUV domain algorithm on the fourth YUV image to obtain a fifth YUV image; The image processing engine calls the ISP to execute an encoding algorithm on the fifth YUV image to obtain the third target image.

20. The method according to claim 19, wherein The electronic device further includes an application layer, the camera application is located in the application layer, and the application layer further includes an AIDL interface. Replacing the second thumbnail with the third target image includes: The image processing engine sends the third target image to the camera application through the AIDL interface; The camera application replaces the second thumbnail with the third target image.

21. The method according to claim 19 or 20, characterized in that The number of the third RAW image is one or more frames, and the image processing engine performs a second RAW domain algorithm on the third RAW image to obtain a fifth RAW image, including: The image processing engine executes the second RAW domain algorithm on all the third RAW images to obtain the fifth RAW image.

22. The method according to any one of claims 19 to 21, characterized in that The encoding algorithm is a JPEG encoding algorithm, and the third target image is an image in JPEG format.

23. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to perform the method according to any one of claims 1 to 22.

24. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 22.

25. A computer-readable storage medium, characterized in that The computer-readable storage medium comprises instructions, which, when executed on an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 22.

Images