Image processing method and device

Abstract

The invention discloses an image processing method and device, and belongs to the technical field of electronics. The method comprises the steps that live image frames are collected, a collected first image frame sequence of a first duration and sensor data are cached through an electronic image stable annular frame buffering queue in a hardware abstraction layer of the electronic equipment, and the sensor data comprise motion information of the electronic equipment in the process of collecting the first image frame sequence; after shooting input is received, the first image frame sequence and the sensor data are acquired, and a second image frame sequence of a second duration is acquired; calculating a jitter compensation parameter through an algorithm hardware module of the electronic equipment according to the first image frame sequence and the sensor data; and performing image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameter to generate a first live image.

Description

Technical Field

[0001] This application belongs to the field of electronic technology, specifically relating to an image processing method and apparatus. Background Technology

[0002] Live images, which are dynamic pictures that include image frames captured by the camera in the period before and after the user triggers the shutter, are becoming increasingly popular with users.

[0003] In related technologies, when an electronic device displays a live image preview interface, it can capture and cache a sequence of image frames for a preset duration at the application layer. Upon detecting a user triggering the shutter, it acquires both the cached sequence of image frames for the preset duration and a sequence of image frames occurring within a short period after the user triggers the shutter. Furthermore, the electronic device performs shake compensation processing on these two image frame sequences using image stabilization software algorithms.

[0004] However, since the anti-shake software algorithm relies on image analysis to perform shake compensation processing on the image frame sequence, the processing method is relatively passive, resulting in a large amplitude of image shake in the final live image generated by the electronic device. Therefore, in related technologies, the effect of the live image generated by the electronic device is poor. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method and apparatus that can improve the quality of live images generated by electronic devices.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring live image frames, and caching a first image frame sequence and sensor data of a first duration acquired through an Electronic Image Stabilization (EIS) circular frame buffer queue in the Hardware Abstraction Layer (HAL) of an electronic device, the sensor data including motion information of the electronic device during the acquisition of the first image frame sequence; upon receiving a shooting input, acquiring the first image frame sequence and the sensor data through the HAL, and acquiring a second image frame sequence of a second duration; calculating a jitter compensation parameter based on the first image frame sequence and the sensor data; and performing image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameter to generate a first live image.

[0007] Secondly, embodiments of this application provide an image processing apparatus, which includes a processing module. The processing module acquires live image frames and caches a first image frame sequence and sensor data of a first duration through an EIS circular frame buffer queue in the HAL of an electronic device. The sensor data includes motion information of the electronic device during the acquisition of the first image frame sequence. The processing module is further configured to, upon receiving a shooting input, acquire the first image frame sequence and the sensor data through the HAL, and acquire a second image frame sequence of a second duration; calculate a shake compensation parameter based on the first image frame sequence and the sensor data; and perform image processing on the first image frame sequence and the second image frame sequence based on the shake compensation parameter to generate a first live image.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the image processing method as described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the image processing method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the image processing method as described in the first aspect.

[0012] In this embodiment, live image frames are acquired, and the acquired first image frame sequence and sensor data of a first duration are cached through the EIS circular frame buffer queue in the HAL of the electronic device. The sensor data includes motion information of the electronic device during the acquisition of the first image frame sequence. After receiving the shooting input, the first image frame sequence and the sensor data are obtained through the HAL, and a second image frame sequence of a second duration is acquired. Based on the first image frame sequence and the sensor data, a shake compensation parameter is calculated. Based on the shake compensation parameter, image processing is performed on the first image frame sequence and the second image frame sequence to generate a first live image. In this scheme, the electronic device uses the EIS circular frame buffer queue in the HAL to cache the first image frame sequence and the corresponding sensor data. This allows the electronic device to acquire and analyze the sensor data cached in the EIS circular frame buffer queue, obtain the motion information of the electronic device, and calculate more accurate jitter compensation processing parameters based on the motion information of the electronic device. Furthermore, the electronic device can directly call the hardware module related to jitter compensation processing through the HAL to perform jitter compensation processing on the first and second image frame sequences according to the obtained jitter compensation processing parameters, better offsetting the image jitter. This results in a smaller image jitter amplitude in the live image generated by the electronic device, improving the quality of the live image generated by the electronic device. Attached Figure Description

[0013] Figure 1 This is one of the flowcharts of the image processing method provided in some embodiments of this application;

[0014] Figure 2 This is a second flowchart of an image processing method provided in some embodiments of this application;

[0015] Figure 3 This is the third flowchart of the image processing method provided in some embodiments of this application;

[0016] Figure 4 This is the fourth flowchart of an image processing method provided in some embodiments of this application;

[0017] Figure 5 This is the fifth flowchart of an image processing method provided in some embodiments of this application;

[0018] Figure 6 This is the sixth flowchart of the image processing method provided in some embodiments of this application;

[0019] Figure 7 This is the seventh flowchart of the image processing method provided in some embodiments of this application;

[0020] Figure 8 This is a schematic diagram of the system architecture used in some embodiments of the image processing method provided in this application;

[0021] Figure 9 These are schematic diagrams of image processing apparatuses provided in some embodiments of this application.

[0022] Figure 10 These are schematic diagrams of the electronic devices provided in some embodiments of this application;

[0023] Figure 11 These are schematic diagrams of the hardware structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] The terms "at least one," "at least one," etc., in this application refer to any one, any two, or a combination of two or more of the included objects. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more, and its meaning is similar to that of "at least one."

[0027] The following explains some concepts and terms involved in the image processing method and apparatus provided in the embodiments of this application.

[0028] Applications: Multiple applications (APPs) can be installed on an electronic device, and icons of these applications are displayed on the main screen of the electronic device. In this application embodiment, the application can be an embedded application, i.e., a system application of the electronic device; alternatively, the application in this application embodiment can also be a downloadable application.

[0029] EIS: EIS is a technology that uses software algorithms to process video signals captured by an image sensor, thereby counteracting or mitigating image instability caused by camera shake.

[0030] HAL: HAL is a standardized interface layer between the operating system and the hardware. It encapsulates the hardware differences of chip manufacturers and provides a unified calling interface to the upper layers.

[0031] Controls: Controls, also known as components, are graphical user interface elements and basic building blocks of the user interface. They are such as windows or text boxes and are displayed in the program interface of any application. Controls can be buttons, text boxes, labels, etc., and are used to control all the data processed by each application and the interactive operations related to that data.

[0032] Image Signal Processor: An Image Signal Processor (ISP) is a dedicated hardware unit that processes raw format data output from an image sensor. It performs a series of standardized processes, such as black level correction, bad pixel repair, de-mosaic interpolation, white balance adjustment, color correction and noise reduction, and edge enhancement, through a pipelined algorithm processing flow.

[0033] Geometric warp processing: Geometric warp processing is an image processing technique that combines jitter compensation data calculated by the EIS algorithm to perform geometric transformations on the acquired image frames, thereby offsetting image jitter.

[0034] The image processing method and apparatus provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0035] With the development of short video technology on the internet, live video shooting has become increasingly popular. Furthermore, as short video apps have begun supporting live shooting functionality, the popularity of this feature has continued to rise. Current live shooting solutions are based on an application-layer implementation, caching 1.5 seconds of video data at the application layer and then capturing another 1.5 seconds of video data when the user takes a photo. To ensure the stability of the video footage, electronic devices employ EIS (Electronic Image Stabilization) software algorithms to compensate for camera shake.

[0036] However, since the anti-shake software algorithm relies on image analysis to perform shake compensation processing on the image frame sequence, the processing method is relatively passive, resulting in a large amplitude of image shaking in the generated live image. Therefore, in related technologies, the live image generated by electronic devices has poor quality.

[0037] The embodiments of this application can be applied to scenarios where it is necessary to capture live images.

[0038] In one possible application scenario, the image processing method provided in this application embodiment can be applied to scenarios where it is necessary to capture live images containing the fireworks explosion process, such as scenario 1.

[0039] Scenario 1: Taking a user capturing a live image of a fireworks explosion using their mobile phone as an example. The user can trigger the phone to capture live image frames, and the phone's HAL (Hardware Algorithm) uses an Electronic Image Stabilization (EIS) circular frame buffer queue to cache a first image frame sequence and sensor data of a first duration. This sensor data includes the phone's motion information during the acquisition of the first image frame sequence. After seeing the fireworks explosion, the user can immediately input a shooting input into the phone. Upon receiving the shooting input, the phone uses the HAL to acquire the first image frame sequence and sensor data, and captures a second image frame sequence of a second duration. The phone's EIS algorithm hardware module calculates a jitter compensation parameter based on the first image frame sequence and sensor data. Based on this jitter compensation parameter, the phone performs image processing on the first and second image frame sequences to generate a first live image containing the fireworks explosion process.

[0040] In another possible application scenario, the image processing method provided in this application embodiment can be applied to a scenario where real-time images containing the passing of a car are captured, such as scenario 2.

[0041] Scenario 2: Taking a user's mobile phone capturing a live image of a car passing by as an example. The user can trigger the phone to capture live image frames, and the phone's HAL (Hardware Algorithm) uses an Electronic Image Stabilization (EIS) circular frame buffer queue to cache a first image frame sequence of a first duration and sensor data, including motion information of the phone during the acquisition of the first image frame sequence. After seeing the car pass by, the user immediately inputs a shooting input into the phone. Upon receiving the shooting input, the phone acquires the first image frame sequence and sensor data through the HAL, and captures a second image frame sequence of a second duration. The phone's EIS algorithm hardware module calculates a jitter compensation parameter based on the first image frame sequence and sensor data. Based on this jitter compensation parameter, image processing is performed on the first and second image frame sequences to generate a first live image containing the car passing by.

[0042] It should be noted that the above scenarios 1 and 2 are merely exemplary examples of some scenarios that may be applied to the embodiments of this application. In actual implementation, the embodiments of this application can also be applied to any possible scenario where there is a need to capture live images. The embodiments of this application are not limited here.

[0043] This application provides an image processing method and apparatus, which acquires live image frames and caches a first image frame sequence and sensor data of a first duration in the electronic image stabilization (EIS) circular frame buffer queue in the HAL of an electronic device. The sensor data includes motion information of the electronic device during the acquisition of the first image frame sequence. After receiving a shooting input, the method acquires the first image frame sequence and the sensor data through the HAL and acquires a second image frame sequence of a second duration. Based on the first image frame sequence and the sensor data, a jitter compensation parameter is calculated. Based on the jitter compensation parameter, the method performs image processing on the first image frame sequence and the second image frame sequence to generate a first live image. In this scheme, the electronic device caches the first image frame sequence and the corresponding sensor data through the EIS circular frame buffer queue in the HAL, and obtains the second image frame sequence through the HAL. Then, it analyzes the first image frame sequence in combination with the sensor data corresponding to the first image frame sequence, calculates the jitter compensation parameter that can accurately cancel the image jitter, and performs jitter compensation processing on the first and second image frame sequences according to the jitter compensation parameter. This results in a smaller image jitter amplitude in the live image generated by the electronic device, thus improving the effect of the live image generated by the electronic device.

[0044] The image processing method provided in this application can be executed by an image processing device, which can be an electronic device, or a functional module or functional entity within an electronic device. The following description uses an electronic device as an example to illustrate the technical solution provided in this application.

[0045] Figure 1 A flowchart of an image processing method provided in an embodiment of this application is shown, such as... Figure 1 As shown, the image processing method provided in this application embodiment may include the following steps 201 to 204.

[0046] Step 201: The electronic device acquires real-time image frames and caches the first image frame sequence and sensor data of the first duration through the EIS circular frame buffer queue in the HAL of the electronic device.

[0047] In some embodiments of this application, the aforementioned live image frame includes at least one image frame captured by a camera for display in the shooting preview area of ​​a live image or for generating a live image.

[0048] In some embodiments of this application, the user can trigger the electronic device to display a live image preview interface to trigger the electronic device to capture the aforementioned live image frames.

[0049] In some embodiments of this application, the aforementioned EIS ring frame buffer queue can be a buffer queue with an EIS trimming redundancy reserve (margin) customized for the EIS function of electronic devices in the HAL layer.

[0050] In some embodiments of this application, the aforementioned EIS ring frame buffer queue can be a dedicated buffer area for direct-access camera drivers and ISP hardware, which is requested by the electronic device through the direct memory access (DMA) mechanism.

[0051] In some embodiments of this application, the read / write principle of the above-mentioned EIS circular frame buffer queue can be First In First Out (FIFO).

[0052] In some embodiments of this application, the capacity of the EIS circular frame buffer queue can be a fixed value, such as 45 image frames. Of course, the capacity of the EIS circular frame buffer queue can also be other values ​​preset by the system. The specific value can be determined according to actual needs, and this application does not limit it.

[0053] In some embodiments of this application, the electronic device can determine the capacity of the EIS circular frame buffer queue based on the frame rate of the camera in the electronic device.

[0054] For example, taking scenario 1 as an example, assuming that the camera in the user's mobile phone captures frames at a rate of 30 frames per second and the first duration is 1.5 seconds, then the capacity of the EIS circular frame buffer queue in the mobile phone can be 30 × 1.5 = 45 frames.

[0055] In some embodiments of this application, the first duration can be a fixed value, such as 1.5s. Of course, the first duration can also be other values ​​preset by the system. The specific duration can be determined according to actual needs, and this application does not limit it.

[0056] In some embodiments of this application, the image frames in the first image frame sequence described above can be raw image frames captured by a camera. Alternatively, the image frames in the first image frame sequence described above can also be image frames obtained by performing basic ISP processing on raw image frames captured by a camera.

[0057] In some embodiments of this application, the aforementioned ISP basic processing includes at least one of the following: format encapsulation, white balance initial adjustment, black level correction, and dead pixel correction. Of course, the aforementioned first ISP basic processing may also include other ISP basic processing, which can be determined according to actual needs, and this application does not limit this.

[0058] In some embodiments of this application, the electronic device may not perform any intensive algorithms on the image frames in the first image frame sequence described above.

[0059] In some embodiments of this application, the aforementioned intensive algorithm refers to an algorithm that consumes a large amount of computational resources during execution.

[0060] In some embodiments of this application, the above-mentioned intensive algorithm includes at least one of the following: EIS, Warp processing, and Multiple Frame Noise Reduce (MFNR). Of course, the above-mentioned intensive algorithm may also include other intensive algorithms, which can be determined according to actual needs, and this application does not limit them.

[0061] In some embodiments of this application, the sensor data includes motion information of the electronic device during the acquisition of the first image frame sequence.

[0062] In some embodiments of this application, the aforementioned sensor data may include gyroscope (Gyro) data.

[0063] In some embodiments of this application, the aforementioned sensor data may also include other sensor data, such as accelerometer data, magnetic sensor data, etc., which can be determined according to actual needs, and this application does not limit this.

[0064] Step 202: After receiving the shooting input, the electronic device acquires the first image frame sequence and the cached sensor data through the HAL mentioned above, and collects the second image frame sequence of the second duration.

[0065] In some embodiments of this application, the above-mentioned shooting input is used to trigger the electronic device to capture live images.

[0066] In some embodiments of this application, the aforementioned first input may include, but is not limited to: touch input by the user through a touch device such as a finger or stylus, a voice command input by the user, a specific gesture input by the user, or other feasible inputs. The specific input can be determined according to actual usage needs, and this application does not impose any limitations on it.

[0067] For example, the above-mentioned touch input can be single-click input, double-click input, or any number of clicks, or it can be long-press input or short-press input.

[0068] For example, the above voice input can be a voice command entered by the user: "Take a live image".

[0069] For example, the specific gesture mentioned above can be any one of the following: a single click gesture, a swipe gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double press gesture, or a double tap gesture.

[0070] In some embodiments of this application, the above-mentioned shooting input can be a click input on the live image shooting control in the live image shooting preview interface displayed by the electronic device.

[0071] In some embodiments of this application, the second duration can be a fixed value, such as 1.5s. Of course, the second duration can also be other values ​​preset by the system. The specific duration can be determined according to actual needs, and this application does not limit it.

[0072] In some embodiments of this application, the second duration may be the same as the first duration. Alternatively, the second duration may be different from the first duration.

[0073] In some embodiments of this application, the image frames in the second image frame sequence described above can be original image frames captured by the camera. Alternatively, the image frames in the second image frame sequence described above can also be image frames obtained by processing original image frames captured by the camera after undergoing basic processing by the first ISP.

[0074] In some embodiments of this application, after receiving the shooting input, the electronic device can immediately lock and obtain the first frame sequence stored in the above-mentioned EIS circular frame buffer queue through HAL, and continuously collect the second image frame sequence for a second duration after receiving the shooting input.

[0075] Step 203: The electronic device calculates jitter compensation parameters based on the first image frame sequence and the cached sensor data.

[0076] In some embodiments of this application, the above-mentioned EIS algorithm hardware module may also be referred to as a hardware-accelerated EIS algorithm module.

[0077] In some embodiments of this application, the EIS algorithm hardware module described above can be a hardware processing unit in an ISP used to calculate jitter compensation parameters.

[0078] In some embodiments of this application, the jitter compensation parameters described above are used to perform warp processing on the image frame sequence to reduce the jitter amplitude of the image in the image frame sequence.

[0079] In some embodiments of this application, the electronic device can send the first image frame sequence and the cached sensor data to the EIS algorithm hardware module. The electronic device calculates the inter-frame motion estimation through the EIS algorithm hardware module, and analyzes the motion information of the electronic device based on the sensor data. By combining the inter-frame motion estimation and the motion information of the electronic device, the screen motion path of the first image frame sequence is calculated, and jitter compensation parameters are obtained based on the screen motion path of the first image frame sequence.

[0080] Step 204: The electronic device performs image processing on the first image frame sequence and the second image frame sequence based on the above jitter compensation parameters to generate the first real-time image.

[0081] In some embodiments of this application, the above image processing may be jitter compensation processing.

[0082] In some embodiments of this application, the electronic device can perform jitter compensation processing on the first image frame sequence and the second image frame sequence through the Warp processing hardware module, and encode the jitter-compensated first image frame sequence and the jitter-compensated second image frame sequence through HAL to obtain video stream data. The electronic device can send the video stream data to the application layer, and the application layer can process the video stream data to obtain the first live image.

[0083] It should be noted that the specific implementation process of the electronic device performing jitter compensation processing on the first image frame sequence and the second image frame sequence through the Warp processing hardware module, and encoding the jitter-compensated first image frame sequence and the jitter-compensated second image frame sequence through HAL to obtain video stream data, the electronic device sending the video stream data to the application layer, and the application layer processing the video stream data to obtain the first live image can be found in the relevant description in the following embodiments. To avoid repetition, this application will not repeat it here.

[0084] This application provides an image processing method in which an electronic device caches a first image frame sequence and corresponding sensor data through an EIS circular frame buffer queue in a HAL (Hydraulic Image Processing Unit). This allows the electronic device to acquire and analyze the sensor data cached in the EIS circular frame buffer queue, obtain motion information of the electronic device, and calculate more accurate jitter compensation parameters based on the motion information. Furthermore, the electronic device can directly call the hardware module related to jitter compensation processing through the HAL to perform jitter compensation processing on the first and second image frame sequences according to the obtained jitter compensation parameters, thus better offsetting the image jitter and reducing the jitter amplitude of the live image generated by the electronic device, thereby improving the effect of the live image generated by the electronic device.

[0085] In some embodiments of this application, combined with Figure 1 ,like Figure 2 As shown, before "acquiring the first image frame sequence and cached sensor data through the HAL of the electronic device" in step 202 above, the image processing method provided in this application embodiment further includes the following step 302, and the above step 202 can be "the electronic device acquires the first image frame sequence and cached sensor data through the above HAL, and collects a second image frame sequence of a second duration", and the above step 204 can be specifically implemented through the following steps 204a to 204c.

[0086] Step 302: The electronic device sends a request message to the HAL through the application layer of the electronic device.

[0087] In some embodiments of this application, the above-mentioned request information may also be referred to as live shooting request information.

[0088] In some embodiments of this application, the above request information is used to request jitter compensation processing for the image frame sequence already cached in the above-mentioned EIS circular frame buffer queue and the image frame sequence of the second duration to be acquired.

[0089] In some embodiments of this application, the above request information may include a live shooting identifier.

[0090] In some embodiments of this application, the aforementioned live shooting identifier is used to characterize the aforementioned request information as a request for shooting a live image.

[0091] For example, the above-mentioned live shooting identifier can be CAPTURE_REQUEST Tag1.

[0092] Step 204a: The electronic device performs image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters through the HAL, and encodes the processed first image frame sequence and the processed second image frame sequence to obtain video stream data.

[0093] In some embodiments of this application, the electronic device can send the above-mentioned jitter compensation parameters to the Warp processing hardware module, and perform image processing on the first image frame sequence and the second image frame sequence through the Warp processing hardware module to reduce the jitter amplitude of the images in the first image frame sequence and the second image frame sequence.

[0094] It should be noted that the specific implementation process of the electronic device sending the above jitter compensation parameters to the Warp processing hardware module and performing image processing on the first image frame sequence and the second image frame sequence by the Warp processing hardware module can be found in the relevant description in the following embodiments. To avoid repetition, this application will not elaborate further here.

[0095] In some embodiments of this application, the electronic device can send the first image frame sequence and the second image frame sequence that have completed the above image processing to the hardware encoder connected to the HAL via HAL, and perform video encoding on the first image frame sequence and the second image frame sequence that have completed the above image processing according to the preset output parameters of the hardware encoder to obtain video stream data.

[0096] In some embodiments of this application, the hardware encoder may include at least one of the following: an H.264 encoder or an H.265 encoder. Of course, the hardware encoder may also include other types of hardware encoders, which can be determined according to actual needs, and this application does not limit this.

[0097] In some embodiments of this application, the preset output parameters include at least one of the following: preset resolution parameters, preset bitrate parameters, and preset frame rate parameters. Of course, the preset output parameters may also include other preset output parameters, which can be determined according to actual needs, and this application does not limit this.

[0098] In some embodiments of this application, the format of the video stream data described above may be Moving Picture Experts Group-4 (MP4) format.

[0099] In some embodiments of this application, the format of the video stream data may also be any of the following: Audio Video Interleaved (AVI) or Flash Video (FLV).

[0100] Step 204b: The electronic device sends the video stream data to the application layer of the electronic device through the HAL mentioned above.

[0101] In some embodiments of this application, the electronic device can send the aforementioned video stream data to the application layer of the electronic device via the CameraService channel through the HAL described above.

[0102] In some embodiments of this application, the Camera Service channel is a channel connecting the HAL and the application layer.

[0103] Step 204c: The electronic device processes the video stream data through its application layer to generate the first live image.

[0104] In some embodiments of this application, the electronic device can sort each image frame in the video stream data according to the timestamp through the application layer of the electronic device and obtain the cover frame, thereby generating a first live image based on the sorted image frames and the cover frame.

[0105] It should be noted that the specific implementation process of the electronic device sorting each image frame in the above video stream data according to the timestamp through the application layer of the electronic device and obtaining the cover frame, thereby generating the first live image based on the sorted image frames and the cover frame, can be found in the relevant description in the following embodiments. To avoid repetition, this application will not elaborate further here.

[0106] In this way, when a live image needs to be captured, the electronic device can send a request message to the HAL through the application layer to notify the HAL to perform image processing on the image frame sequence cached in the EIS circular frame buffer queue and the image frame sequence to be acquired. This allows the electronic device to perform image processing on the image frame sequence cached in the EIS circular frame buffer queue only when a live image needs to be captured, and to cache only the image frame sequence of the first duration when a live image is not needed, without performing image processing on the image frame sequence of the first duration, thereby reducing the power consumption of the electronic device when using the live image function.

[0107] In some embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, step 204a can be implemented through steps 304a to 304c as described below.

[0108] Step 304a: The electronic device sends the jitter compensation parameters to the Warp processing hardware module of the electronic device through the HAL.

[0109] In some embodiments of this application, the Warp processing hardware module described above can be a hardware processing unit in an ISP used to perform jitter compensation processing on an image frame sequence according to jitter compensation parameters.

[0110] In some embodiments of this application, the electronic device can send the jitter compensation parameters to the Warp processing hardware module via the HAL and a cross-process channel through a buffer queue (BufferQueue).

[0111] Step 304b: The electronic device performs Warp processing on the first image frame sequence and the second image frame sequence based on the above jitter compensation parameters through the Warp processing hardware module.

[0112] In some embodiments of this application, the above-described warp processing may include at least one of the following: rotation, translation, scaling, and cropping. Of course, the above-described warp processing may also include other warp processing methods, which can be determined according to actual needs, and this application does not limit this.

[0113] In some embodiments of this application, the electronic device can calculate the translation compensation amount, rotation and scaling transformation matrix, and non-center cropping offset based on the above jitter compensation parameters, and then perform unified translation, rotation, scaling, and cropping processing on the first image frame sequence and the second image frame sequence according to the translation compensation amount, rotation and scaling transformation matrix, and non-center cropping offset, respectively.

[0114] Step 304c: The electronic device sends the processed first image frame sequence and the processed second image frame sequence to the HAL via the Warp processing hardware module.

[0115] In some embodiments of this application, the electronic device can send the processed first image frame sequence and second image frame sequence to the HAL via the Warp processing hardware module and the BufferQueue cross-process channel.

[0116] In this way, electronic devices can send jitter compensation parameters to the Warp processing hardware module through HAL, thereby enabling the Warp processing hardware module to quickly perform geometric transformation processing on the image frame sequence and subsequently generate live images based on the geometrically transformed image frame sequence, thus improving the speed at which electronic devices generate live images.

[0117] In some embodiments of this application, combined with Figure 3 ,like Figure 4 As shown, step 204c can be implemented through steps 404a to 404c as described below.

[0118] Step 404a: The electronic device sorts the image frames in the video stream data according to the timestamp of each image frame in the video stream data through the application layer.

[0119] In some embodiments of this application, the electronic device can obtain the metadata of each image frame in the video stream data to determine the timestamp of each image frame in the video stream data.

[0120] In some embodiments of this application, the electronic device may sort the image frames in the video stream data in chronological order from front to back. Alternatively, the electronic device may sort the image frames in the video stream data in chronological order from back to front.

[0121] Step 404b: The electronic device selects an image frame from the video stream data as the cover frame based on the timestamp, through the application layer.

[0122] In some embodiments of this application, the electronic device can record the moment when the user triggers the shutter, and select the image frame corresponding to the moment when the user triggers the shutter from the video stream data as the cover frame based on the timestamp at the application layer.

[0123] In some embodiments of this application, the image frame in the video stream data corresponding to the moment when the user triggers the shutter can be the first image frame in the second image frame sequence.

[0124] Step 404c: The electronic device generates the first live image based on the cover frame and the sorted image frames in the video stream data through the application layer.

[0125] In some embodiments of this application, the electronic device can encapsulate the cover frame into a cover photo, package the cover photo and the video stream data into a compressed file, and mark the compressed file as a live image file to generate the first live image.

[0126] In some embodiments of this application, the format of the cover photograph may include at least one of the following: High Efficiency Image Container (HEIC) format, Joint Photographic Experts Group (JPEG) format, and Portable Network Graphics (PNG) format.

[0127] In this way, electronic devices can sort each image frame in the video stream data according to the timestamp at the application layer, and select an image frame as the cover frame according to the timestamp, thereby generating a live image with a continuous picture during playback based on the sorted image frames and the cover frame.

[0128] In some embodiments of this application, combined with Figure 1 ,like Figure 5 As shown, step 204 can be implemented in detail through steps 504a and 504b below.

[0129] Step 504a: The electronic device performs jitter compensation processing on the first image frame sequence and the second image frame sequence based on jitter compensation parameters.

[0130] In some embodiments of this application, the electronic device can perform jitter compensation processing on the first image frame sequence and the second image frame sequence based on jitter compensation parameters through the Warp processing hardware module.

[0131] It should be noted that the specific implementation process of the electronic device performing jitter compensation processing on the first image frame sequence and the second image frame sequence based on jitter compensation parameters through the Warp processing hardware module can be found in the relevant description in the above embodiments. To avoid repetition, this application will not repeat it here.

[0132] Step 504b: The electronic device performs image enhancement processing on the first image frame sequence and the second image frame sequence after jitter compensation processing through the image enhancement hardware module of the electronic device to generate the aforementioned first real-time image.

[0133] In some embodiments of this application, the image enhancement hardware module includes at least one of the following: a digital signal processor (DSP), a neural network processing unit (NPU), and a graphics processing unit (GPU). Of course, the image enhancement hardware module may also include other image enhancement hardware modules, which can be determined according to actual needs, and this application does not limit this.

[0134] In some embodiments of this application, the above-mentioned image enhancement processing refers to image processing that can enhance the display quality of each image frame in the first image frame sequence after jitter compensation processing and the second image frame sequence after jitter compensation processing.

[0135] In some embodiments of this application, the image enhancement processing methods described above include at least one of the following: MFNR, Temporal Noise Reduction (Temporal NR), High Dynamic Range (HDR) fusion, Image Quality Calibration (IQC), Zoom Adaptation (ZA), and Dynamic Resource Scheduling (DRS). Of course, the image enhancement processing described above may also include other image enhancement methods, which can be determined according to actual needs, and this application does not limit them.

[0136] In some embodiments of this application, the electronic device can use the above-described image enhancement hardware module to perform corresponding image enhancement processing on the first image frame sequence after jitter compensation processing and the second image frame sequence after jitter compensation processing, so as to improve the display quality of the image frames.

[0137] For example, an electronic device can use a DSP or NPU to perform pixel-level fusion processing on the jitter-compensated first image frame sequence and the jitter-compensated second image frame sequence to achieve MFNR of the jitter-compensated first image frame sequence and the jitter-compensated second image frame sequence, thereby improving the purity of the displayed image frames in low-light scenes. Alternatively, the electronic device can use the HDR hardware processing unit of the ISP to perform multi-frame exposure fusion to achieve HDR fusion of the jitter-compensated first image frame sequence and the jitter-compensated second image frame sequence, optimizing the dynamic range of the image frames in high-contrast scenes.

[0138] For example, the electronic device can perform hardware-level sharpening, color correction, and gamma adjustment on the first and second image frame sequences after jitter compensation using the ISP and GPU to achieve IQC (Initial Quality Control) for the jitter-compensated first and second image frame sequences, thereby correcting the image quality and color parameters of the image frames. Alternatively, if the electronic device detects that zooming has occurred when acquiring a certain image frame from the jitter-compensated first and second image frame sequences, it can use the image cropping and resolution scaling modules in the ISP to perform zoom cropping and resolution scaling of the image frame, avoiding the image quality loss caused by software scaling.

[0139] In some embodiments of this application, after the electronic device performs image enhancement processing on the first image frame sequence and the second image frame sequence after jitter compensation processing, it can obtain a third image frame sequence and a fourth image frame sequence. The electronic device can encode the third image frame sequence and the fourth image frame sequence to obtain video stream data, and generate a live image based on the video stream data.

[0140] It should be noted that the specific implementation process of the electronic device generating live images based on video stream data can be found in the relevant descriptions in the above embodiments. To avoid repetition, this application will not repeat the details here.

[0141] In this way, electronic devices can perform jitter compensation processing on the image frame sequence, and then perform image enhancement processing on the obtained image frame sequence to improve the display quality of each image frame in the image frame sequence, thereby improving the display quality of the live image generated subsequently based on the image frame sequence.

[0142] In some embodiments of this application, combined with Figure 1 ,like Figure 6 As shown, prior to step 204 above, the image processing method provided in this application embodiment further includes step 205 as described below.

[0143] Step 205: The electronic device stores the first live image to the local storage directory of the electronic device through the application layer of the electronic device, and generates an image query index for the first live image through the HAL mentioned above.

[0144] In some embodiments of this application, the aforementioned local storage directory may be a system-preset directory, or it may be a directory set by the user. The specific location can be determined according to actual needs, and this application does not limit it.

[0145] In some embodiments of this application, the image query index described above is used to retrieve the first live image described above.

[0146] In some embodiments of this application, the image query index may include a file path and an identifier of a first real-world image.

[0147] In some embodiments of this application, the image query index may also include information such as the date the first live image was captured and the storage space occupied by the first live image. The specific details can be determined according to actual needs, and this application does not impose any limitations on this.

[0148] In some embodiments of this application, the electronic device can store the image query index in the database of the electronic device, and the electronic device can read the image query index stored in the database through the first application to quickly obtain the first live image.

[0149] In some embodiments of this application, the type of the first application mentioned above includes at least one of the following: photo album application, video application, and social application. Of course, the first application mentioned above may also include other types of applications, which can be determined according to actual needs, and this application does not limit this.

[0150] In some embodiments of this application, after the electronic device generates an image query index for the first live image through the HAL, it can clear the first image frame sequence cached in the EIS circular frame buffer queue and restore the circular caching logic of the queue to cache an image frame sequence of the first duration for the next live image capture.

[0151] In some embodiments of this application, after the electronic device generates an image query index for the first real-time image through the HAL described above, it can release the occupied hardware resources such as the EIS algorithm hardware module, the Warp processing hardware module, and the encoder, so as to release the computing power resources of the electronic device.

[0152] In some embodiments of this application, after the electronic device generates an image query index for the first live image through the HAL, it can release the memory (buffer) temporarily occupied by the electronic device during the generation of the first live image, and only retain the memory occupied by the EIS circular frame buffer queue for caching the image frame sequence of the first duration, thereby avoiding the problem of memory leakage in the electronic device.

[0153] In this way, electronic devices can store the generated live video in their local storage directory and generate a query index for retrieving the live video. This allows subsequent electronic devices to access the query index through applications and quickly retrieve the live video based on the query index, thus improving the retrieval efficiency of live videos.

[0154] This application proposes an image processing method that decentralizes the capture, caching, and management of live frame data to the HAL (Host Algorithm), abandoning the traditional application-layer self-caching approach. This fully leverages the high-efficiency transmission and processing capabilities of the underlying hardware to improve the quality of live images generated by electronic devices and reduce the energy consumption during live image generation. For example, as... Figure 7 As shown, the image processing method proposed in this application may include the following steps 40 to 52:

[0155] Step 40: The electronic device receives the user's input by clicking the live shooting button and displays the live image shooting preview interface.

[0156] Step 41: The electronic device caches the image frames acquired by the camera within the first time period and the gyroscope data within the first time period in the HAL via the EIS circular frame buffer queue.

[0157] In some embodiments of this application, the electronic device can adapt the HAL layer during the process to accommodate hardware differences between different chip platforms.

[0158] In some embodiments of this application, the electronic device can dynamically adjust the capacity of the EIS circular frame buffer queue according to the frame processing speed of different platform ISPs and the trimming margin requirements of the EIS algorithm hardware module.

[0159] For example, when the platform ISP's frame processing speed is fast, the electronic device can appropriately increase the capacity of the EIS circular frame buffer queue. Alternatively, when the platform ISP's frame processing speed is slow, the electronic device can appropriately decrease the capacity of the EIS circular frame buffer queue.

[0160] For example, when the EIS algorithm hardware module of the platform has a high trimming margin requirement, the electronic device can appropriately reduce the capacity of the EIS circular frame buffer queue. Alternatively, when the EIS algorithm hardware module of the platform has a low trimming margin requirement, the electronic device can appropriately increase the capacity of the EIS circular frame buffer queue.

[0161] In some embodiments of this application, the electronic device can adjust the buffering frequency of the EIS circular frame buffer queue according to the hardware energy efficiency ratio of different platforms.

[0162] For example, when the platform's hardware energy efficiency is relatively high, the electronic device can appropriately increase the buffering frequency of the EIS circular frame buffer queue. Alternatively, when the platform's hardware energy efficiency is relatively low, the electronic device can appropriately decrease the buffering frequency of the EIS circular frame buffer queue.

[0163] In some embodiments of this application, the electronic device reuses the native frame buffer queue of EIS jitter compensation processing, namely the EIS circular frame buffer queue. This queue is originally used for multi-frame synthesis stabilization and is naturally adapted to the low power consumption and high compatibility requirements of buffering image frames when shooting live images, without the need to build an additional independent buffer link.

[0164] Step 42: The electronic device receives the user's shooting input, immediately locks the image frames cached in the EIS circular frame buffer queue, and then captures image frames within the second time period through the camera.

[0165] Step 43: The electronic device sends the image frames cached in the EIS circular frame buffer queue, i.e. the image frames acquired within the first time period, into the EIS algorithm module, and calculates the anti-shake compensation parameters by combining the gyroscope data and inter-frame motion estimation.

[0166] Step 44: The electronic device performs jitter compensation processing on the first and second half of the frame sequence through the platform hardware Warp module according to the image stabilization compensation parameters.

[0167] Step 45: The electronic device performs image enhancement operations on the processed image frame sequence as needed, resulting in two image frame sequences after image enhancement processing.

[0168] Step 46: The electronic device sends the two image frame sequences after image enhancement processing to the hardware encoder connected to HAL, and performs video encoding on the first frame sequence according to preset parameters to obtain basic video stream data.

[0169] Step 47: The electronic device transmits the basic video stream data to the application layer.

[0170] Step 48: At the application layer, the electronic device sorts each image frame in the video stream data by timestamp.

[0171] Step 49: At the application layer, the electronic device selects an image frame from the sorted video stream data using timestamps as the cover frame.

[0172] Step 50: The electronic device generates a live image based on the video stream data and the cover frame.

[0173] Step 51: The electronic device stores the generated live image in the electronic device and generates file index information.

[0174] Step 52: The electronic device releases the hardware resources it occupies, clears the EIS circular frame buffer queue, and caches the first duration of image frames for the next capture.

[0175] It should be noted that the specific implementation process of each step in steps 40 to 52 above can be found in the relevant descriptions in the above embodiments. To avoid repetition, this application will not repeat them here.

[0176] This application cleverly reuses the caching capability of the EIS hardware mechanism by deeply integrating live streaming functionality into HAL. At the same time, in order to achieve performance and power consumption, it enables the function of triggering EIS on demand. When no photo is being taken, the image frames acquired within a certain period of time are cached without any algorithms. When a photo input is received, the cached image frames are sent to EIS for processing, and the platform's WARP and noise reduction algorithms are executed to improve the quality of the live streaming image. This fundamentally solves the problems of high power consumption, limited image quality, and poor stability of traditional application-layer live streaming solutions.

[0177] Furthermore, this application fully utilizes powerful heterogeneous computing hardware in the terminal, such as ISP, DSP, NPU, and GPU, and leverages the efficiency advantages of the underlying system. While ensuring a smooth user experience with low preview latency, it significantly optimizes the energy efficiency of live image generation and substantially improves the image quality of the final generated live images, such as stability, clarity, and purity, providing users with a higher-quality live image shooting experience. This application also enhances the quality of upper-layer application functions through collaborative innovation of terminal-side hardware capabilities and software architecture. Moreover, by directly calling hardware modules to execute algorithm processing via HAL, compared to pure software implementation at the application layer, the processing speed for the same algorithm task is increased by 2-3 times, and power consumption is reduced by more than 50%, while avoiding overheating of the device caused by the central processing unit (CPU) running at full load.

[0178] For example, such as Figure 8As shown, the system architecture used in the image processing method provided in some embodiments of this application includes an application layer and a HAL. Assuming that the sampling rate of the camera in the electronic device is 30 frames per second and the capacity of the EIS circular buffer queue is 45 frames, when the user triggers the electronic device to capture live image frames, the EIS circular buffer queue in the HAL can cache the image frame sequence from frame 1 to frame 45, a total of 1.5 seconds. In the absence of receiving notification information for taking a picture from the application layer, a first-in-first-out update strategy is adopted, starting from the image frame with the earlier cache time, continuously replacing the image frames cached in the EIS circular buffer queue, and discarding the replaced image frames with the earlier cache time. Upon receiving a notification from the application layer to take a picture, the electronic device calculates shake compensation parameters using the EIS algorithm. Based on these parameters, the device performs shake compensation processing on the 1.5s image frame queue buffered in the EIS circular buffer queue and the 1.5s image frame queue captured after the user triggers the electronic device to take a live picture. Then, the hardware noise reduction module performs Crop and Upscale processing on the shake-compensated image frame queue. The noise-reduced image frame queue is then encoded to obtain MP4 format video stream data. This MP4 format video stream data is sent to the application layer, where a cover photo is obtained based on the video stream data. Finally, the application layer packages the video stream data and the cover photo into live data to generate a live file.

[0179] Each of the above-described method embodiments, or various possible implementations of each method embodiment, can be executed individually or in combination of any two or more. The specific implementation can be determined according to actual usage requirements, and this application does not impose any restrictions on this.

[0180] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0181] Figure 9 A schematic diagram of a possible structure of the image processing apparatus involved in some embodiments of this application is shown. For example... Figure 9 As shown, the image processing device 70 may include a processing module 71 and a storage module 72.

[0182] The processing module 71 is used to acquire live image frames; the storage module 72 is used to cache the acquired first image frame sequence and sensor data of a first duration through the EIS circular frame buffer queue in the HAL of the electronic device, the sensor data including the motion information of the electronic device during the acquisition of the first image frame sequence; the processing module 71 is also used to acquire the first image frame sequence and the sensor data through the HAL after receiving the shooting input, and acquire a second image frame sequence of a second duration; and to calculate the shake compensation parameter based on the first image frame sequence and the sensor data; and to perform image processing on the first image frame sequence and the second image frame sequence based on the shake compensation parameter to generate a first live image.

[0183] In one possible implementation, the processing module 71 is configured to send a request message to the HAL through the application layer of the electronic device before acquiring the first image frame sequence and the sensor data through the HAL. The request message requests jitter compensation processing for the image frame sequence already cached in the EIS circular frame buffer queue and the image frame sequence of the second duration to be acquired. The processing module 71 is further configured to perform image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters through the HAL, and encode the processed first image frame sequence and the processed second image frame sequence to obtain video stream data. The processing module 71 is further configured to send the video stream data to the application layer through the HAL, and perform data processing on the video stream data through the application layer to generate the first live image.

[0184] In one possible implementation, the processing module 71 is specifically configured to send the jitter compensation parameters to the Warp processing hardware module of the electronic device via the HAL, and to perform geometric transformation processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters via the Warp processing hardware module; and to send the processed first image frame sequence and the processed second image frame sequence to the HAL via the Warp processing hardware module.

[0185] In one possible implementation, the processing module 71 is specifically configured to sort the image frames in the video stream data according to the timestamp of each image frame in the video stream data through the application layer, and select an image frame from the video stream data as a cover frame based on the timestamp through the application layer; and generate the first live image based on the cover frame and the sorted image frames in the video stream data through the application layer.

[0186] In one possible implementation, the processing module 71 is specifically used to perform jitter compensation processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters, and to perform image enhancement processing on the jitter-compensated first image frame sequence and the jitter-compensated second image frame sequence through the image enhancement hardware module of the electronic device to generate the first live image. The image enhancement hardware module includes at least one of the following: DSP, NPU, GPU, ISP.

[0187] In one possible implementation, the storage module 72 is further configured to, after the processing module 71 performs image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters to generate a first live image, store the first live image in the local storage directory of the electronic device through the application layer of the electronic device, and generate an image query index for the first live image through the HAL.

[0188] This application provides an image processing device that uses an EIS circular frame buffer queue in the HAL to cache a first image frame sequence and corresponding sensor data. This allows the image processing device to acquire and analyze the cached sensor data in the EIS circular frame buffer queue, obtain motion information of the image processing device, and calculate more accurate jitter compensation parameters based on the motion information. Furthermore, the image processing device can directly call the hardware module related to jitter compensation processing through the HAL to perform jitter compensation processing on the first and second image frame sequences based on the obtained jitter compensation parameters, thus better compensating for image jitter. This results in a smaller jitter amplitude in the live image generated by the image processing device, improving the quality of the live image generated by the image processing device.

[0189] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0190] The image processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0191] The image processing apparatus provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0192] Optionally, such as Figure 10 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0193] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0194] Figure 11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0195] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0196] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0197] The processor 110 is configured to acquire live image frames and cache the acquired first image frame sequence and sensor data for a first duration through the EIS circular frame buffer queue in the HAL of the electronic device. The sensor data includes motion information of the electronic device during the acquisition of the first image frame sequence. The processor 110 is also configured to acquire the first image frame sequence and the sensor data through the HAL after receiving the shooting input, and acquire a second image frame sequence for a second duration; calculate a shake compensation parameter based on the first image frame sequence and the sensor data; and perform image processing on the first image frame sequence and the second image frame sequence based on the shake compensation parameter to generate a first live image.

[0198] Optionally, the processor 110 is further configured to send a request message to the HAL through the application layer of the electronic device before acquiring the first image frame sequence and the sensor data through the HAL. The request message is used to request jitter compensation processing for the image frame sequence already cached in the EIS circular frame buffer queue and the image frame sequence of the second duration to be acquired. The processor 110 is further configured to perform image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters through the HAL, and encode the processed first image frame sequence and the processed second image frame sequence to obtain video stream data. The processor 110 is further configured to send the video stream data to the application layer through the HAL, and perform data processing on the video stream data through the application layer to generate the first live image.

[0199] Optionally, the processor 110 is specifically configured to send the jitter compensation parameters to the Warp processing hardware module of the electronic device via the HAL, and to perform geometric transformation processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters via the Warp processing hardware module; and to send the processed first image frame sequence and the processed second image frame sequence to the HAL via the Warp processing hardware module.

[0200] Optionally, the processor 110 is specifically configured to sort the image frames in the video stream data according to the timestamp of each image frame in the video stream data through the application layer, and select an image frame from the video stream data as a cover frame based on the timestamp through the application layer; and generate the first live image based on the cover frame and the sorted image frames in the video stream data through the application layer.

[0201] Optionally, the processor 110 is specifically used to perform jitter compensation processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters, and to perform image enhancement processing on the jitter-compensated first image frame sequence and the jitter-compensated second image frame sequence through the image enhancement hardware module of the electronic device to generate the first live image. The image enhancement hardware module includes at least one of the following: DSP, NPU, GPU, ISP.

[0202] Optionally, the processor 110 is further configured to, after performing image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters to generate a first live image, store the first live image in the local storage directory of the electronic device through the application layer of the electronic device, and generate an image query index for the first live image through the HAL.

[0203] This application provides an electronic device that uses an EIS circular frame buffer queue in a HAL (Hybrid Image Array) to cache a first image frame sequence and corresponding sensor data. This allows the electronic device to acquire and analyze the sensor data cached in the EIS circular frame buffer queue, obtain motion information of the electronic device, and calculate more accurate jitter compensation parameters based on the motion information. Furthermore, the electronic device can directly call the hardware module related to jitter compensation processing through the HAL to perform jitter compensation processing on the first and second image frame sequences according to the obtained jitter compensation parameters, thus better compensating for image jitter and reducing the jitter amplitude of the live image generated by the electronic device, thereby improving the quality of the live image generated by the electronic device.

[0204] The electronic device provided in this application embodiment can implement all the processes implemented in the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here. The beneficial effects of the various implementation methods in this embodiment can be found in the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, it will not be described again here.

[0205] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0206] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0207] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0208] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0209] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0210] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0211] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0212] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the image processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0213] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0214] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0215] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Real-time image frames are acquired and cached in the Electronic Image Stabilization System (EIS) circular frame buffer queue in the Hardware Abstraction Layer (HAL) of the electronic device for a first duration, along with sensor data. The sensor data includes motion information of the electronic device during the acquisition of the first image frame sequence. After receiving the shooting input, the first image frame sequence and the sensor data are obtained through the HAL, and a second image frame sequence of a second duration is acquired. Calculate jitter compensation parameters based on the first image frame sequence and the sensor data; Based on the jitter compensation parameters, image processing is performed on the first image frame sequence and the second image frame sequence to generate a first live image.

2. The method according to claim 1, characterized in that, Before acquiring the first image frame sequence and the sensor data through the HAL, the method further includes: The electronic device sends a request message to the HAL through the application layer of the electronic device. The request message is used to request jitter compensation processing for the image frame sequence already cached in the EIS circular frame buffer queue and the image frame sequence of the second duration to be acquired. The step of performing image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters to generate a first live image includes: The HAL is used to perform image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters, and the processed first image frame sequence and the processed second image frame sequence are encoded to obtain video stream data. The video stream data is sent to the application layer via the HAL; The application layer processes the video stream data to generate the first live image.

3. The method according to claim 2, characterized in that, The step of performing image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters using the HAL includes: The jitter compensation parameters are sent to the geometric deformation warp processing hardware module of the electronic device via the HAL. The Warp processing hardware module performs geometric transformation processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters. The Warp processing hardware module sends the processed first image frame sequence and the processed second image frame sequence to the HAL.

4. The method according to claim 2 or 3, characterized in that, The step of processing the video stream data through the application layer to generate the first live image includes: The application layer sorts the image frames in the video stream data according to the timestamp of each image frame in the video stream data. Through the application layer, an image frame is selected from the video stream data based on the timestamp as the cover frame; The first live image is generated through the application layer based on the cover frame and the sorted image frames in the video stream data.

5. The method according to claim 1, characterized in that, The step of performing image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters to generate a first live image includes: The first image frame sequence and the second image frame sequence are subjected to jitter compensation processing based on the jitter compensation parameters; The image enhancement hardware module of the electronic device performs image enhancement processing on the first image frame sequence after jitter compensation processing and the second image frame sequence after jitter compensation processing to generate the first live image. The image enhancement hardware module includes at least one of the following: digital signal processor (DSP), neural network processor (NPU), image processor (GPU), and image signal processor (ISP).

6. An image processing apparatus, characterized in that, The device includes: a processing module and a storage module; The processing module is used to acquire real-time image frames; The storage module is used to cache the first image frame sequence and sensor data acquired for a first duration through the electronic image stabilization (EIS) circular frame buffer queue in the hardware abstraction layer (HAL) of the electronic device. The sensor data includes motion information of the electronic device during the acquisition of the first image frame sequence. The processing module is further configured to, upon receiving the shooting input, acquire the first image frame sequence and the sensor data stored in the storage module through the HAL, and acquire a second image frame sequence of a second duration; and calculate shake compensation parameters based on the first image frame sequence and the sensor data; and perform image processing on the first image frame sequence and the second image frame sequence based on the shake compensation parameters to generate a first live image.

7. The apparatus according to claim 6, characterized in that, The processing module is further configured to send a request message to the HAL through the application layer of the electronic device before acquiring the first image frame sequence and the sensor data through the HAL. The request message is used to request jitter compensation processing for the image frame sequence already cached in the EIS circular frame buffer queue and the image frame sequence of the second duration to be acquired. The processing module is further configured to perform image processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters through the HAL, and to encode the processed first image frame sequence and the processed second image frame sequence to obtain video stream data; The processing module is further configured to send the video stream data to the application layer through the HAL, and to process the video stream data through the application layer to generate the first live image.

8. The apparatus according to claim 7, characterized in that, The processing module is specifically used to send the jitter compensation parameters to the geometric deformation Warp processing hardware module of the electronic device through the HAL, and to perform geometric transformation processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters through the Warp processing hardware module. Furthermore, the processed first image frame sequence and the processed second image frame sequence are sent to the HAL via the Warp processing hardware module.

9. The apparatus according to claim 7 or 8, characterized in that, The processing module is specifically used to sort the image frames in the video stream data according to the timestamp of each image frame in the video stream data through the application layer, and select an image frame from the video stream data as the cover frame based on the timestamp through the application layer. Furthermore, the first live image is generated through the application layer based on the cover frame and the sorted image frames in the video stream data.

10. The apparatus according to claim 6, characterized in that, The processing module is specifically used to perform jitter compensation processing on the first image frame sequence and the second image frame sequence based on the jitter compensation parameters, and to perform image enhancement processing on the jitter-compensated first image frame sequence and the jitter-compensated second image frame sequence through the image enhancement hardware module of the electronic device to generate the first live image. The image enhancement hardware module includes at least one of the following: digital signal processor (DSP), neural network processor (NPU), image processor (GPU), and image signal processor (ISP).

Images