Method and device for stabilizing image frame

Abstract

The embodiment of the invention provides a method and device for stabilizing an image frame. The method comprises the steps of receiving a first operation of a user; in response to the first operation, opening a camera application, and sequentially drawing and rendering to obtain a first image frame, a second image frame and a third image frame; based on the first frame rate, synthesizing the first image frame to obtain a first synthesized image frame, synthesizing the second image frame to obtain a second synthesized image frame, and synthesizing the third image frame to obtain a third synthesized image frame; the terminal device displays the first composite image frame in the first period, displays the second composite image frame in the second period, and displays the third composite image frame in the third period; the time interval between the first period and the second period is the same as the time interval between the second period and the third period. In this way, the situation of picture jamming caused by non-uniform image frames before picture layer synthesis can be reduced in the picture layer synthesis stage, so that the displayed picture can be in a uniform state in a period.

Description

Method and device for stabilizing image frames

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on December 2, 2022, with application number 202211541688.5 and application name “Method and Device for Stabilizing Image Frames”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a method and device for stabilizing image frames. Background Art

[0003] With the popularization and development of the Internet, more terminal devices can support photo and video recording functions, and support the display of preview images during the photo and video recording process, so that users can determine whether to take a photo or record a video based on the image displayed in the preview interface.

[0004] Typically, when a user opens a camera preview screen, the terminal device captures an image sequence using the camera and processes and displays each frame in the sequence to create a preview screen. However, due to the differences between the image frames, the timing of each processed frame is uneven, causing the camera preview screen to experience lag.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method and apparatus for stabilizing image frames, so that a terminal device can ensure the smoothness of the preview screen when performing layer synthesis on image frames.

[0007] In a first aspect, an embodiment of the present application provides a method for stabilizing image frames, which is applied to a terminal device and includes: receiving a first user operation; receiving the first user operation; in response to the first operation, launching a camera application and sequentially rendering a first image frame, a second image frame, and a third image frame, wherein the time interval between the second image frame and the first image frame is a first time interval, the time interval between the third image frame and the second image frame is a second time interval, and the first time interval and the second time interval are different in size; synthesizing the first image frame based on a first frame rate to obtain a first synthesized image frame, synthesizing the second image frame to obtain a second synthesized image frame, and synthesizing the third image frame to obtain a third synthesized image frame; displaying the first synthesized image frame within a first period, displaying the second synthesized image frame within a second period, and displaying the third synthesized image frame within a third period, wherein the time interval between the first period and the second period is the third time interval, and the time interval between the second period and the third period is a fourth time interval, and the third time interval and the fourth time interval are the same in size. In this way, image freezes caused by uneven image frames before layer synthesis can be reduced in the layer synthesis stage, so that the displayed image can be uniform within the period.

[0008] In one possible implementation, synthesizing the first image frame based on the first frame rate to obtain a first synthesized image frame, synthesizing the second image frame based on the second image frame to obtain a second synthesized image frame, and synthesizing the third image frame based on the third image frame to obtain a third synthesized image frame includes: when it is determined within a fourth period that the first moment is greater than or equal to a target time interval determined based on the first frame rate, synthesizing the first image frame within the fourth period to obtain the first synthesized image frame; when it is determined within a fifth period that the fifth time interval between the second moment and the time of synthesizing the first image frame is greater than or equal to the target time interval, synthesizing the second image frame within the fifth period to obtain the second synthesized image frame; the fifth period is after the fourth period; when it is determined within a sixth period that the sixth time interval between the third moment and the time of synthesizing the second image frame is greater than or equal to the target time interval, synthesizing the third image frame within the sixth period to obtain the third synthesized image frame; the sixth period is after the fifth period. In this way, the terminal device can ensure uniform display of adjacent image frames during the image synthesis processing stage based on the first frame rate and the time interval when adjacent frames are synthesized.

[0009] In one possible implementation, the method further includes: when it is determined within the fifth cycle that the fifth time interval is less than the target time interval, not synthesizing the second image frame within the fifth cycle; when it is determined within the fifth cycle that the fifth time interval between the second moment and the time of synthesizing the first image frame is greater than or equal to the target time interval, synthesizing the second image frame within the fifth cycle to obtain a second synthesized image frame, including: when it is determined within the seventh cycle that the seventh time interval between the fourth moment and the time of synthesizing the first image frame is greater than or equal to the target time interval, synthesizing the second image frame within the seventh cycle to obtain a second synthesized image frame; the seventh cycle is after the fifth cycle and before the sixth cycle. In this way, when the terminal device determines that performing layer synthesis on image frames within a certain cycle may result in uneven image display, it does not synthesize the image frames within that cycle to ensure the stability of subsequent image display.

[0010] In a possible implementation, the first image frame, the second image frame, and the third image frame are all image frames of the preview layer, so that image frames of the same layer can be processed at the same frame rate.

[0011] In one possible implementation, a terminal device includes: a camera module and a camera hardware abstraction layer service. Before sequentially rendering to obtain a first image frame, a second image frame, and a third image frame, the method further includes: the camera module generating a first image sequence based on a first frame rate and sending the first image sequence to the camera hardware abstraction layer service; wherein the first image sequence includes: a first original image frame, a second original image frame, and a third original image frame; the time interval between the first original image frame and the second original image frame is an eighth time interval, the time interval between the second original image frame and the third original image frame is a ninth time interval, and the eighth time interval and the ninth time interval are equal; and the camera hardware abstraction layer service performing image processing on the first original image frame to obtain a first process image frame, performing image processing on the second original image frame to obtain a second process image frame, and performing image processing on the third original image frame to obtain a third process image frame. In this way, even if the camera provider's frame output time is uneven, the surface flinger can ensure sufficient layer synthesis time when performing layer synthesis on the buffer and perform layer synthesis at a stable consumption rate to reduce screen freezes.

[0012] In one possible implementation, a terminal device includes: a camera service, and a layer view, and the method further includes: when the camera service receives a first process image frame, a second process image frame, and a third process image frame sent by a camera hardware abstraction layer service, the camera service sets a first identifier for the first process image frame, the second process image frame, and the third process image frame, respectively; wherein the first identifier indicates a first frame rate; the camera service sends the first process image frame with the first identifier, the second process image frame with the first identifier, and the third process image frame with the first identifier to the layer view; and sequentially draws and renders to obtain the first image frame, the second image frame, and the third image frame, including: the layer view sequentially draws and renders the first process image frame to obtain the first image frame, draws and renders the second process image frame to obtain the second image frame, and draws and renders the third process image frame to obtain the third image frame.

[0013] In one possible implementation, a terminal device includes: a display composition system that, based on a first frame rate, respectively composites a first image frame to obtain a first composite image frame, composites a second image frame to obtain a second composite image frame, and composites a third image frame to obtain a third composite image frame. The system includes: upon receiving a first identifier sent by a layer view, the display composition system composites the first image frame to obtain the first composite image frame, composites the second image frame to obtain the second composite image frame, and composites the third image frame to obtain the third composite image frame based on the first frame rate indicated in the first identifier. In this way, the display composition system can determine the composite frame rate based on the identifier to achieve uniform layer composition.

[0014] In a possible implementation, the first identifier is obtained based on the SetConsumeRate interface, and the SetConsumeRate interface is used to instruct the layer view to pass the first identifier to the display composition system.

[0015] In one possible implementation, the first frame rate is less than 60 Hz, so that the terminal device can maintain a relatively stable picture at a frame rate below 60 Hz.

[0016] In a second aspect, an embodiment of the present application provides a method for stabilizing image frames, which is applied to a terminal device, and the terminal device includes: a layer view, and a display synthesis system. The method includes: receiving a first operation of a user; in response to the first operation, starting a camera application, and the layer view sequentially drawing and rendering to obtain a first image frame, a second image frame, and a third image frame; wherein the time interval between the second image frame and the first image frame is a first time interval, and the time interval between the third image frame and the second image frame is a second time interval, and the first time interval is different from the second time interval; the layer view sends the first image frame to the display synthesis system within an eighth period based on the first frame rate, sends the second image frame to the display synthesis system within a ninth period based on the first frame rate, and sends the third image frame to the display synthesis system within a tenth period based on the first frame rate; the time interval between the eighth period and the ninth period is a tenth time interval, and the time interval between the ninth period and the tenth period is an eleventh time interval, and the tenth time interval is the same as the eleventh time interval; the display synthesis system synthesizes the first image frame to obtain a first synthesized image frame, synthesizes the second image frame to obtain a second synthesized image frame, and synthesizes the third image frame to obtain a third synthesized image frame. In this way, the layer view can stably send image frames to the display synthesis system based on the first frame rate, thereby ensuring the stability of the synthesis of the display synthesis system and the subsequent screen display of the picture.

[0017] In one possible implementation, based on the first frame rate, the first image frame is sent to the display composition system within an eighth cycle, the second image frame is sent to the display composition system within a ninth cycle, and the third image frame is sent to the display composition system within a tenth cycle, including: when it is determined within the eighth cycle that the fifth moment is greater than or equal to a target time interval determined based on the first frame rate, the first image frame is sent to the display composition system within the eighth cycle; when it is determined within the ninth cycle that the twelfth time interval between the sixth moment and the time when the first image frame is sent to the display composition system is greater than or equal to the target time interval, the second image frame is sent to the display composition system within the ninth cycle; when it is determined within the tenth cycle that the thirteenth time interval between the seventh moment and the time when the second image frame is sent to the display composition system is greater than or equal to the target time interval, the third image frame is sent to the display composition system within the tenth cycle. In this way, the layer view can evenly send buffers to the surface flinger based on the first frame rate and the time intervals between two adjacent buffers sent to the BBQ, thereby alleviating screen freezes when the screen displays consecutive frames.

[0018] In one possible implementation, the method further includes: when the twelfth time interval in the ninth cycle is less than the target time interval, not sending the second image frame to the display composition system in the ninth cycle; when the twelfth time interval between the sixth moment and the time when the first image frame is sent to the display composition system in the ninth cycle is determined to be greater than or equal to the target time interval, sending the second image frame to the display composition system in the ninth cycle, including: when the fourteenth time interval between the eighth moment and the time when the first image frame is sent to the display composition system in the eleventh cycle is determined to be greater than or equal to the target time interval, sending the second image frame to the display composition system in the eleventh cycle; wherein the eleventh cycle is after the ninth cycle and before the tenth cycle. In this way, when the layer view determines that sending image frames in a certain cycle may cause uneven image display, it does not perform composition processing on the image frames in that cycle to ensure the stability of subsequent image display.

[0019] In a possible implementation, the first image frame, the second image frame, and the third image frame are all image frames of the preview layer, so that image frames of the same layer can be processed at the same frame rate.

[0020] In one possible implementation, the first frame rate is less than 60 Hz, so that the terminal device can maintain a relatively stable picture at a frame rate below 60 Hz.

[0021] In a third aspect, an embodiment of the present application provides an apparatus for stabilizing image frames, and the method is applied to a terminal device, wherein a processing unit is used to receive a first operation of a user; the processing unit is also used to start a camera application in response to the first operation, and to draw and render in sequence to obtain a first image frame, a second image frame, and a third image frame, wherein the time interval between the second image frame and the first image frame is a first time interval, the time interval between the third image frame and the second image frame is a second time interval, and the first time interval is different from the second time interval; the processing unit is also used to synthesize the first image frame based on the first frame rate to obtain a first synthesized image frame, synthesize the second image frame to obtain a second synthesized image frame, and synthesize the third image frame to obtain a third synthesized image frame; the display unit is also used to display the first synthesized image frame in a first period, the second synthesized image frame in a second period, and the third synthesized image frame in a third period, the time interval between the first period and the second period is a third time interval, the time interval between the second period and the third period is a fourth time interval, and the third time interval is the same as the fourth time interval.

[0022] In one possible implementation, the processing unit is further used to: when it is determined in the fourth cycle that the first moment is greater than or equal to the target time interval determined based on the first frame rate, synthesize the first image frame in the fourth cycle to obtain a first synthesized image frame; when it is determined in the fifth cycle that the second moment and the fifth time interval between the time of synthesizing the first image frame is greater than or equal to the target time interval, synthesize the second image frame in the fifth cycle to obtain a second synthesized image frame; the fifth cycle is after the fourth cycle; when it is determined in the sixth cycle that the third moment and the sixth time interval between the time of synthesizing the second image frame is greater than or equal to the target time interval, synthesize the third image frame in the sixth cycle to obtain a third synthesized image frame; the sixth cycle is after the fifth cycle.

[0023] In one possible implementation, the processing unit is further used to: when it is determined in the fifth cycle that the fifth time interval is less than the target time interval, not synthesize the second image frame in the fifth cycle; when it is determined in the fifth cycle that the fifth time interval between the second moment and the time for synthesizing the first image frame is greater than or equal to the target time interval, synthesize the second image frame in the fifth cycle to obtain a second synthesized image frame, including: when it is determined in the seventh cycle that the fourth moment and the seventh time interval between the time for synthesizing the first image frame is greater than or equal to the target time interval, synthesize the second image frame in the seventh cycle to obtain a second synthesized image frame; the seventh cycle is after the fifth cycle, and the seventh cycle is before the sixth cycle.

[0024] In a possible implementation, the first image frame, the second image frame, and the third image frame are all image frames of the preview layer.

[0025] In one possible implementation, a terminal device includes: a camera module, a camera hardware abstraction layer service, and before sequentially drawing and rendering to obtain a first image frame, a second image frame, and a third image frame, the method also includes: a processing unit, further used to: generate a first image sequence based on a first frame rate, and send the first image sequence to the camera hardware abstraction layer service; wherein the first image sequence includes: a first original image frame, a second original image frame, and a third original image frame; the time interval between the first original image frame and the second original image frame is an eighth time interval, and the time interval between the second original image frame and the third original image frame is a ninth time interval, and the eighth time interval is the same as the ninth time interval; image processing is performed on the first original image frame to obtain a first process image frame, image processing is performed on the second original image frame to obtain a second process image frame, and image processing is performed on the third original image frame to obtain a third process image frame.

[0026] In one possible implementation, a terminal device includes: a camera service, and a layer view, and the method also includes: a processing unit, which is further used to: when receiving a first process image frame, a second process image frame, and a third process image frame sent by a camera hardware abstraction layer service, set a first identifier for the first process image frame, the second process image frame, and the third process image frame, respectively; wherein the first identifier indicates a first frame rate; the camera service sends the first process image frame with the first identifier, the second process image frame with the first identifier, and the third process image frame with the first identifier to the layer view; draw and render the first process image frame in sequence to obtain a first image frame, draw and render the second process image frame to obtain a second image frame, and draw and render the third process image frame to obtain a third image frame.

[0027] In one possible implementation, the terminal device includes: a display synthesis system, a processing unit, and is further used to: upon receiving a first identifier sent by a layer view, synthesize the first image frame based on a first frame rate indicated in the first identifier to obtain a first synthesized image frame, synthesize the second image frame to obtain a second synthesized image frame, and synthesize the third image frame to obtain a third synthesized image frame.

[0028] In a possible implementation, the first identifier is obtained based on the SetConsumeRate interface, and the SetConsumeRate interface is used to instruct the layer view to pass the first identifier to the display composition system.

[0029] In a possible implementation, the first frame rate is less than 60 Hz, so that the terminal device can maintain a relatively stable picture at a frame rate below 60 Hz.

[0030] In a fourth aspect, an embodiment of the present application provides an apparatus for stabilizing image frames, and the method is applied to a terminal device, the terminal device including: a layer view, and a display synthesis system, the method including: a processing unit for receiving a first operation of a user; in response to the first operation, starting a camera application, the processing unit is further used to draw and render in sequence to obtain a first image frame, a second image frame, and a third image frame; wherein the time interval between the second image frame and the first image frame is a first time interval, the time interval between the third image frame and the second image frame is a second time interval, and the first time interval is different from the second time interval; the processing unit is further used to send the first image frame to the display synthesis system within an eighth cycle based on the first frame rate, send the second image frame to the display synthesis system within a ninth cycle based on the first frame rate, and send the third image frame to the display synthesis system within a tenth cycle based on the first frame rate; the time interval between the eighth cycle and the ninth cycle is a tenth time interval, the time interval between the ninth cycle and the tenth cycle is an eleventh time interval, and the tenth time interval is the same as the eleventh time interval; the processing unit is further used to synthesize the first image frame to obtain a first synthesized image frame, synthesize the second image frame to obtain a second synthesized image frame, and synthesize the third image frame to obtain a third synthesized image frame.

[0031] In one possible implementation, the processing unit is further used to: when it is determined in the eighth cycle that the fifth moment is greater than or equal to the target time interval determined based on the first frame rate, send the first image frame to the display synthesis system in the eighth cycle; when it is determined in the ninth cycle that the twelfth time interval between the sixth moment and the time when the first image frame is sent to the display synthesis system is greater than or equal to the target time interval, send the second image frame to the display synthesis system in the ninth cycle; when it is determined in the tenth cycle that the thirteenth time interval between the seventh moment and the time when the second image frame is sent to the display synthesis system is greater than or equal to the target time interval, send the third image frame to the display synthesis system in the tenth cycle.

[0032] In one possible implementation, the processing unit is further used to: not send the second image frame to the display synthesis system within the ninth cycle when the twelfth time interval within the ninth cycle is less than the target time interval; send the second image frame to the display synthesis system within the ninth cycle when the twelfth time interval between the sixth moment determined within the ninth cycle and the time when the first image frame is sent to the display synthesis system is greater than or equal to the target time interval, including: sending the second image frame to the display synthesis system within the eleventh cycle when the fourteenth time interval between the eighth moment determined within the eleventh cycle and the time when the first image frame is sent to the display synthesis system is greater than or equal to the target time interval; wherein the eleventh cycle is after the ninth cycle and before the tenth cycle.

[0033] In a possible implementation, the first image frame, the second image frame, and the third image frame are all image frames of the preview layer.

[0034] In a possible implementation, the first frame rate is less than 60 Hz.

[0035] In a fifth aspect, an embodiment of the present application provides a terminal device comprising a processor and a memory, the memory being used to store code instructions; the processor being used to run the code instructions so that the terminal device executes the method described in the first aspect or any one of the implementations of the first aspect, or executes the method described in the second aspect or any one of the implementations of the second aspect.

[0036] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed, the computer executes the method described in the first aspect or any one of the implementations of the first aspect, or executes the method described in the second aspect or any one of the implementations of the second aspect.

[0037] In the seventh aspect, a computer program product comprises a computer program, which, when run, enables a computer to execute the method described in the first aspect or any one of the implementations of the first aspect, or to execute the method described in the second aspect or any one of the implementations of the second aspect.

[0038] In an eighth aspect, a chip includes a processor, and the processor is used to call a computer program in a memory so that the computer program performs the method described in the first aspect or any implementation of the first aspect, or executes the method described in the second aspect or any implementation of the second aspect.

[0039] It should be understood that the second to seventh aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG1 is a schematic diagram of a scenario provided by an embodiment of the present application;

[0041] FIG2 is a schematic diagram of a software architecture of a terminal device provided in an embodiment of the present application;

[0042] FIG3 is a schematic diagram of the software architecture of another terminal device provided in an embodiment of the present application;

[0043] FIG4 is a schematic diagram of a consumption buffer provided in an embodiment of the present application;

[0044] FIG5 is a schematic diagram of a return frame time provided in an embodiment of the present application;

[0045] FIG6 is a schematic diagram of another consumption buffer provided in an embodiment of the present application;

[0046] FIG7 is a schematic diagram of another consumption buffer provided in an embodiment of the present application;

[0047] FIG8 is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0048] FIG9 is a method for stabilizing image frames provided by an embodiment of the present application;

[0049] FIG10 is a schematic diagram of yet another consumption buffer provided in an embodiment of the present application;

[0050] FIG11 is another method for stabilizing image frames provided by an embodiment of the present application;

[0051] FIG12 is a schematic diagram of yet another consumption buffer provided in an embodiment of the present application;

[0052] FIG13 is a schematic structural diagram of a device for stabilizing image frames provided in an embodiment of the present application;

[0053] FIG14 is a schematic diagram of the hardware structure of another terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first value and the second value are merely used to distinguish different values ​​and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit different values.

[0055] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0056] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple.

[0057] Figure 1 is a schematic diagram of a scenario provided by an embodiment of the present application. In the embodiment corresponding to Figure 1, a mobile phone is used as an example for illustration, and this example does not constitute a limitation on the embodiment of the present application.

[0058] When the terminal device receives an operation from the user to open the camera application, the terminal device can capture an image sequence based on the camera, and through image processing and display of each image frame in the image sequence, the terminal device can display the interface shown in Figure 1a. This interface can be a preview screen corresponding to the photo function, and the preview screen can include: target 101, which can be located in the middle of the preview screen. The interface shown in Figure 1a can also include controls for starting photography, controls for opening an album, controls for switching cameras, and other function controls corresponding to the camera application.

[0059] However, due to the differences between the image frames, such as the display content and / or lighting effects of each image frame, the frame output time of each image frame after image processing is uneven, causing the camera preview screen to freeze. For example, when the target 101 moves to the lower left, the position of the target 101 in the preview screen should also move to the lower left. However, due to the uneven frame output time of each image frame, the target 101 in the interface shown in Figure 1b should be at the dotted line position, but due to the screen freeze, the target 101 is still in the middle position of the preview screen. The content displayed in the preview screen is inconsistent with the actual situation of the target, which brings a poor user experience.

[0060] Specifically, the process of displaying the preview screen on the terminal device can refer to the description of the embodiment corresponding to Figure 3. Figure 2 is a schematic diagram of the software architecture of a terminal device provided in an embodiment of the present application.

[0061] As shown in Figure 2, the terminal device's layered architecture divides the software into several layers, each with distinct roles and divisions of labor. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into multiple layers: from top to bottom, the application (APP) layer, the application framework layer, the system library layer, the hardware abstraction layer (HAL), and the kernel layer.

[0062] The application layer provides the applications required by the terminal device. The application layer may include: camera, desktop, sharing, Bluetooth, voice interaction applications, and contacts. The camera application can be a program that comes with the terminal device system or a third-party application.

[0063] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0064] The application framework layer may also include: a camera service, a surface flinger (SF), views, an input manager, and a buffer queue. Views include visual controls, such as controls for displaying text or images. Views can be used to build applications. A display interface can be composed of one or more views. For example, a display interface that includes a text notification icon may include a view for displaying text and a view for displaying images.

[0065] Among them, the camera service, or camera framework, is used to create a bufferqueue; the surface flinger is used to provide layer synthesis services, responsible for layer rendering, layer classification, and layer synthesis for each layer.

[0066] The system library may include: browser kernel, 3D graphics, font library, etc. The terminal device may also include Android runtime.

[0067] HAL is used to abstract hardware and can provide an interface for querying hardware devices for upper-level applications, or it can also provide data storage services for upper-level applications.

[0068] The HAL may include: camera hardware abstraction layer services (camera HAL), display, sensor, and hardware composer (HWC).

[0069] Among them, the camera HAL or can be called the camera provider, which is used to perform image processing on the input raw image frames, such as image cropping, image special effects processing, anti-shake processing, and image recognition, and input the processed image frames into the camera service; HWC is used to overlay the layer data prepared by the surface flinger and forward it to the display driver.

[0070] The kernel layer may include: camera module (camera sensor), display driver, Bluetooth driver, ultra wide band (UWB) driver, sensor driver, touch screen driver and audio driver, etc.

[0071] Among them, the camera sensor is used to obtain the original image frame image according to the instruction, and the original image sequence may include multiple original image frames. The format of any original image frame among the multiple original image frames can be unprocessed (RAW) format; the display driver is used to display the image frame input by the HWC.

[0072] It is understandable that the software architecture of the terminal device may also include other content, which is not limited in the embodiments of the present application.

[0073] Based on the embodiment corresponding to Figure 2, the display process of the preview screen by the terminal device may involve: camera, bufferqueue, camera service, camera HAL, surface flinger, HWC, display driver and camera sensor, etc.

[0074] For example, FIG3 is a schematic diagram of the software architecture of another terminal device provided in an embodiment of the present application.

[0075] The camera can include: a surface view, which can be used to create a BLAST buffer queue (or BBQ for short). The camera (or surface view in the camera) can be used to render the received image frames.

[0076] When the terminal device receives an operation such as the user opening the camera application, the surface view in the camera application can create a bufferqueue and BBQ, and the surface view can also instruct the camera service to capture image frames.

[0077] During the creation of the bufferqueue, the bufferqueue can establish a connection between the consumer surface view and the producer camera service. The consumer will register an onFrameAvailable listener with the bufferqueue. When the producer calls the queueBuffer function to transfer the buffer to the bufferqueue, the consumer will receive a callback message (or callback function).

[0078] During the process of creating BBQ, BBQ can establish a connection between the producer surface view and the consumer surface flinger, and BBQ can also act as a consumer of bufferqueue.

[0079] When the surface view instructs the camera service to capture image frames, the surface view can send instructions for capturing image frames to the camera service (or instructions for creating a preview stream for the user); the camera service will generate a set of register configurations (or called output parameters) for the camera sensor to configure the camera sensor's output mode, image size, and output frame rate parameters, etc. The camera service sends the instructions for capturing image frames and the output parameters to the camera HAL; the camera HAL sends the instructions for capturing image frames and the output parameters to the camera sensor. Optionally, when the camera sensor receives the instructions for capturing image frames and the output parameters, it can capture the original image sequence according to the output parameters and send the original image sequence to the camera HAL; the camera HAL performs image processing on each original image frame in the original image sequence and returns the processed image frames to the camera service.

[0080] Furthermore, the camera service caches the processed image frame (or buffer) returned by the camera HAL into the bufferqueue; the bufferqueue sends the buffer to the surface view and notifies the surface view to consume it; the surface view calls the acquireBuffer function to consume the buffer, and calls the queueBuffer function to send the buffer to the BBQ; the BBQ sends the buffer to the surface flinger through the transaction object; the surface flinger can perform layer synthesis on the received buffer according to the consumption rate received by the surface flinger, such as the surface flinger can perform layer synthesis on the buffer when it receives the vertical synchronization (Vsync) event.

[0081] Among them, the Vsync signal is a periodic signal, and the Vsync signal period can be set according to the screen refresh rate. For example, when the screen refresh rate is 60Hz, the Vsync signal period can be 16.6ms, that is, the terminal device generates a control signal every 16.6ms to trigger the Vsync signal period.

[0082] It should be noted that Vsync signals can be divided into software Vsync signals and hardware Vsync signals. Software Vsync signals include Vsync-APP and Vsync-SF. Vsync-APP is used to trigger the application's rendering process for image frames. Vsync-SF is used to trigger the surface flinger's layer compositing process for image frames. Hardware Vsync signals are used to trigger the screen display refresh process.

[0083] Typically, the software Vsync signal and the hardware Vsync signal maintain periodic synchronization. For example, if the hardware Vsync signal switches from 60Hz to 120Hz, Vsync-APP and Vsync-SF will also change synchronously, switching from 60Hz to 120Hz.

[0084] For example, when the surface flinger receives a buffer, the surface flinger can request a Vsync signal; when the surface flinger determines that there are other buffers that need to be consumed, it can request a Vsync signal.

[0085] For example, the surface flinger can initiate the process of layer compositing on the buffer upon receiving any Vsync signal. Upon receiving any Vsync signal, the surface flinger can determine whether a buffer is received. If a buffer is received, it performs layer compositing on it. If no buffer is received, the Vsync signal is not used in the image frame composition. In a possible implementation, the surface flinger can also release the buffer after processing it, so that the buffer is released by the surface flinger and the BBQ to the bufferqueue.

[0086] The embodiment of the present application provides a schematic diagram of a normal consumption buffer of a terminal device. Figure 4 is a schematic diagram of a consumption buffer provided by the embodiment of the present application.

[0087] As shown in FIG4 , in chronological order, the contents displayed by the terminal device may be: frame 1, frame 2, frame 3, frame 4 and frame 5.

[0088] Taking the display of frame 1 as an example, the camera sensor of the terminal device captures frame 1, and based on the description in the embodiment corresponding to Figure 2, sends frame 1 to the application along the camera sensor, camera HAL and camera service, and the camera application draws and renders frame 1. After the drawing and rendering of frame 1 is completed, the application of the terminal device sends the drawn and rendered frame 1 to the surface flinger through BBQ. The surface flinger synthesizes the drawn and rendered frame 1. After the synthesis of frame 1 is completed, the terminal device can start the display driver by calling the kernel layer and display the content corresponding to frame 1 on the screen. Similarly, the synthesis and display process of frames 2, 3, 4 and 5 can be referred to frame 1 and will not be repeated here. In Figure 3, each frame lags by 2 Vsync signal cycles from drawing to display.

[0089] It can be understood that when frames 1, 2, 3, 4 and 5 are evenly sent from the camera sensor to the application and evenly transmitted from the application to the surface flinger, the terminal device can smoothly display frames 1, 2, 3, 4 and 5 on the screen.

[0090] However, due to the differences in raw image frames, the camera HAL takes different amounts of time to process each frame. This results in uneven frame return times from the camera HAL to the camera service. This can cause lag when the terminal device displays image frames with uneven frame return times. The frame return time is defined as the time it takes for the camera HAL to send the processed frame to the camera service.

[0091] For example, FIG5 is a schematic diagram of a return frame time interval provided in an embodiment of the present application. As shown in FIG5 , the horizontal axis can be understood as the sequence number of each image frame, and the vertical axis can be understood as the return frame time interval between the return frame time of the current image frame and the return frame time of the previous image frame. It can be seen that the return frame time interval between the image frame with sequence number 2 and the image frame with sequence number 1 can be approximately 18 milliseconds (ms); the return frame time interval between the image frame with sequence number 4 and the image frame with sequence number 3 can be approximately 63ms; the return frame time interval between the image frame with sequence number 7 and the image frame with sequence number 6 can be approximately 10ms.

[0092] It is understandable that the camera HAL takes different amounts of time to process each raw image frame, resulting in uneven time intervals between adjacent image frames in the image sequence emitted by the camera HAL. As shown in FIG5 , the minimum time interval between the two frames can be around 10 ms, and the maximum can be around 60 ms.

[0093] Typically, the terminal device can provide two specific situations where the preview image is stuck due to uneven frame return time of the camera HAL.

[0094] In one implementation, the camera HAL can mark the first expected display timestamp in the buffer. The surface flinger will also generate a second expected display timestamp corresponding to the buffer, and the surface flinger can compare the first expected display timestamp corresponding to the buffer with the second expected display timestamp corresponding to the buffer. If the first expected display timestamp is earlier than the second expected display timestamp, the surface flinger will take out all buffers, and the bufferqueue will eliminate the old buffers, retaining only the latest buffer that meets the requirements for consumption. In this case, when the camera HAL returns frames quickly, some buffers will be discarded, causing the terminal device to display lag in the preview screen.

[0095] For example, Figure 6 is a schematic diagram of another consumption buffer provided by an embodiment of the present application. As shown in Figure 6, due to the uneven frame return timing of the camera HAL, the application can receive frames 1, 2, 3, and 4 within the Vsync1 cycle. The first expected display timestamp marked on frame 1 is timestamp 1, the first expected display timestamp marked on frame 2 is timestamp 2, the first expected display timestamp marked on frame 3 is timestamp 3, and the first expected display timestamp marked on frame 4 is timestamp 4.

[0096] Furthermore, the application can set frame 1, frame 2, frame 3 and frame 4 in the BBQ. After the surface flinger receives frame 1, frame 2, frame 3 and frame 4, the second expected display timestamp can be set for frame 1, frame 2, frame 3 and frame 4 respectively. When the surface flinger determines that timestamp 1 of frame 1, timestamp 2 of frame 2, and timestamp 3 of frame 3 are all earlier than the second expected display timestamp, and timestamp 4 of frame 4 is later than the second expected display timestamp, the surface flinger can retain the latest buffer, such as consuming frame 3, and display frame 3 within the period of Vsync3. In this scenario, the discarded frames 1, 2 and 3 will cause a freeze in the display of the preview screen.

[0097] In another implementation, the surface flinger consumes one buffer at a time during the buffer compositing process, and the remaining buffers are stored in the bufferqueue. When processing any buffer, the surface flinger only knows the timestamp of the frame returned by the camera HAL, but has no idea when the camera sensor actually outputs the image. This means that even if the camera sensor outputs images at a certain frame rate, the time when the surface flinger actually composites the buffers and outputs the image is postponed according to the timestamp of the camera HAL frame return. Therefore, the uneven timestamps of the camera HAL frame return can cause preview screen lag.

[0098] For example, Figure 7 is a schematic diagram of another consumption buffer provided by an embodiment of the present application. As shown in Figure 7, when the camera HAL frame return time is uneven, the time interval between two frames when the application performs image rendering may also be uneven. For example, the application may render frame 1 within the Vsync1 cycle, frame 2 within the Vsync2 cycle, frame 3 within the Vsync4 cycle, frame 4 within the Vsync5 cycle, and frame 5 within the Vsync8 cycle.

[0099] Furthermore, the application can send the rendered frames to the surface flinger via BBQ. At this time, when the surface flinger synthesizes each frame, the synthesis time interval between adjacent frames is also uneven, resulting in uneven display of adjacent frames on the screen. For example, the screen displays frame 1 within the Vsync3 cycle, displays frame 2 within the Vsync4 cycle, and displays frame 3 within the Vsync6 cycle, etc., which gives the user a sense of lag.

[0100] In view of this, an embodiment of the present application provides a method for stabilizing image frames, so that a terminal device can ensure the smoothness of the preview screen when performing layer synthesis on the image frames.

[0101] It is understandable that the above-mentioned terminal devices may also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The terminal devices may be mobile phones with touch screens, smart TVs, wearable devices, tablet computers (Pads), computers with wireless transceiver functions, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the terminal devices.

[0102] Therefore, in order to better understand the embodiment of the present application, the structure of the terminal device of the embodiment of the present application is introduced below. For example, FIG8 is a schematic diagram of the hardware structure of a terminal device provided in the embodiment of the present application.

[0103] The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, an indicator 192, a camera 193, and a display screen 194. The sensor module 180 may include one or more of the following: a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, or a bone conduction sensor.

[0104] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0105] The processor 110 may include one or more processing units. The different processing units may be independent devices or integrated into one or more processors. The processor 110 may also be provided with a memory for storing instructions and data.

[0106] USB interface 130 is an interface that complies with USB standards and specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. USB interface 130 can be used to connect a charger to charge the terminal device, or to transfer data between the terminal device and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices.

[0107] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 141 is used to connect the charging management module 140 to the processor 110.

[0108] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0109] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. The antennas in a terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0110] Mobile communication module 150 can provide wireless communication solutions for terminal devices, including 2G / 3G / 4G / 5G. It can include at least one filter, switch, power amplifier, and low-noise amplifier (LNA). Mobile communication module 150 receives electromagnetic waves from antenna 1, filters and amplifies the received electromagnetic waves, and transmits them to the modem processor for demodulation.

[0111] The wireless communication module 160 can provide wireless communication solutions applied to terminal devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), etc.

[0112] The terminal device implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

[0113] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0114] The terminal device can realize the shooting function through the ISP, camera 193, video codec, GPU, display 194 and application processor.

[0115] The camera 193 is used to capture static images or videos. In some embodiments, the terminal device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0116] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0117] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 can include a program storage area and a data storage area.

[0118] The terminal device can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0119] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The speaker 170A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The terminal device can listen to music or listen to hands-free calls through the speaker 170A. The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the terminal device answers a call or voice message, the voice can be heard by placing the receiver 170B close to the human ear. The headphone jack 170D is used to connect a wired headset. The microphone 170C, also known as the "microphone" or "microphone", is used to convert sound signals into electrical signals. In an embodiment of the present application, the terminal device may have a microphone 170C.

[0120] Keys 190 include a power button, a volume button, and other buttons. Keys 190 can be mechanical or touch-sensitive. The terminal device can receive key inputs and generate key signal inputs related to user settings and function control of the terminal device. Indicator 192 can be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and the like.

[0121] The software system of the terminal device can adopt a layered architecture, event-driven architecture, micro-kernel architecture, microservice architecture, or cloud architecture, etc., which will not be described here.

[0122] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other. For the same or similar concepts or processes, some embodiments may not be described in detail.

[0123] To address the issue of display freezes on terminal devices caused by uneven frame return times of the camera HAL, an embodiment of the present application provides a method for stabilizing image frames. The image frame stabilization process can be performed before the display synthesis system synthesizes the image frames (see the embodiment corresponding to FIG9 ), or after the application renders the image frames (see the embodiment corresponding to FIG11 ).

[0124] For example, FIG9 shows a method for stabilizing image frames according to an embodiment of the present application. In the embodiment corresponding to FIG9 , the terminal device may include: a surface view, a camera HAL, a camera service, and a surface flinger.

[0125] As shown in FIG9 , the method for stabilizing image frames may include the following steps:

[0126] S901. Surface view sends an instruction to camera service to create a 30 Hz preview stream.

[0127] The 30Hz in the instruction for creating a 30Hz preview stream can be used as an example and cannot be used to limit the embodiments of the present application. The 30Hz can also be referred to as the first frame rate. It is understandable that since the terminal device may experience more noticeable freezes when the frame rate is low, the first frame rate can generally be less than 60Hz.

[0128] For example, when Surface View receives a user request to open the preview screen, it can send a command to the Camera Service to create a 30Hz preview stream. In a possible implementation, the Camera Service can also return a response message to Surface View. The operation to open the preview screen can be: opening the camera app, or switching to the preview screen from other functions of the camera app, etc., which is not limited in this embodiment of the application.

[0129] S902: The camera service sends image output parameters to the camera HAL.

[0130] The image output parameters may include one or more of the following: an image output mode, an image size of an image output, or a first frame rate (such as 30 Hz) and other data.

[0131] Suitably, when the camera HAL receives the image output parameters, the camera HAL may send the image output parameters to the camera sensor, so that the camera sensor can output images according to the image output parameters to generate a raw image sequence; the camera sensor sends the raw image sequence to the camera HAL for image processing; further, the camera HAL may execute the step shown in S905 after performing image processing on each image in the raw image sequence.

[0132] In a possible implementation, when the camera HAL receives the image output parameters, it can also return a response message to the camera service.

[0133] S903. Surface view creates a BBQ (the producer is surface view and the consumer is surface flinger).

[0134] Among them, BBQ can also be a consumer of bufferqueue.

[0135] S904. Surface view creates a bufferqueue (the producer is the camera service and the consumer is the surface view).

[0136] Among them, the specific steps of surface view creating BBQ and bufferqueue can be found in the description of the embodiment corresponding to Figure 2, which will not be repeated here.

[0137] S905. The camera HAL sends the target cache data (layer name) to the camera service.

[0138] The target buffer data (or target buffer) can be the image processing data obtained by the camera HAL, and any buffer data can correspond to a layer name. The target buffer data can be understood as any buffer generated when the camera HAL outputs images at a 30Hz output frame rate. When the target buffer data is used to support the preview image, the layer name can be the camera's layer view (or preview layer), such as SurfaceView[camera].

[0139] Exemplarily, the camera HAL can receive the original image sequence returned by the camera sensor and perform image processing on each image frame in the original image sequence. For example, the camera HAL can perform image processing on the original cache data in the original image sequence to obtain target cache data, and then send the target cache data and the layer name corresponding to the target cache data to the camera service.

[0140] It is understandable that due to the differences between the image frames in the original image sequence, the camera HAL takes different amounts of time to process each frame. Consequently, the camera HAL returns frames to the camera service at uneven times, resulting in preview freezes. This image processing process may include one or more of the following: image cropping, image effects processing, stabilization, or image recognition, which are not limited in this embodiment.

[0141] S906. The camera service adds a consumption rate identifier to the target cache data and sends the target cache data (consumption rate identifier, layer name) to the surface view.

[0142] The consumption rate identifier may indicate a target consumption rate, which may be the same as the first frame rate, for example, 30 Hz. This consumption rate identifier enables the surface flinger to determine the synthesis of target cache data based on the target consumption rate, thereby reducing image lag caused by uneven frame output timing of the camera HAL.

[0143] For example, the camera service can create a BufferItem object, associate it with the camera HAL's target buffer data, set a consumption rate flag for the target buffer data in the BufferItem object, and then call the bufferqueue interface to send the BufferItem object to the surface view. It is understood that the camera service can set a consumption rate flag for any buffer it receives.

[0144] S907. The surface view sends the target cache data (consumption rate identifier, layer name) to the surface flinger through BBQ.

[0145] In a possible implementation, when the surface view receives the target cache data (consumption rate identifier, layer name), it can determine whether the consumption rate identifier of the target cache data and the layer name of the target cache data have changed compared to the consumption rate identifier and the layer name of the previous buffer. For example, when the surface view determines that the consumption rate identifier of the target cache data has changed, it sends the target cache data with the consumption rate identifier to the surface flinger; or, when it determines that the layer name of the target cache data has changed, it sends the target cache data with the layer name to the surface flinger; or, when the surface view determines that neither the consumption rate identifier nor the layer name has changed, it can send the target cache data to the surface flinger.

[0146] It is understandable that even if the surface flinger receives the target cache data but does not receive the consumption rate identifier and layer name, the surface flinger can determine that the consumption rate identifier and layer name of the target cache data have not changed based on the layer name and consumption rate identifier of the previously cached buffer.

[0147] When the surface view sends the target cache data to the surface flinger through BBQ, the surface view can receive the target cache data in the BufferItem object; the surface view calls the acquireBuffer function to consume the target cache data, and calls the queueBuffer function to send the BufferItem object to the BBQ; after receiving the target cache data, the BBQ calls the setTransaction interface to pass the target cache data to the surface flinger through the transaction object.

[0148] It can be understood that the transaction object is a medium object for communicating with the surface flinger. In addition to the target cache data, the transaction object can also include data corresponding to the surface, such as the layer name, the length and width of the interface that needs to be updated, transparency, and zoom and translation parameters. Since there are many parameters that need to be sent to the surface flinger within one frame, the surface view will set all the above-mentioned surface-related parameters in the transaction object and send them to the surface flinger together to reduce process communication time. Similarly, in addition to setting the target cache data in the transaction object, the bufferqueue can also put the consumption rate identifier corresponding to the buffer, buffer id update data, format change data, buffer size change data, etc. into the transaction object, and then send them to the surface flinger together.

[0149] In an embodiment of the present application, the added interface 1 may be the interface SetConsumeRate1, which is used to instruct the camera application to pass the consumption rate identifier of the current layer to the Surface view.

[0150] The added interface 2 may be SetConsumeRate2, which is used to instruct the surface view to pass the consumption rate identifier of the current layer and the layer name to the surface flinger.

[0151] The added interface 3 can be SetConsumeRate3, which is used to instruct the camera service to pass the consumption rate identifier of the current layer to the bufferqueue.

[0152] The added interface 4 may be SetConsumeRate4, which is used to instruct the surface view to transmit the BBQ consumption rate identifier to the BBQ.

[0153] When an application needs to set a consumption rate identifier, it can be set in the buffer or in the surface. The buffer and the surface are interrelated. Before the transaction object sends parameters to the surface flinger, they will be associated with the layer_state_t structure object, which contains all the information about a layer change, such as whether the layer is visible, the offset of the layer, whether it is a safe layer, the surfaceid (or layer name), the bufferid (for example, the name of the target cache data), the length and width of the interface, transparency, and zoom and translation parameters. After the surface flinger receives the layer_state_t, it can parse out which layer it corresponds to and what the layer name is within the current process.

[0154] When creating a transaction object, the sc parameter can be a pointer to the layer, and the rate parameter is the target consumption rate. After calling the setConsumeRate method to set the consumption rate, the camera application can call the apply method to send the parameters to the surface flinger. The sc parameter can be passed in the BBQ, which stores the layer corresponding to the current bufferqueue.

[0155] The surface view can implement a drawing and rendering function for the target cache data, and after drawing and rendering the target cache data, send the rendered first target cache data to the surface flinger through the BBQ.

[0156] S908: The surface flinger caches the first target cache data in a transaction queue.

[0157] S909. The surface flinger obtains the target consumption rate from the consumption rate identifier, and obtains the last consumption time corresponding to the layer name of the first target cache data.

[0158] The surface flinger may store a first mapping relationship between a layer and the last consumption time corresponding to the layer. The last consumption time described in step S909 may be understood as the time when synthesis of the last cached data of the layer began. The surface flinger may store the first mapping relationship in a hash table or other manner.

[0159] For example, when the surface flinger receives a Vsync signal, the surface flinger can traverse the transaction queue. If the first target cache data in the transaction queue is set with a consumption rate flag, the surface flinger can obtain the layer name corresponding to the first target cache data, and then obtain the last consumption time corresponding to the layer name of the first target cache data according to the first mapping relationship.

[0160] For example, when the layer name of the first target cache data received by the surface flinger is SurfaceView[camera], the surface flinger can find the consumption time of the last cache data consumed by the surface flinger of the SurfaceView[camera].

[0161] In a possible implementation, the surface flinger can also be configured with a correspondence between layers and the consumption rates corresponding to the layers, so that upon receiving the first target cache data and the layer name of the first target cache data, the surface flinger can determine the target consumption rate based on the layer name corresponding to the first target cache data. In this case, even if the surface flinger does not receive the consumption rate identifier, it can still determine the synthesis status of the first target cache data based on the preset consumption rate associated with the layer.

[0162] S910 : When determining that the target consumption rate meets a preset range, the surface flinger determines whether to perform layer synthesis on the first target cache data according to the last consumption time and the current system time.

[0163] The preset range may include a consumption rate that satisfies a range of 1 Hz to 120 Hz, for example, 1 Hz to 60 Hz, etc., which is not limited in the present embodiment. Any consumption rate within the preset range may be less than the system refresh rate.

[0164] For example, when the surface flinger determines that the target consumption rate meets a preset range, the surface flinger can calculate a first difference between the current system time and the last consumption time corresponding to the layer of the target cache data. For example, the first difference can be the value obtained by subtracting the last consumption time corresponding to the layer of the target cache data from the current system time. The first difference can be understood as the actual time interval between the time when the first target cache data is layer synthesized and the time when the previous buffer of the first target cache data is layer synthesized.

[0165] Furthermore, when the surface flinger determines that the first difference is less than 1 second / target consumption rate*m, the surface flinger may not perform layer synthesis on the first target buffer data. In a possible implementation, when the surface flinger determines that layer synthesis is not required for the first target buffer data, the surface flinger may continue to determine whether to perform layer synthesis on the first target buffer data when the next Vsync signal arrives.

[0166] The 1 second / target consumption rate can be understood as the time required to achieve normal layer synthesis of the first target cache data, or it can also be understood as the theoretical time interval between the time it takes for the first target cache data to be synthesized and the time it takes for the previous buffer of the first target cache data to be synthesized. The value of m can be 0.75 or 0.8, or it can be within the range of 0.5-1. m can be used to offset screen freezes caused by uneven frame output time of the camera HAL. The value of m is not limited in the embodiments of this application.

[0167] It can be understood that the first difference is less than 1 second / target consumption rate*m, which can be understood as: the actual time interval between the first target cache data and the previous buffer of the first target cache data is less than the theoretical time interval between the two buffers, that is, in order to ensure the target consumption rate, the surface flinger cannot complete the layer synthesis of the first target cache data within the actual time interval.

[0168] Alternatively, when the surface flinger determines that the first difference is greater than or equal to 1 second / target consumption rate*m, the surface flinger may perform layer synthesis on the first target cache data and set the last consumption time of the layer to the current system time.

[0169] It can be understood that the first difference greater than or equal to 1 second / target consumption rate*m can be understood as: the actual time interval between the two buffers is greater than or equal to the theoretical time interval between the two buffers, that is, the surface flinger can complete the layer synthesis of the first target cache data within a sufficient time interval.

[0170] In a possible implementation, when the surface flinger determines that the target consumption rate does not meet the preset range, such as when the target consumption rate is 0Hz, -1Hz, or a value greater than 120Hz, the surface flinger can determine that no special processing is required for the target consumption rate, perform layer synthesis on the first target cache data, and set the last consumption time corresponding to the layer name to 0.

[0171] In a possible implementation, if there is an unconsumed buffer in the surface flinger, for example, the transaction queue includes other data in addition to the first target cache data, the surface flinger can request the next Vsync signal and, when the next Vsync signal arrives, continue to process the unconsumed buffer based on the steps shown in S909-S910.

[0172] After S910, the surface flinger can send the buffer after the layer synthesis to the HWC. After the HWC performs overlay and other processing on the buffer after the layer synthesis, it continues to forward it to the display driver for display, so that the terminal device displays the preview screen shown in a of Figure 1.

[0173] Based on this, even if the camera HAL's frame output time is uneven, the surface flinger can ensure sufficient layer synthesis time when synthesizing the buffer layer, and synthesize the layer at a stable consumption rate to reduce screen freezes.

[0174] In a possible implementation, the process of displaying the preview screen also includes the display of the control layer. For example, the camera HAL can obtain multiple control image frames based on the frame rate corresponding to the control layer; and send the multiple control image frames to the surface flinger along the camera HAL, camera service, and surface view. Since the frame rate corresponding to the control layer (such as 60Hz) is usually greater than the frame rate corresponding to the preview layer (such as the target consumption rate, 30hz), the surface flinger can obtain the control layer within the cycle of synthesizing any preview layer, and obtain the preview screen by synthesizing and displaying the preview layer and the control layer.

[0175] It is understandable that when the camera HAL in the step shown in S905 returns an image sequence, the terminal device can execute the steps shown in S906-S910 for each image frame in the image sequence, that is, perform a frame stabilization operation in the surface flinger, so that the terminal device can evenly display continuous image frames. For example, when the image sequence returned by the camera HAL includes: frame 1, frame 2 and frame 3, the time interval between frame 1 and frame 2 is interval 1, the time interval between frame 2 and frame 3 is interval 2, and the size of interval 1 is different from that of interval 2. Suitably, surface flinger can obtain a stable picture after processing the frame 1, frame 2 and frame 3.

[0176] For example, Figure 10 is a schematic diagram of another consumption buffer provided in an embodiment of the present application. In the embodiment corresponding to Figure 10, an example is provided in which the screen refresh rate is 60Hz and the target consumption rate determined by the surface flinger is 30Hz. When the target consumption rate is 30Hz and m is 0.8, the surface flinger can determine that the theoretical time interval between two adjacent buffers can be: 1 / 30Hz * 0.8, which is approximately 27ms.

[0177] As shown in Figure 10, the camera HAL's frame output time is uneven, causing the application to render frame 1 within the Vsync1 cycle, frame 2 within the Vsync2 cycle, frame 3 within the Vsync4 cycle, frame 4 within the Vsync5 cycle, and frame 5 within the Vsync8 cycle.

[0178] For frame 1, the application can render frame 1 within the Vsync1 period (such as about 0ms), and cache frame 1 in the BBQ after rendering; the surface flinger obtains frame 1. Since frame 1 is the first frame, the surface flinger can synthesize frame 1 within the Vsync1 period (such as about 16.6ms); the screen displays frame 1 within the Vsync3 period (such as about 33.2ms).

[0179] For frame 2, the application can render frame 2 within the Vsync2 cycle (such as about 16.6ms), and cache frame 2 in the BBQ after the rendering is completed; the surface flinger obtains frame 2. Since the actual time interval between the current system time of frame 2 (such as a certain moment in the Vsync3 cycle) and the consumption time of frame 1 is less than the theoretical time interval, the surface flinger can temporarily not synthesize frame 2. Furthermore, when the surface flinger receives Vsync4, it can determine that the actual time interval between the current system time of frame 2 (such as a certain moment in the Vsync4 cycle) and the consumption time of frame 1 is greater than the theoretical time interval, so the surface flinger can synthesize frame 2 within the Vsync4 cycle. Among them, the description of the theoretical time interval and the actual time interval can be found in the steps shown in S910.

[0180] Similarly, the way the surface flinger determines whether to synthesize any frame when the Vsync signal arrives can be seen in Frames 1 and 2, and will not be repeated here. As shown in Figure 10, the surface flinger can determine based on the target consumption rate and the time interval between the synthesis of two adjacent buffers, allowing the surface flinger to synthesize the buffers relatively evenly, alleviating screen freezes.

[0181] For example, FIG11 is another method for stabilizing an image frame provided by an embodiment of the present application. As shown in FIG11 , the method for stabilizing an image frame may include the following steps:

[0182] S1101. Surface view sends an instruction to camera service to create a 30 Hz preview stream.

[0183] S1102: The camera service sends image output parameters to the camera HAL.

[0184] S1103. Surface view creates a BBQ (the producer is surface view and the consumer is surface flinger).

[0185] S1104. Surface view creates a bufferqueue (the producer is the camera service and the consumer is the surface view).

[0186] S1105. The camera HAL sends the target cache data (layer name) to the camera service.

[0187] S1106. The camera service adds a consumption rate identifier to the target cache data and sends the target cache data (consumption rate identifier, layer name) to the surface view.

[0188] The description of steps S1101-S1106 can be found in steps S901-S906, and will not be repeated here.

[0189] S1107 : The surface view renders the target cache data to obtain second target cache data.

[0190] S1108. The surface view obtains the target consumption rate from the consumption rate identifier, and obtains the last sending time corresponding to the layer name of the second target cache data.

[0191] The surface view may store a first mapping between a layer and its corresponding last send time, where the last send time can be understood as the time when the last cached data of the layer was sent to the surface flinger. In possible implementations, the last send time can also be understood as the time when the last cached data of the layer was synthesized in the surface flinger.

[0192] For example, when the layer name of the second target cache data received by the surface view is SurfaceView[camera], the surface view can find the sending time of the last cache data consumed by the surface flinger of the SurfaceView[camera].

[0193] S1109. The surface view determines whether to send the second target cache data to the surface flinger according to the last sending time and the current system time.

[0194] For example, the surface view can calculate a second difference between the current system time and the last send time corresponding to the layer, such as the second difference can be the value obtained by subtracting the last send time corresponding to the layer from the current system time. The second difference can be understood as the actual time interval between the send time of the second target cache data and the send time of the previous buffer of the second target cache data sent to the surface flinger.

[0195] Furthermore, when the surface view determines that the second difference is less than 1 second / target consumption rate*m, the surface view may not send the second target cache data to the surface flinger in the current cycle. In a possible implementation, when the surface view determines that the second target cache data may not be sent to the surface flinger in the current cycle, the surface view may continue to determine whether to send the second target cache data to the surface flinger when the next Vsync signal arrives.

[0196] Alternatively, when the surface view determines that the second difference is greater than or equal to 1 second / target consumption rate*m, the surface flinger can send the second target cache data to the surface flinger and set the last sending time of the layer to the current system time.

[0197] It's understood that Surface View can perform frame stabilization for the target buffer data based on the target consumption rate, for example, determining whether to send the second target buffer data to the Surface Flinger within the VSync signal period. This way, when Surface View performs frame stabilization on each frame in the image sequence, it can ensure that the rendered image is output to the Surface Flinger as evenly as possible, thereby reducing image lag.

[0198] S1110. Surface view sends the second target cache data (consumption rate identifier, layer name) to surface flinger through BBQ.

[0199] S1111. The surface flinger caches the second target cache data in the transaction queue.

[0200] S1112. The surface flinger performs layer synthesis on the second target cache data.

[0201] In a possible implementation, when the camera HAL service in the step shown in S1105 returns an image sequence, the terminal device can execute the steps shown in S1106-S1112 for each image frame in the image sequence, that is, perform frame stabilization operations in the surface view, so that the terminal device can evenly display continuous image frames. For example, when the image sequence returned by the camera HAL includes: frame 1, frame 2, and frame 3, the time interval between frame 1 and frame 2 is interval 1, and the time interval between frame 2 and frame 3 is interval 2, and the size of interval 1 is different from that of interval 2. Suitably, the surface view can perform drawing and rendering processing on frame 1, frame 2, and frame 3, and send the three rendered frames to the surface flinger in sequence according to the target consumption rate, so that the terminal device can obtain a stable picture.

[0202] Based on this, even if the camera HAL's frame output time is uneven, the surface view can still send image frames stably to the surface flinger, allowing the surface flinger to perform layer synthesis based on a stable consumption rate to reduce screen freezes.

[0203] For example, Figure 12 is a schematic diagram of another consumption buffer provided in an embodiment of the present application. In the embodiment corresponding to Figure 12, an example is given in which the screen refresh rate is 60Hz and the target consumption rate determined by the surface flinger based on the layer is 30Hz. When the target consumption rate is 30Hz and m is 0.8, the surface flinger can determine that the theoretical time interval between two adjacent buffers can be: 1 / 30Hz*0.8, which is approximately 27ms. For a description of the theoretical time interval and the actual time interval, please refer to the step shown in S1109.

[0204] As shown in FIG12 , the camera HAL output frame time is uneven, and the image frame input time to the application is uneven. The application can send the image frames evenly to the surface flinger based on the method for stabilizing image frames in the embodiment corresponding to FIG11 .

[0205] For example, the application may render frame 1 in the Vsync1 cycle, frame 2 in the Vsync2 cycle, frame 3 in the Vsync4 cycle, frame 4 in the Vsync5 cycle, and frame 5 in the Vsync8 cycle.

[0206] After rendering, for any frame, such as frame 2, the actual time interval between the current system time of frame 2 and the time when frame 1 is sent to the BBQ (or surface flinger) is less than the theoretical time interval. Therefore, the application can temporarily not send frame 2 to the BBQ within the Vsync2 cycle. Furthermore, when the application receives Vsync3, it can determine that the actual time interval between the current system time of frame 2 and the time when frame 1 is sent to the BBQ (or surface flinger) is greater than the theoretical time interval. Therefore, the application can send frame 2 to the BBQ within the Vsync3 cycle.

[0207] Similarly, the manner in which the application determines whether to send frames 3, 4, and 5 when the Vsync signal arrives can be referred to the description of frame 2, which will not be repeated here.

[0208] As shown in Figure 12, the application can determine the target consumption rate and the time interval between sending two adjacent buffers to the BBQ (or surface flinger), allowing the application to send buffers to the BBQ and surface flinger relatively evenly, thereby alleviating the screen freeze when displaying consecutive frames. For example, the application can stabilize the image frames so that frames 1, 2, 3, 4, and 5 are displayed evenly.

[0209] It can be understood that the method for stabilizing image frames provided in the embodiments of the present application can not only be used in the scenario where the camera displays a preview screen as shown in Figure 9 or Figure 11, but can also be applied to the scenario where the received video is stuck due to network delay. This is not limited to the embodiments of the present application.

[0210] The method provided in the embodiment of the present application is described above in conjunction with Figures 4-12. The device for executing the above method provided in the embodiment of the present application is described below. As shown in Figure 13, Figure 13 is a schematic structural diagram of a device for stabilizing image frames provided in the embodiment of the present application. The device for stabilizing image frames can be a terminal device in the embodiment of the present application, or a chip or chip system within the terminal device.

[0211] As shown in FIG13 , an apparatus 1300 for stabilizing image frames can be used in a communication device, circuit, hardware component, or chip. The apparatus 1300 includes a display unit 1301 and a processing unit 1302. The display unit 1301 is configured to support the display step performed by the apparatus 1300 for stabilizing image frames, and the processing unit 1302 is configured to support the information processing step performed by the apparatus 1300 for stabilizing image frames.

[0212] In a possible implementation, the apparatus for stabilizing image frames 1300 may also include a communication unit 1303. Specifically, the communication unit is configured to support the apparatus for stabilizing image frames 1300 in executing the steps of sending and receiving data. The communication unit 1303 may be an input or output interface, pin, or circuit.

[0213] In a possible embodiment, the apparatus for stabilizing image frames may further include a storage unit 1304. The processing unit 1302 and the storage unit 1304 are connected via a circuit. The storage unit 1304 may include one or more memories, which may be devices in one or more devices or circuits used to store programs or data. The storage unit 1304 may exist independently and be connected to the processing unit 1302 of the apparatus for stabilizing image frames via a communication line. The storage unit 1304 may also be integrated with the processing unit 1302.

[0214] The storage unit 1304 can store computer-executable instructions for the method in the terminal device, so that the processing unit 1302 executes the method in the above embodiment. The storage unit 1304 can be a register, a cache, or a RAM, etc. The storage unit 1304 can be integrated with the processing unit 1302. The storage unit 1304 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions. The storage unit 1304 can be independent of the processing unit 1302.

[0215] Figure 14 is a schematic diagram of the hardware structure of another terminal device provided in an embodiment of the present application. As shown in Figure 14, the terminal device includes a processor 1401, a communication line 1404 and at least one communication interface (communication interface 1403 is exemplified in Figure 14).

[0216] The processor 1401 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0217] Communications link 1404 may include circuitry to transmit information between the aforementioned components.

[0218] The communication interface 1403 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0219] Possibly, the terminal device may further include a memory 1402 .

[0220] The memory 1402 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be independent and connected to the processor via a communication line 1404. The memory may also be integrated with the processor.

[0221] The memory 1402 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 1401. The processor 1401 is used to execute the computer-executable instructions stored in the memory 1402, thereby implementing the method provided by the embodiment of the present application.

[0222] Possibly, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, and the embodiments of the present application do not specifically limit this.

[0223] In a specific implementation, as an embodiment, the processor 1401 may include one or more CPUs, such as CPU0 and CPU1 in FIG14 .

[0224] In a specific implementation, as an embodiment, the terminal device may include multiple processors, such as processor 1401 and processor 1405 in Figure 14. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0225] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, available media can include magnetic media (e.g., floppy disk, hard disk or tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid state disk (SSD)), etc.

[0226] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part via software, hardware, firmware, or any combination thereof. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one location to another. The storage medium can be any target medium that can be accessed by a computer.

[0227] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically using lasers.

[0228] The above combinations are also included within the scope of computer-readable media. The above are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for stabilizing an image frame, characterized in that: The method is applied to a terminal device, and the method includes: receiving a first operation from a user; In response to the first operation, starting a camera application, and sequentially rendering and drawing to obtain a first image frame, a second image frame, and a third image frame, wherein the time interval between the second image frame and the first image frame is a first time interval, the time interval between the third image frame and the second image frame is a second time interval, and the first time interval and the second time interval are different in size; synthesizing the first image frame based on a first frame rate to obtain a first synthesized image frame, synthesizing the second image frame to obtain a second synthesized image frame, and synthesizing the third image frame to obtain a third synthesized image frame; The first composite image frame is displayed within a first period, the second composite image frame is displayed within a second period, and the third composite image frame is displayed within a third period, the time interval between the first period and the second period is a third time interval, the time interval between the second period and the third period is a fourth time interval, and the third time interval is the same as the fourth time interval.

2. The method according to claim 1, characterized in that The synthesizing the first image frame based on the first frame rate to obtain a first synthesized image frame, synthesizing the second image frame to obtain a second synthesized image frame, and synthesizing the third image frame to obtain a third synthesized image frame includes: When it is determined within a fourth period that the first moment is greater than or equal to a target time interval determined based on the first frame rate, synthesizing the first image frame within the fourth period to obtain the first synthesized image frame; When a fifth time interval between the second moment determined in a fifth cycle and the time of synthesizing the first image frame is greater than or equal to the target time interval, synthesizing the second image frame in the fifth cycle to obtain the second synthesized image frame; the fifth cycle is after the fourth cycle; When the sixth time interval between the third moment determined in the sixth cycle and the time of synthesizing the second image frame is greater than or equal to the target time interval, the third image frame is synthesized in the sixth cycle to obtain the third synthesized image frame; the sixth cycle is after the fifth cycle.

3. The method according to claim 2, characterized in that The method further comprises: When it is determined within the fifth cycle that the fifth time interval is less than the target time interval, not synthesizing the second image frame within the fifth cycle; When the fifth time interval between the second moment determined in the fifth cycle and the time of synthesizing the first image frame is greater than or equal to the target time interval, the second image frame is synthesized in the fifth cycle to obtain the second synthesized image frame, including: when the seventh time interval between the fourth moment determined in the seventh cycle and the time of synthesizing the first image frame is greater than or equal to the target time interval, the second image frame is synthesized in the seventh cycle to obtain the second synthesized image frame; the seventh cycle is after the fifth cycle and before the sixth cycle.

4. The method according to any one of claims 1 to 3, characterized in that The first image frame, the second image frame, and the third image frame are all image frames of a preview layer.

5. The method according to any one of claims 1 to 4, characterized in that The terminal device includes: a camera module and a camera hardware abstraction layer service. Before sequentially rendering to obtain a first image frame, a second image frame, and a third image frame, the method further includes: The camera module generates a first image sequence based on the first frame rate and sends the first image sequence to the camera hardware abstraction layer service; wherein the first image sequence includes: a first original image frame, a second original image frame, and a third original image frame; the time interval between the first original image frame and the second original image frame is an eighth time interval, the time interval between the second original image frame and the third original image frame is a ninth time interval, and the eighth time interval is the same as the ninth time interval; The camera hardware abstraction layer service performs image processing on the first original image frame to obtain a first process image frame, performs image processing on the second original image frame to obtain a second process image frame, and performs image processing on the third original image frame to obtain a third process image frame.

6. The method according to claim 5, characterized in that The terminal device includes: a camera service and a layer view, and the method further includes: When the camera service receives the first process image frame, the second process image frame, and the third process image frame sent by the camera hardware abstraction layer service, the camera service sets a first identifier for the first process image frame, the second process image frame, and the third process image frame, respectively; wherein the first identifier indicates the first frame rate; The camera service sends the first process image frame with the first identifier, the second process image frame with the first identifier, and the third process image frame with the first identifier to the layer view; The sequential drawing and rendering to obtain the first image frame, the second image frame, and the third image frame includes: the layer view sequentially drawing and rendering the first process image frame to obtain the first image frame, drawing and rendering the second process image frame to obtain the second image frame, and drawing and rendering the third process image frame to obtain the third image frame.

7. The method according to claim 6, characterized in that The terminal device includes: a display synthesis system, wherein the first image frame is synthesized based on the first frame rate to obtain a first synthesized image frame, the second image frame is synthesized to obtain a second synthesized image frame, and the third image frame is synthesized to obtain a third synthesized image frame, including: When the display synthesis system receives the first identifier sent by the layer view, the display synthesis system synthesizes the first image frame to obtain the first synthesized image frame, synthesizes the second image frame to obtain the second synthesized image frame, and synthesizes the third image frame to obtain the third synthesized image frame based on the first frame rate indicated in the first identifier.

8. The method according to claim 6 or 7, characterized in that The first identifier is obtained based on the SetConsumeRate interface, and the SetConsumeRate interface is used to instruct the layer view to pass the first identifier to the display composition system.

9. The method according to any one of claims 1 to 7, characterized in that The first frame rate is less than 60 Hz.

10. A method for stabilizing an image frame, characterized in that: The method is applied to a terminal device, the terminal device including: a layer view and a display synthesis system. The method includes: receiving a first operation from a user; In response to the first operation, the camera application is started, and the layer view is sequentially drawn and rendered to obtain a first image frame, a second image frame, and a third image frame; wherein the time interval between the second image frame and the first image frame is a first time interval, and the time interval between the third image frame and the second image frame is a second time interval. interval, the first time interval and the second time interval are different in size; The layer view sends the first image frame to the display composition system based on the first frame rate within an eighth period, sends the second image frame to the display composition system based on the first frame rate within a ninth period, and sends the third image frame to the display composition system based on the first frame rate within a tenth period; the time interval between the eighth period and the ninth period is the tenth time interval, the time interval between the ninth period and the tenth period is the eleventh time interval, and the tenth time interval is the same as the eleventh time interval; The display synthesis system synthesizes the first image frame to obtain a first synthesized image frame, synthesizes the second image frame to obtain a second synthesized image frame, and synthesizes the third image frame to obtain a third synthesized image frame.

11. The method according to claim 10, characterized in that The method includes sending the first image frame to the display composition system based on the first frame rate within an eighth cycle, sending the second image frame to the display composition system based on the first frame rate within a ninth cycle, and sending the third image frame to the display composition system based on the first frame rate within a tenth cycle, including: When it is determined within the eighth cycle that the fifth moment is greater than or equal to the target time interval determined based on the first frame rate, sending the first image frame to the display composition system within the eighth cycle; When a twelfth time interval between a sixth moment determined in the ninth cycle and the time when the first image frame is sent to the display composition system is greater than or equal to the target time interval, sending the second image frame to the display composition system in the ninth cycle; When the thirteenth time interval between the seventh moment determined in the tenth cycle and the time when the second image frame is sent to the display synthesis system is greater than or equal to the target time interval, the third image frame is sent to the display synthesis system in the tenth cycle.

12. The method according to claim 11, characterized in that The method further comprises: When the twelfth time interval in the ninth cycle is less than the target time interval, not sending the second image frame to the display composition system in the ninth cycle; The step of sending the second image frame to the display synthesis system within the ninth cycle when the twelfth time interval between the sixth moment determined within the ninth cycle and the time when the first image frame is sent to the display synthesis system is greater than or equal to the target time interval includes: sending the second image frame to the display synthesis system within the eleventh cycle when the fourteenth time interval between the eighth moment determined within the eleventh cycle and the time when the first image frame is sent to the display synthesis system is greater than or equal to the target time interval; wherein the eleventh cycle is after the ninth cycle and before the tenth cycle.

13. A terminal device, characterized in that: The terminal device includes a processor, and the processor is used to call a computer program in a memory to execute the method according to any one of claims 1 to 9, or execute the method according to any one of claims 10 to 12.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 9, or executes the method according to any one of claims 10 to 12.

15. A chip, characterized in that: The chip includes a processor, and the processor is used to call a computer program in a memory to execute the method according to any one of claims 1 to 9, or to execute the method according to any one of claims 10 to 12.

Images