Image processing method and apparatus

Abstract

The embodiment of the present application provides a kind of method and device for stabilizing image frame, receive the first operation of user;In response to the first operation, open camera application, and render in turn to draw to obtain first image frame, second image frame and third image frame;First frame rate is based on respectively to first image frame is synthesized to obtain first synthesis image frame, to second image frame is synthesized to obtain second synthesis image frame, and to third image frame is synthesized to obtain third synthesis image frame;Terminal equipment displays first synthesis image frame in first period, displays second synthesis image frame in second period, and displays third synthesis image frame in third period;The time interval between first period and second period is same with the time interval between second period and third period.Such, can reduce the picture jam condition due to image frame inhomogeneous before layer synthesis in layer synthesis stage, so that the picture displayed can be in uniform state in period.

Description

[0001] This application is a divisional application; the original application number is 202211541688.5, and the original filing date is 2022.

[0002] The entire contents of the original application are incorporated herein by reference on December 2nd. Technical Field

[0003] This application relates to the field of terminal technology, and in particular to a method and apparatus for stabilizing image frames. Background Technology

[0004] With the popularization and development of the Internet, many terminal devices can support photo and video recording functions, and during the photo and video recording process, they can display a preview screen, allowing users to determine whether to take a photo or record a video based on the screen displayed in the preview interface.

[0005] Normally, when a terminal device receives a user's request to open the camera preview, it can capture an image sequence from the camera and obtain a preview by processing and displaying each image frame in the sequence. However, due to the differences between the image frames, the frame output times of the processed image frames are uneven, causing stuttering in the camera preview. Summary of the Invention

[0006] This application provides a method and apparatus for stabilizing image frames, enabling terminal devices to ensure the smoothness of the preview screen when performing layer compositing on image frames.

[0007] In a first aspect, embodiments of this application provide a method for stabilizing image frames. The method is applied to a terminal device and includes: receiving a first operation from a user; receiving the first operation from the user; responding to the first operation, launching a camera application and sequentially rendering a first image frame, a second image frame, and a third image frame, wherein the time interval between the second image frame and the first image frame is a first time interval, and the time interval between the third image frame and the second image frame is a second time interval, the first time interval and the second time interval being of different sizes; synthesizing the first image frame based on a first frame rate to obtain a first synthesized image frame, synthesizing the second image frame to obtain a second synthesized image frame, and synthesizing the third image frame to obtain a third synthesized image frame; displaying the first synthesized image frame within a first period, displaying the second synthesized image frame within a second period, and displaying the third synthesized image frame within a third period, the time interval between the first period and the second period being a third time interval, and the time interval between the second period and the third period being a fourth time interval, the third time interval and the fourth time interval being of the same size. This reduces screen stuttering caused by uneven image frames before layer synthesis during the layer synthesis stage, ensuring that the displayed image remains uniform within the period.

[0008] In one possible implementation, a first composite image frame is obtained by synthesizing a first image frame based on a first frame rate, a second composite image frame is obtained by synthesizing a second image frame, and a third composite image frame is obtained by synthesizing a third image frame. This includes: within a fourth period, when a first moment is determined to be greater than or equal to a target time interval determined based on the first frame rate, the first image frame is synthesized within the fourth period to obtain the first composite image frame; within a fifth period, when a second moment and a fifth time interval between the time of synthesizing the first image frame are determined to be greater than or equal to the target time interval, the second image frame is synthesized within the fifth period to obtain the second composite image frame; the fifth period is after the fourth period; within a sixth period, when a third moment and a sixth time interval between the time of synthesizing the second image frame are determined to be greater than or equal to the target time interval, the third image frame is synthesized within the sixth period to obtain the third composite image frame; the sixth period is after the fifth period. This allows the terminal device to ensure uniform display of adjacent image frames during the image synthesis processing stage, based on the first frame rate and the time interval between adjacent frame synthesis.

[0009] In one possible implementation, the method further includes: if the fifth time interval is determined to be less than the target time interval within the fifth period, then the second image frame is not synthesized within the fifth period; if the fifth time interval between the second moment and the time of synthesizing the first image frame is determined to be greater than or equal to the target time interval within the fifth period, then the second image frame is synthesized within the fifth period to obtain the second synthesized image frame, including: if the seventh time interval between the fourth moment and the time of synthesizing the first image frame is determined to be greater than or equal to the target time interval within the seventh period, then the second image frame is synthesized within the seventh period to obtain the second synthesized image frame; the seventh period is after the fifth period and before the sixth period. This allows the terminal device to avoid synthesizing image frames within a certain period if layer synthesis of image frames within that period may lead to uneven image display, thus ensuring the stability of subsequent image display.

[0010] In one possible implementation, the first, second, and third image frames are all image frames from the preview layer. This ensures that image frames from the same layer can be processed at the same frame rate.

[0011] In one possible implementation, the terminal device includes a camera module and a camera hardware abstraction layer service. Before sequentially rendering and drawing a first image frame, a second image frame, and a third image frame, the method further includes: the camera module generating a first image sequence based on a first frame rate and sending the first image sequence to the camera hardware abstraction layer service; wherein, the first image sequence includes a first original image frame, a second original image frame, and a third original image frame; the time interval between the first original image frame and the second original image frame is an eighth time interval, and the time interval between the second original image frame and the third original image frame is a ninth time interval, the eighth time interval and the ninth time interval being the same size; the camera hardware abstraction layer service performs image processing on the first original image frame to obtain a first process image frame, performs image processing on the second original image frame to obtain a second process image frame, and performs image processing on the third original image frame to obtain a third process image frame. In this way, even if the camera provider's frame output time is uneven, the surface flinger can ensure sufficient layer compositing time when compositing layers on the buffer, and perform layer compositing at a stable consumption rate to reduce screen stuttering.

[0012] In one possible implementation, the terminal device includes a camera service and a layer view. The method further includes: when the camera service receives a first process image frame, a second process image frame, and a third process image frame sent by the camera hardware abstraction layer service, the camera service sets a first identifier for each of the first, second, and third process image frames; wherein the first identifier indicates a first frame rate; the camera service sends the first process image frame with the first identifier, the second process image frame with the first identifier, and the third process image frame with the first identifier to the layer view; and sequentially drawing and rendering the first image frame, the second image frame, and the third image frame includes: the layer view sequentially drawing and rendering the first process image frame to obtain the first image frame, drawing and rendering the second process image frame to obtain the second image frame, and drawing and rendering the third process image frame to obtain the third image frame.

[0013] In one possible implementation, the terminal device includes a display compositing system that composites a first image frame with a first frame rate to obtain a first composite image frame, composites a second image frame with a second frame to obtain a second composite image frame, and composites a third image frame with a third composite image frame. The system further includes: upon receiving a first identifier sent by a layer view, the display compositing system, based on the first frame rate indicated in the first identifier, composites the first image frame with the first frame rate to obtain the first composite image frame, the second image frame with the second frame to obtain the second composite image frame, and the third image frame with the third composite image frame to obtain the third composite image frame. This allows the display compositing system to determine the composite frame rate based on the identifier, thereby achieving uniform layer compositing.

[0014] In one possible implementation, the first identifier is obtained based on the SetConsumeRate interface, which is used to instruct the layer view to pass the first identifier to the display composition system.

[0015] In one possible implementation, the first frame rate is less than 60Hz, allowing the terminal device to maintain a relatively stable image at frame rates below 60Hz.

[0016] Secondly, embodiments of this application provide a method for stabilizing image frames. The method is applied to a terminal device, which includes a layer view and a display compositing system. The method includes: receiving a first operation from a user; responding to the first operation, launching a camera application, and the layer view sequentially drawing and rendering a first image frame, a second image frame, and a third image frame; wherein the time interval between the second image frame and the first image frame is a first time interval, and the time interval between the third image frame and the second image frame is a second time interval, and the first time interval and the second time interval are of different sizes; the layer view sends the first image frame to the display compositing system within an eighth cycle based on a first frame rate, sends the second image frame to the display compositing system within a ninth cycle based on the first frame rate, and sends the third image frame to the display compositing system within a tenth cycle based on the first frame rate; the time interval between the eighth cycle and the ninth cycle is a tenth time interval, and the time interval between the ninth cycle and the tenth cycle is an eleventh time interval, and the tenth time interval and the eleventh time interval are of the same size; the display compositing system composites the first image frame to obtain a first composite image frame, composites the second image frame to obtain a second composite image frame, and composites the third image frame to obtain a third composite image frame. This allows the layer view to stably send image frames to the display compositing system based on the first frame rate, thereby ensuring the stability of the display compositing system's compositing and subsequent screen display.

[0017] In one possible implementation, based on a first frame rate, a first image frame is sent to the display compositing system within an eighth cycle, a second image frame is sent to the display compositing system within a ninth cycle, and a third image frame is sent to the display compositing system within a tenth cycle. This includes: within the eighth cycle, when a fifth time point is determined to be greater than or equal to a target time interval determined based on the first frame rate, sending the first image frame to the display compositing system within the eighth cycle; within the ninth cycle, when a twelfth time interval between a sixth time point and the time when the first image frame is sent to the display compositing system is greater than or equal to the target time interval, sending the second image frame to the display compositing system within the ninth cycle; and within the tenth cycle, when a thirteenth time interval between a seventh time point and the time when the second image frame is sent to the display compositing system is greater than or equal to the target time interval, sending the third image frame to the display compositing system within the tenth cycle. In this way, the layer view can send the buffer evenly to the surface flinger based on the first frame rate and the time intervals between sending the buffer to the BBQ from two adjacent buffers, thereby alleviating screen stuttering when displaying consecutive frames.

[0018] In one possible implementation, the method further includes: if the twelfth time interval within the ninth cycle is less than the target time interval, not sending the second image frame to the display compositing system within the ninth cycle; if, within the ninth cycle, the twelfth time interval between the sixth moment and the time when the first image frame is sent to the display compositing system is greater than or equal to the target time interval, sending the second image frame to the display compositing system within the ninth cycle includes: if, within the eleventh cycle, the fourteenth time interval between the eighth moment and the time when the first image frame is sent to the display compositing system is greater than or equal to the target time interval, sending the second image frame to the display compositing system within the eleventh cycle; wherein the eleventh cycle is after the ninth cycle and before the tenth cycle. This allows the layer view to avoid compositing image frames within a certain cycle if sending image frames within that cycle may lead to uneven image display, thus ensuring the stability of subsequent image display.

[0019] In one possible implementation, the first, second, and third image frames are all image frames from the preview layer. This ensures that image frames from the same layer can be processed at the same frame rate.

[0020] In one possible implementation, the first frame rate is less than 60Hz, allowing the terminal device to maintain a relatively stable image at frame rates below 60Hz.

[0021] Thirdly, embodiments of this application provide an apparatus for stabilizing image frames. The method is applied to a terminal device. A processing unit is used to receive a first operation from a user. The processing unit is further used to respond to the first operation by launching a camera application and sequentially drawing and rendering a first image frame, a second image frame, and a third image frame. The time interval between the second image frame and the first image frame is a first time interval, and the time interval between the third image frame and the second image frame is a second time interval. The first time interval and the second time interval are of different sizes. The processing unit is further used to synthesize the first image frame based on a first frame rate to obtain a first synthesized image frame, synthesize the second image frame to obtain a second synthesized image frame, and synthesize the third image frame to obtain a third synthesized image frame. A display unit is further used to display the first synthesized image frame in a first period, the second synthesized image frame in a second period, and the third synthesized image frame in a third period. The time interval between the first period and the second period is a third time interval, and the time interval between the second period and the third period is a fourth time interval. The third time interval and the fourth time interval are of the same size.

[0022] In one possible implementation, the processing unit is further configured to: when it is determined within the fourth period that the first moment is greater than or equal to the target time interval determined based on the first frame rate, synthesize the first image frame within the fourth period to obtain a first synthesized image frame; when it is determined within the fifth period that the second moment and the fifth time interval between the time of synthesizing the first image frame are greater than or equal to the target time interval, synthesize the second image frame within the fifth period to obtain a second synthesized image frame; the fifth period is after the fourth period; when it is determined within the sixth period that the third moment and the sixth time interval between the time of synthesizing the second image frame are greater than or equal to the target time interval, synthesize the third image frame within the sixth period to obtain a third synthesized image frame; the sixth period is after the fifth period.

[0023] In one possible implementation, the processing unit is further configured to: when it is determined within the fifth period that the fifth time interval is less than the target time interval, not to synthesize the second image frame within the fifth period; and when it is determined within the fifth period that the fifth time interval between the second moment and the time of synthesizing the first image frame is greater than or equal to the target time interval, to synthesize the second image frame within the fifth period to obtain the second synthesized image frame, including: when it is determined within the seventh period that the seventh time interval between the fourth moment and the time of synthesizing the first image frame is greater than or equal to the target time interval, to synthesize the second image frame within the seventh period to obtain the second synthesized image frame; the seventh period is after the fifth period and before the sixth period.

[0024] In one possible implementation, the first image frame, the second image frame, and the third image frame are all image frames of the preview layer.

[0025] In one possible implementation, the terminal device includes a camera module and a camera hardware abstraction layer service. Before sequentially drawing and rendering a first image frame, a second image frame, and a third image frame, the method further includes a processing unit, which is used to: generate a first image sequence based on a first frame rate and send the first image sequence to the camera hardware abstraction layer service; wherein the first image sequence includes a first original image frame, a second original image frame, and a third original image frame; the time interval between the first original image frame and the second original image frame is an eighth time interval, and the time interval between the second original image frame and the third original image frame is a ninth time interval, the eighth time interval and the ninth time interval are of the same size; image processing is performed on the first original image frame to obtain a first process image frame, image processing is performed on the second original image frame to obtain a second process image frame, and image processing is performed on the third original image frame to obtain a third process image frame.

[0026] In one possible implementation, the terminal device includes a camera service and a layer view. The method further includes a processing unit, which is configured to: upon receiving a first process image frame, a second process image frame, and a third process image frame sent by the camera hardware abstraction layer service, set a first identifier for each of the first process image frame, the second process image frame, and the third process image frame; wherein the first identifier indicates a first frame rate; the camera service sends the first process image frame with the first identifier, the second process image frame with the first identifier, and the third process image frame with the first identifier to the layer view; and sequentially draw and render the first process image frame to obtain a first image frame, draw and render the second process image frame to obtain a second image frame, and draw and render the third process image frame to obtain a third image frame.

[0027] In one possible implementation, the terminal device includes: a display compositing system and a processing unit, further configured to: upon receiving a first identifier sent by a layer view, based on a first frame rate indicated in the first identifier, respectively composite a first image frame to obtain a first composite image frame, composite a second image frame to obtain a second composite image frame, and composite a third image frame to obtain a third composite image frame.

[0028] In one possible implementation, the first identifier is obtained based on the SetConsumeRate interface, which is used to instruct the layer view to pass the first identifier to the display composition system.

[0029] In one possible implementation, the initial frame rate is less than 60Hz. This allows the terminal device to maintain a relatively stable image at frame rates below 60Hz.

[0030] Fourthly, embodiments of this application provide an apparatus for stabilizing image frames. The method is applied to a terminal device, which includes a layer view and a display compositing system. The method includes: a processing unit for receiving a first operation from a user; and, in response to the first operation, launching a camera application. The processing unit is further configured to sequentially draw and render a first image frame, a second image frame, and a third image frame. The time interval between the second image frame and the first image frame is a first time interval, and the time interval between the third image frame and the second image frame is a second time interval. The first time interval and the second time interval are of different sizes. The processing unit is further configured to send the first image frame to the display compositing system within an eighth cycle based on a first frame rate, send the second image frame to the display compositing system within a ninth cycle based on the first frame rate, and send the third image frame to the display compositing system within a tenth cycle based on the first frame rate. The time interval between the eighth and ninth cycles is a tenth time interval, and the time interval between the ninth and tenth cycles is an eleventh time interval. The tenth and eleventh time intervals are of the same size. The processing unit is further configured to composite the first image frame to obtain a first composite image frame, composite the second image frame to obtain a second composite image frame, and composite the third image frame to obtain a third composite image frame.

[0031] In one possible implementation, the processing unit is further configured to: send a first image frame to the display compositing system within the eighth cycle when it is determined that the fifth time point is greater than or equal to the target time interval determined based on the first frame rate; send a second image frame to the display compositing system within the ninth cycle when it is determined that the twelfth time interval between the sixth time point and the time when the first image frame is sent to the display compositing system is greater than or equal to the target time interval; and send a third image frame to the display compositing system within the tenth cycle when it is determined that the thirteenth time interval between the seventh time point and the time when the second image frame is sent to the display compositing system is greater than or equal to the target time interval.

[0032] In one possible implementation, the processing unit is further configured to: not send the second image frame to the display compositing system within the ninth cycle when the twelfth time interval within the ninth cycle is less than the target time interval; and to send the second image frame to the display compositing system within the ninth cycle when the twelfth time interval between the sixth moment and the time when the first image frame is sent to the display compositing system is greater than or equal to the target time interval, including: sending the second image frame to the display compositing system within the eleventh cycle when the fourteenth time interval between the eighth moment and the time when the first image frame is sent to the display compositing system is greater than or equal to the target time interval; wherein the eleventh cycle is after the ninth cycle and before the tenth cycle.

[0033] In one possible implementation, the first image frame, the second image frame, and the third image frame are all image frames of the preview layer.

[0034] In one possible implementation, the first frame rate is less than 60Hz.

[0035] Fifthly, embodiments of this application provide a terminal device, including a processor and a memory, wherein the memory is used to store code instructions; the processor is used to run the code instructions, causing the terminal device to perform the method described in the first aspect or any implementation thereof, or to perform the method described in the second aspect or any implementation thereof.

[0036] Sixthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed, cause a computer to perform the method described in the first aspect or any implementation thereof, or to perform the method described in the second aspect or any implementation thereof.

[0037] A seventh aspect is a computer program product comprising a computer program that, when run, causes a computer to perform the method described in the first aspect or any implementation thereof, or to perform the method described in the second aspect or any implementation thereof.

[0038] Eighth aspect, a chip including a processor for calling a computer program in memory, such that the computer program performs a method as described in the first aspect or any implementation thereof, or executes a method as described in the second aspect or any implementation thereof.

[0039] It should be understood that the second to seventh aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0040] Figure 1 A scenario diagram provided for an embodiment of this application;

[0041] Figure 2 This application provides a schematic diagram of the software architecture of a terminal device.

[0042] Figure 3 This is a schematic diagram of the software architecture of another terminal device provided in an embodiment of this application;

[0043] Figure 4 A schematic diagram of a consumption buffer provided in an embodiment of this application;

[0044] Figure 5 A schematic diagram illustrating a frame return time provided in an embodiment of this application;

[0045] Figure 6 A schematic diagram of another consumption buffer provided in an embodiment of this application;

[0046] Figure 7 A schematic diagram of another consumption buffer provided in an embodiment of this application;

[0047] Figure 8 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application;

[0048] Figure 9 A method for stabilizing image frames is provided in the embodiments of this application;

[0049] Figure 10 A schematic diagram illustrating another type of consumption buffer provided in an embodiment of this application;

[0050] Figure 11 Another method for stabilizing image frames provided in this application embodiment;

[0051] Figure 12 A schematic diagram illustrating another type of consumption buffer provided in an embodiment of this application;

[0052] Figure 13 A schematic diagram of the structure of an image frame stabilization device provided in an embodiment of this application;

[0053] Figure 14 This is a schematic diagram of the hardware structure of another terminal device provided in an embodiment of this application. Detailed Implementation

[0054] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. For example, the first value and the second value are only used to distinguish different values ​​and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0055] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0056] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0057] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application. Figure 1 In the corresponding embodiments, a mobile phone is used as an example for illustration. This example does not constitute a limitation on the embodiments of this application.

[0058] When a terminal device receives a user's command to open the camera application, it can capture an image sequence using the camera and process and display each image frame in the sequence, enabling the terminal device to display images such as... Figure 1 The interface shown in 'a' is a preview screen for the camera function. This preview screen may include target 101, which may be positioned in the center of the preview screen. Figure 1 The interface shown in 'a' can also include controls for starting the camera, controls for opening the photo album, controls for switching cameras, and other functional controls corresponding to the camera application.

[0059] However, due to the differences between image frames, such as inconsistencies in displayed content and / or lighting effects, the frame output times of the processed image frames are uneven, causing stuttering in the camera preview. For example, when target 101 moves to the lower left, its position in the preview should also move to the lower left. However, due to the uneven frame output times, Figure 1 In the interface shown in b, target 101 should be at the position of the dotted line, but due to screen lag, target 101 is still in the middle of the preview screen. The content displayed in the preview screen is inconsistent with the actual target, which brings a poor user experience.

[0060] Specifically, the process of displaying the preview screen on the terminal device can be found in [link to documentation]. Figure 3 The description in the corresponding embodiments. Figure 2This is a schematic diagram of the software architecture of a terminal device provided in an embodiment of this application.

[0061] like Figure 2 As shown, the layered architecture of a terminal device divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into multiple layers, from top to bottom: the application (APP) layer, the application framework layer, the system library, the hardware abstraction layer (HAL), and the kernel layer.

[0062] The application layer provides the applications required by the terminal device. The application layer may include: camera, desktop, sharing, Bluetooth, voice interaction applications, and contacts, etc. The camera can be a program pre-installed on the terminal device's system or an application provided by a third party.

[0063] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer includes a set of predefined functions.

[0064] The application framework layer may also include: a camera service, a surface flinger (SF), views, an input manager, and a buffer queue. Views include visual controls, such as controls for displaying text and controls for displaying images. Views are used to build the application. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0065] Among them, the camera service, or camera framework, is used to create the buffer queue; the surface flinger is used to provide layer compositing services, and is responsible for processing each layer separately, such as layer rendering, layer classification, and layer compositing.

[0066] The system library may include: browser kernel, 3D graphics, and font libraries, etc. The terminal device may also include the Android runtime.

[0067] HAL is used to abstract hardware, providing an interface for upper-layer applications to query hardware devices, or to provide data storage services for upper-layer applications.

[0068] The HAL can include: camera hardware abstraction layer service (camera HAL), display screen, sensor, and hardware composer (HWC).

[0069] Among them, the camera HAL, or camera provider, is used to perform image processing on the input raw image frames, such as image cropping, image effects processing, image stabilization, and image recognition, and inputs the processed image frames into the camera service; HWC is used to overlay the layer data prepared by the surface flinger and forward it to the display driver.

[0070] The kernel layer can include: camera module (camera sensor), display driver, Bluetooth driver, ultra-wide band (UWB) driver, sensor driver, touch screen driver, and audio driver, etc.

[0071] The camera sensor is used to acquire raw image frames according to instructions. The raw image sequence may include multiple raw image frames, and the format of any one of the multiple raw image frames may be raw (RAW) format. The display driver is used to display the image frames input by HWC.

[0072] It is understood that the software architecture of a terminal device may include other elements, and this application embodiment does not limit this.

[0073] based on Figure 2 In a corresponding embodiment, the process of displaying the preview screen by the terminal device may involve: camera, buffer queue, camera service, camera HAL, surface flinger, HWC, display driver, and camera sensor, etc.

[0074] For example, Figure 3 This is a schematic diagram of the software architecture of another terminal device provided in an embodiment of this application.

[0075] The camera can include a surface view, which can be used to create a BLASTbufferqueue (or BBQ for short). The camera (or the surface view within the camera) can be used to draw and render received image frames.

[0076] When the terminal device receives a user's operation such as opening the camera application, the surface view in the camera application can create a buffer queue and BBQ, and the surface view can also instruct the camera service to capture image frames.

[0077] During the creation of the bufferqueue, the bufferqueue can establish a connection between the consumer surface view and the producer camera service. The consumer will register an onFrameAvailable listener with the bufferqueue. When the producer calls the queueBuffer function to send the buffer to the bufferqueue, the consumer will receive a callback message (or callback function).

[0078] During the creation of BBQ, BBQ can establish a connection between the producer surface view and the consumer surface flinger, and BBQ can also act as a consumer of the buffer queue.

[0079] During the process of the surface view instructing the camera service to acquire image frames, the surface view can send instructions to the camera service for acquiring image frames (or instructions to create a preview stream for the user). The camera service generates a set of register configurations (or output parameters) for the camera sensor, used to configure the camera sensor's output mode, image size, and output frame rate parameters, etc. The camera service then sends the instructions for acquiring image frames and the output parameters to the camera HAL. The camera HAL then sends the instructions for acquiring image frames and the output parameters to the camera sensor. Accordingly, upon receiving the instructions for acquiring image frames and the output parameters, the camera sensor can acquire a raw image sequence according to the output parameters and send the raw image sequence to the camera HAL. The camera HAL performs image processing on each raw image frame in the raw image sequence and returns the processed image frames to the camera service.

[0080] Furthermore, the camera service caches the processed image frames (or buffers) returned by the camera HAL into a buffer queue; the buffer queue sends the buffer to the surface view and notifies the surface view to consume it; the surface view calls the acquireBuffer function to consume the buffer and calls the queueBuffer function to send the buffer to the BBQ; the BBQ sends the buffer to the surface flinger through the transaction object; the surface flinger can perform layer compositing on the received buffer according to the consumption rate received by the surface flinger, such as when the surface flinger receives a vertical synchronization (Vsync) event to perform layer compositing on the buffer.

[0081] The Vsync signal is a periodic signal, and its period can be set according to the screen refresh rate. For example, when the screen refresh rate is 60Hz, the Vsync signal period can be 16.6ms, meaning that the terminal device generates a control signal every 16.6ms to trigger the Vsync signal period.

[0082] It should be noted that Vsync signals can be divided into software Vsync signals and hardware Vsync signals. Software Vsync signals include Vsync-APP and Vsync-SF. Vsync-APP is used to trigger the application's drawing and rendering process of image frames. Vsync-SF is used to trigger the surface flinger's layer compositing process of image frames. Hardware Vsync signals are used to trigger the screen display refresh process.

[0083] Normally, the software Vsync signal and the hardware Vsync signal maintain periodic synchronization. Taking a change between 60Hz and 120Hz as an example, if the hardware Vsync signal switches from 60Hz to 120Hz, Vsync-APP and Vsync-SF change synchronously from 60Hz to 120Hz.

[0084] For example, when the surface flinger receives a buffer, it can request the Vsync signal; when the surface flinger determines that there are other buffers that need to be consumed, it can also request the Vsync signal.

[0085] For example, the surface flinger can initiate the layer composition process for the buffer upon receiving any Vsync signal. Upon receiving any Vsync signal, the surface flinger can determine whether a buffer has been received; if a buffer is present, it performs layer composition; otherwise, the Vsync signal is not involved in image frame composition. In a possible implementation, the surface flinger can also release the buffer after processing it, allowing the buffer to be released into the buffer queue by the surface flinger and BBQ.

[0086] This application provides a schematic diagram of a terminal device normally consuming a buffer. Figure 4 This is a schematic diagram of a consumption buffer provided in an embodiment of this application.

[0087] like Figure 4 As shown, in chronological order, the content displayed by the terminal device can be: frame 1, frame 2, frame 3, frame 4, and frame 5.

[0088] Taking the display of frame 1 as an example, the camera sensor of the terminal device acquires frame 1 and, based on... Figure 2 The corresponding embodiment describes sending frame 1 along the camera sensor, camera HAL, and camera service to the application, where the camera application renders frame 1. After rendering, the terminal device's application sends the rendered frame 1 to the surface flinger via BBQ. The surface flinger composites the rendered frame 1. After framing is complete, the terminal device can start the display driver by calling the kernel layer to display the content corresponding to frame 1 on the screen. Similarly, the framing and display process for frames 2, 3, 4, and 5 can be referred to in frame 1 and will not be repeated here. Figure 3 Each frame, from drawing to display, lags by 2 Vsync signal cycles.

[0089] Understandably, if frames 1, 2, 3, 4, and 5 are evenly sent from the camera sensor to the application and then evenly transmitted from the application to the surface flinger, the terminal device can smoothly display frames 1, 2, 3, 4, and 5 on the screen.

[0090] However, due to the differences between the original image frames, the time taken by the camera HAL to process each original image frame varies, resulting in uneven frame return times from the camera HAL to the camera service. This can cause stuttering when the terminal device displays image frames with uneven return times. The frame return time can be understood as the time it takes for the camera HAL to process the image frame and then send it back to the camera service.

[0091] For example, Figure 5 This is a schematic diagram illustrating a frame return time interval provided in an embodiment of this application. For example... Figure 5 The coordinate axes shown represent the sequence number of each image frame on the horizontal axis and the time interval between the current frame's return time and the previous frame's return time on the vertical axis. It can be seen that the time interval between image frame number 2 and image frame number 1 is approximately 18 milliseconds (ms); the time interval between image frame number 4 and image frame number 3 is approximately 63 ms; and the time interval between image frame number 7 and image frame number 6 is approximately 10 ms.

[0092] It is understandable that the camera HAL takes different amounts of time to process each original image frame, resulting in uneven frame return intervals between adjacent image frames in the image sequence emitted by the camera HAL. Figure 5 The minimum frame return interval shown can be around 10ms, and the maximum can be around 60ms.

[0093] Typically, terminal devices can provide two specific scenarios where the preview image stutters due to uneven camera HAL frame return times.

[0094] In one implementation, the camera HAL can mark a first expected display timestamp in the buffer. SurfaceFlinger also generates a second expected display timestamp for the corresponding buffer, and can compare the first expected display timestamp with the second expected display timestamp. If the first expected display timestamp is earlier than the second expected display timestamp, SurfaceFlinger will retrieve all buffers, discard older buffers from the buffer queue, and only retain the latest buffer that meets the requirements for consumption. In this case, when the camera HAL returns frames quickly, some buffers will be discarded, causing stuttering when the terminal device displays the preview image.

[0095] For example, Figure 6 This is a schematic diagram of another consumption buffer provided in an embodiment of this application. For example... Figure 6 As shown, due to the uneven frame return time of the camera HAL, the application can receive frames 1, 2, 3 and 4 within the period of Vsync1. The first expected display time stamp of frame 1 is timestamp 1, the first expected display time stamp of frame 2 is timestamp 2, the first expected display time stamp of frame 3 is timestamp 3, and the first expected display time stamp of frame 4 is timestamp 4.

[0096] Furthermore, the application can set frames 1, 2, 3, and 4 in the BBQ. After the surface flinger receives frames 1, 2, 3, and 4, it can set a second expected display timestamp for each frame. If the surface flinger determines that timestamp 1 of frame 1, timestamp 2 of frame 2, and timestamp 3 of frame 3 are all earlier than the second expected display timestamp, and timestamp 4 of frame 4 is later than the second expected display timestamp, the surface flinger can retain the latest buffer, such as consuming frame 3, and displaying frame 3 within the Vsync3 cycle. In this scenario, discarding frames 1, 2, and 3 will cause stuttering in the preview display.

[0097] In another implementation, the surface flinger consumes one buffer at a time during the layer compositing process, storing the remaining buffers in a buffer queue. When processing any buffer, the surface flinger only knows the timestamp of the camera HAL frame return, but not the actual time when the camera sensor outputs the image. This means that even if the camera sensor outputs the image at a certain frame rate, the actual time when the surface flinger performs layer compositing on the buffer and outputs the image is delayed according to the timestamp of the camera HAL frame return. Therefore, the uneven timestamp of the camera HAL frame return can cause stuttering in the preview.

[0098] For example, Figure 7 This is a schematic diagram of another consumption buffer provided in an embodiment of this application. For example... Figure 7As shown, when the camera HAL frame return time is uneven, the time interval between two frames when the application is rendering the image can also be uneven. For example, the application can render frame 1 in Vsync1 cycle, frame 2 in Vsync2 cycle, frame 3 in Vsync4 cycle, frame 4 in Vsync5 cycle, and frame 5 in Vsync8 cycle.

[0099] Furthermore, the application can send the rendered frames to the surface flinger via BBQ. When the surface flinger composites the frames, the time interval between the composites of adjacent frames is not uniform, resulting in uneven display of adjacent frames on the screen. For example, the screen displays frame 1 in Vsync3 cycle, frame 2 in Vsync4 cycle, and frame 3 in Vsync6 cycle, which brings a sense of lag to the user.

[0100] In view of this, embodiments of this application provide a method for stabilizing image frames, enabling terminal devices to ensure the smoothness of the preview screen when performing layer compositing on image frames.

[0101] It is understood that the aforementioned terminal devices can also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Terminal devices can include mobile phones with touchscreens, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments of this application do not limit the specific technologies or device forms used in the terminal devices.

[0102] Therefore, in order to better understand the embodiments of this application, the structure of the terminal device of the embodiments of this application will be described below. For example, Figure 8This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application.

[0103] The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, an indicator 192, a camera 193, and a display screen 194, etc. The sensor module 180 may include one or more of the following: a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, or a bone conduction sensor, etc.

[0104] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0105] The processor 110 may include one or more processing units. These processing units may be independent devices or integrated within one or more processors. The processor 110 may also include memory for storing instructions and data.

[0106] USB port 130 is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge terminal devices, and can also be used for data transfer between terminal devices and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0107] The charging management module 140 receives charging input from the charger. The charger can be a wireless charger or a wired charger. The power management module 141 connects the charging management module 140 to the processor 110.

[0108] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0109] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Antennas in terminal equipment can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0110] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on terminal devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation.

[0111] The wireless communication module 160 can provide solutions for wireless communication applications on terminal devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), etc.

[0112] The terminal device implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering.

[0113] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device may include one or N display screens 194, where N is a positive integer greater than 1.

[0114] Terminal devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0115] Camera 193 is used to capture still images or videos. In some embodiments, the terminal device may include one or N cameras 193, where N is a positive integer greater than 1.

[0116] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0117] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area.

[0118] The terminal device can implement audio functions such as music playback and recording through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor.

[0119] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The speaker 170A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device can listen to music or hands-free calls through the speaker 170A. The receiver 170B, also called a "handpiece," is used to convert audio electrical signals into sound signals. When the terminal device answers a phone call or voice message, it can listen to the voice by bringing the receiver 170B close to the ear. The headphone jack 170D is used to connect wired headphones. The microphone 170C, also called a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. In this embodiment, the terminal device may have one microphone 170C.

[0120] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. The terminal device can receive button input and generate key signal inputs related to user settings and function control of the terminal device. Indicator 192 can be an indicator light, used to indicate charging status, power level changes, messages, missed calls, notifications, etc.

[0121] The software system of terminal devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc., which will not be elaborated here.

[0122] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other. The same or similar concepts or processes may not be described again in some embodiments.

[0123] To address the issue of stuttering during display on terminal devices caused by uneven frame return times from the camera HAL, this application provides a method for stabilizing image frames. The image frame stabilization process can be performed before the display compositing system composites the image frames (see [link]). Figure 9 (Corresponding embodiments), or the process of stabilizing the image frame can also be performed after the application has drawn and rendered the image frame (see...). Figure 11 (Corresponding embodiment).

[0124] For example, Figure 9 This application provides a method for stabilizing image frames. Figure 9 In corresponding embodiments, the terminal device may include: surface view, camera HAL, camera service, and surfaceflinger, etc.

[0125] like Figure 9 As shown, the method for stabilizing image frames may include the following steps:

[0126] S901, the surface view sends a command to the camera service to create a 30Hz preview stream.

[0127] The 30Hz in the instruction for creating a 30Hz preview stream is merely an example and does not constitute a limitation on the embodiments of this application. This 30Hz can also be referred to as the first frame rate. It is understood that because the stuttering of the terminal device is more noticeable when the frame rate is low, this first frame rate is typically less than 60Hz.

[0128] For example, when the surface view receives an operation from the user to open the preview screen, the surface view can send an instruction to the camera service to create a 30Hz preview stream; in a possible implementation, the camera service can also return a response message to the surface view. The operation to open the preview screen can be: opening the camera application, or switching from other functions of the camera application to the preview screen, etc., and this embodiment does not limit the specific actions taken.

[0129] S902, camera service sends output parameters to camera HAL.

[0130] The output parameters may include one or more of the following: output mode, output image size, or first frame rate (e.g., 30Hz).

[0131] Accordingly, when the camera HAL receives the output parameters, the camera HAL can continue to send the output parameters to the camera sensor, so that the camera sensor can output the image according to the output parameters and generate the original image sequence; the camera sensor sends the original image sequence to the camera HAL for image processing; further, the camera HAL can perform the steps shown in S905 after performing image processing on each image in the original image sequence.

[0132] In one possible implementation, when the camera HAL receives the output parameters, it can also return a response message to the camera service.

[0133] S903, Surface View creates BBQ (producer is Surface View, consumer is SurfaceFlinger).

[0134] BBQ can also be used as a consumer of the buffer queue.

[0135] S904, Surface view creates buffer queue (producer is camera service, consumer is surface view).

[0136] For detailed steps on creating the BBQ and buffer queue in the surface view, please refer to [link / reference]. Figure 2 The descriptions in the corresponding embodiments will not be repeated here.

[0137] S905, camera HAL sends target cache data (layer name) to camera service.

[0138] The target cache data (or target buffer) can be the data obtained after image processing by the camera HAL, and each cache data can correspond to a layer name. The target cache data can be understood as any buffer generated by the camera HAL when outputting images at a frame rate of 30Hz. When the target cache data is data used to support the preview screen, the layer name can be the camera's layer view (or preview layer), such as SurfaceView[camera].

[0139] For example, the camera HAL can receive the raw image sequence returned by the camera sensor and perform image processing on each image frame in the raw image sequence. For example, the camera HAL can perform image processing on the raw cache data in the raw image sequence to obtain target cache data, and then send the target cache data and the layer name corresponding to the target cache data to the camera service.

[0140] It is understandable that due to the differences between the image frames in the original image sequence, the time taken for the camera HAL to process each image frame varies, resulting in uneven frame return times from the camera HAL to the camera service and causing stuttering in the preview. This image processing may include one or more of the following: image cropping, image effects processing, image stabilization, or image recognition, etc., which are not limited in this embodiment.

[0141] S906, the camera service adds a consumption rate identifier to the target cached data and sends the target cached data (consumption rate identifier, layer name) to the surface view.

[0142] The consumption rate identifier can indicate the target consumption rate, which can be the same as the first frame rate, for example, 30Hz. This consumption rate identifier allows the surface flinger to determine the synthesis of target cached data based on the target consumption rate, thereby reducing screen stuttering caused by uneven frame output time of the camera HAL.

[0143] For example, a camera service can create a BufferItem object, associate it with the target cached data of the camera HAL, set a consumption rate flag for the target cached data in the BufferItem object, and then call the bufferqueue interface to send the BufferItem object to the surface view. It can be understood that the camera service can set a consumption rate flag for any received buffer.

[0144] S907, Surface View sends target cache data (consumption rate identifier, layer name) to Surface Flinger via BBQ.

[0145] In a possible implementation, when the surface view receives target cached data (consumption rate identifier, layer name), it can determine whether the consumption rate identifier and layer name of the target cached data have changed compared to those of the previous buffer. For example, if the surface view determines that the consumption rate identifier of the target cached data has changed, it sends the target cached data with the consumption rate identifier to the surface flinger; or, if it determines that the layer name of the target cached data has changed, it sends the target cached data with the layer name to the surface flinger; or, if the surface view determines that neither the consumption rate identifier nor the layer name has changed, it can send the target cached data to the surface flinger.

[0146] Understandably, even if the surface flinger receives the target cached data but not the consumption rate identifier and layer name, the surface flinger can still determine that the consumption rate identifier and layer name of the target cached data have not changed based on the layer name and consumption rate identifier of the previously cached buffer.

[0147] During the process of the surface view sending target cached data to the surface flinger via BBQ, the surface view can receive the target cached data in the BufferItem object; the surface view calls the acquireBuffer function to consume the target cached data, and calls the queueBuffer function to send the BufferItem object to BBQ; after receiving the target cached data, BBQ calls the setTransaction interface to pass the target cached data to the surface flinger through the transaction object.

[0148] Understandably, the transaction object serves as the medium for communication with the surface flinger. Besides the target cache data, this transaction object can also include surface-related data, such as the layer name, the surface's width, height, transparency, and scaling / panning parameters that need updating. Since many parameters need to be sent to the surface flinger within a single frame, the surface view sets all these surface-related parameters in the transaction object and sends them together to the surface flinger, reducing inter-process communication time. Similarly, the buffer queue can not only set the target cache data in the transaction object, but also place the buffer's consumption rate identifier, buffer ID update data, format change data, and buffer size change data in the transaction object, and then send them to the surface flinger together.

[0149] In this embodiment of the application, the added interface 1 can be interface SetComsumeRate1, which is used to instruct the camera application to pass the consumption rate identifier of the current layer to the Surface view.

[0150] The added interface 2 can be SetConsumeRate2, which is used to instruct the surface view to pass the current layer's consumption rate flag and layer name to the surface flinger.

[0151] The added interface 3 can be SetComsumeRate3, which is used to instruct the cameraservice to pass the consumption rate identifier of the current layer to the bufferqueue.

[0152] The added interface 4 can be SetComsumeRate4, which is used to instruct the surface view to pass the BBQ consumption rate indicator to the BBQ.

[0153] When an application needs to set a consumption rate flag, it can set it in either the buffer or the surface. The buffer and surface are interconnected. Before the transaction object sends parameters to the surfaceflinger, it is associated with a `layer_state_t` structure object. This structure contains all information about layer changes, such as: whether the layer is visible, the layer offset, whether it is a safe layer, the surfaceid (or layer name), the bufferid (e.g., the name of the target cached data), the width and height of the interface, transparency, and scaling and panning parameters. After receiving the `layer_state_t`, the surfaceflinger can parse out the corresponding layer and its name within the current process.

[0154] During the creation of the transaction object, the parameter `sc` can be a pointer to the layer, and the parameter `rate` is the target consumption rate. After calling the `setConsumeRate` method to set the consumption rate flag, the camera application can call the `apply` method to send the parameters to the surface flinger. The parameter `sc` can be passed in BBQ, which stores which layer the current buffer queue corresponds to.

[0155] The surface view can render the target cached data and, after rendering the target cached data, send the rendered target cached data to surfaceflinger via BBQ.

[0156] S908 and Surface Flinger cache the first target cache data in the transaction queue.

[0157] S909 and surface flinger obtain the target consumption rate from the consumption rate identifier and the last consumption time corresponding to the layer name of the first target cache data.

[0158] The surface flinger can store a first mapping relationship between a layer and its corresponding last consumption time. The last consumption time described in step S909 can be understood as the moment when the previous cached data of the layer began to be synthesized. The surface flinger can store the above first mapping relationship in the form of a hash table or similar method.

[0159] For example, when the Surface Flinger receives the Vsync signal, the Surface Flinger can traverse the transaction queue. If the first target cache data in the transaction queue has a consumption rate flag set, the Surface Flinger can obtain the layer name corresponding to the first target cache data, and then obtain the last consumption time corresponding to the layer name of the first target cache data according to the first mapping relationship.

[0160] For example, when the layer name of the first target cached data received by the Surface Flinger is SurfaceView[camera], the Surface Flinger can look up the consumption time of the previous cached data of that SurfaceView[camera].

[0161] In a possible implementation, the surface flinger can also be configured with a correspondence between layers and their corresponding consumption rates. This allows the surface flinger to determine the target consumption rate based on the layer name corresponding to the first target cache data when it receives the first target cache data and the layer name of that first target cache data. In this case, even if the surface flinger does not receive a consumption rate identifier, it can still judge the synthesis status of the first target cache data based on the preset consumption rate associated with the layer.

[0162] When the S910 and surface flinger determine that the target consumption rate meets the preset range, they decide whether to perform layer synthesis on the first target cached data based on the last consumption time and the current system time.

[0163] The preset range can be a consumption rate ranging from 1Hz to 120Hz, or it can be a value ranging from 1Hz to 60Hz. This application embodiment does not limit this. Any consumption rate value within the preset range can be less than the system refresh rate.

[0164] For example, when the surface flinger determines that the target consumption rate meets a preset range, it can calculate a first difference between the current system time and the last consumption time corresponding to the layer of the target cache data. This first difference can be the value obtained by subtracting the last consumption time corresponding to the layer of the target cache data from the current system time. This first difference can be understood as the actual time interval between the time when the first target cache data is layered and the time when the previous buffer of the first target cache data is layered.

[0165] Furthermore, when the surface flinger determines that the first difference is less than 1 second / target consumption rate * m, the surface flinger may not perform layer compositing on the first target cache data. In a possible implementation, if the surface flinger determines that layer compositing on the first target cache data is not necessary, the surface flinger can continue to determine whether to perform layer compositing on the first target cache data when the next Vsync signal arrives.

[0166] Here, 1 second / target consumption rate can be understood as the time required for the first target cache data to be properly composited into layers, or it can be understood as the theoretical time interval between the time it takes for the first target cache data to be composited into layers and the time it takes for the previous buffer of the first target cache data to be composited into layers. The value of m can be 0.75 or 0.8, or the value of m can be in the range of 0.5-1, and m can be used to compensate for the stuttering caused by uneven frame output time of cameraHAL. In this embodiment, the value of m is not limited.

[0167] It is understandable that the first difference being less than 1 second / target consumption rate * m can be interpreted as: the actual time interval between the first target cache data and the previous buffer of the first target cache data is less than the theoretical time interval between the two buffers. In other words, in order to ensure that the target consumption rate cannot complete the layer composition of the first target cache data within the actual time interval, the surface flinger must ensure that the target consumption rate cannot complete the layer composition of the first target cache data within the actual time interval.

[0168] Alternatively, when the surface flinger determines that the first difference is greater than or equal to 1 second / target consumption rate * m, the surface flinger can perform layer synthesis on the first target cached data and set the last consumption time of the layer to the current system time.

[0169] It can be understood that the first difference being greater than or equal to 1 second / target consumption rate * m can be interpreted as: the actual time interval between the two buffers being greater than or equal to the theoretical time interval between the two buffers, meaning that SurfaceFlinger can complete the layer synthesis for the first target cached data within a sufficient time interval.

[0170] In possible implementations, when the surface flinger determines that the target consumption rate does not meet the preset range, such as the target consumption rate being 0Hz, -1Hz, or greater than 120Hz, the surface flinger can determine that no special processing is needed for the target consumption rate, perform layer synthesis on the first target cached data, and set the last consumption time corresponding to the layer name to 0.

[0171] In a possible implementation, if there are unconsumed buffers in the surface flinger, for example, if the transaction queue includes data other than the first target cache data, the surface flinger can request the next Vsync signal, and upon the arrival of the next Vsync signal, continue to process the unconsumed buffers according to the steps shown in S909-S910.

[0172] After S910, the surface flinger can send the composited buffer to the HWC (Hardware Wiring Center). The HWC then processes the composited buffer, performing overlay and other processing, before forwarding it to the display driver for display, allowing the terminal device to display as shown. Figure 1 The preview shown in 'a' is shown in the image.

[0173] Based on this, even if the camera HAL frame output time is uneven, the surface flinger can ensure sufficient layer compositing time when compositing layers on the buffer, and perform layer compositing at a stable consumption rate to reduce screen stuttering.

[0174] In a possible implementation, the preview display process also includes the display of control layers. For example, the camera HAL can acquire multiple control image frames based on the frame rate corresponding to the control layer; and send these multiple control image frames along the camera HAL, camera service, and surface view to the surface flinger. Since the frame rate corresponding to the control layer (e.g., 60Hz) is usually greater than the frame rate corresponding to the preview layer (e.g., target consumption rate, 30Hz), the surfaceflinger can acquire the control layer within the cycle of compositing any preview layer, and obtain the preview image through the compositing and display of the preview layer and control layer.

[0175] Understandably, when the camera HAL returns an image sequence in step S905, the terminal device can perform steps S906-S910 on each image frame in the image sequence, that is, perform frame stabilization in the surface flinger, so that the terminal device can display continuous image frames uniformly. For example, when the image sequence returned by the camera HAL includes frame 1, frame 2, and frame 3, the time interval between frame 1 and frame 2 is the first interval, and the time interval between frame 2 and frame 3 is the second interval, with the first and second intervals being of different sizes. Accordingly, the surface flinger can obtain a stable image after processing frame 1, frame 2, and frame 3.

[0176] For example, Figure 10 This is a schematic diagram illustrating yet another type of consumer buffer provided in embodiments of this application. Figure 10 In the corresponding embodiment, an example is given with a screen refresh rate of 60Hz and a target consumption rate of 30Hz determined by the surface flinger. When the target consumption rate is 30Hz and m is 0.8, the surface flinger can determine that the theoretical time interval between two adjacent buffers is 1 / 30Hz * 0.8, which is approximately 27ms.

[0177] like Figure 10 As shown, the frame output time of the camera HAL is uneven, causing the application to draw and render frame 1 in Vsync1 cycle, frame 2 in Vsync2 cycle, frame 3 in Vsync4 cycle, frame 4 in Vsync5 cycle, and frame 5 in Vsync8 cycle.

[0178] For frame 1, the application can draw and render frame 1 within a Vsync1 period (e.g., around 0ms), and cache frame 1 in BBQ after drawing and rendering is complete; the surface flinger obtains frame 1, and since frame 1 is the first frame, the surface flinger can composite frame 1 within a Vsync1 period (e.g., around 16.6ms); the screen displays frame 1 within a Vsync3 period (e.g., around 33.2ms).

[0179] For frame 2, the application can render frame 2 within a Vsync2 period (approximately 16.6ms) and cache it in BBQ after rendering. The surface flinger acquires frame 2. Since the actual time interval between the current system time of frame 2 (e.g., a certain moment within a Vsync3 period) and the consumption time of frame 1 is less than the theoretical time interval, the surface flinger can temporarily refrain from compositing frame 2. Furthermore, upon receiving Vsync4, the surface flinger can determine that the actual time interval between the current system time of frame 2 (e.g., a certain moment within a Vsync4 period) and the consumption time of frame 1 is greater than the theoretical time interval. Therefore, the surface flinger can composit frame 2 within a Vsync4 period. The descriptions of the theoretical and actual time intervals can be found in step S910.

[0180] Similarly, the method by which the surface flinger determines whether to composite any frame upon the arrival of the Vsync signal can be found in frames 1 and 2, and will not be repeated here. See also Figure 10 As can be seen, Surface Flinger can determine the target consumption rate and the time interval between compositing two adjacent buffers, enabling it to composite the buffers relatively evenly and alleviate screen stuttering.

[0181] For example, Figure 11 Another method for stabilizing image frames is provided as an embodiment of this application. For example... Figure 11 As shown, a method for stabilizing image frames may include the following steps:

[0182] S1101, the surface view sends a command to the camera service to create a 30Hz preview stream.

[0183] S1102, camera service sends output parameters to camera HAL.

[0184] S1103, Surface View creates BBQ (producer is Surface View, consumer is SurfaceFlinger).

[0185] S1104. Surface view creates buffer queue (producer is camera service, consumer is surface view).

[0186] S1105, Camera HAL sends target cache data (layer name) to Camera service.

[0187] S1106, the camera service adds a consumption rate identifier to the target cached data and sends the target cached data (consumption rate identifier, layer name) to the surface view.

[0188] The descriptions of steps S1101-S1106 can be found in steps S901-S906, and will not be repeated here.

[0189] S1107, the surface view draws and renders the target cache data to obtain the second target cache data.

[0190] S1108, the surface view obtains the target consumption rate from the consumption rate identifier and obtains the last transmission time corresponding to the layer name of the second target cache data.

[0191] The surface view can store a primary mapping between a layer and its last sent time. This last sent time can be understood as the moment when the layer's previous cached data was sent to the surface flinger. Alternatively, this last sent time could also be understood as the time when the layer's previous cached data was composited in the surface flinger.

[0192] For example, when the layer name of the second target cached data received by the surface view is SurfaceView[camera], the surface view can look up the sending time of the previous cached data of that SurfaceView[camera] consumed by the surface flinger.

[0193] S1109, the surface view determines whether to send the second target cached data to the surface flinger based on the last sending time and the current system time.

[0194] For example, the surface view can calculate a second difference between the current system time and the last sent time of the layer. This second difference can be the value obtained by subtracting the last sent time of the layer from the current system time. This second difference can be understood as the actual time interval between the sent time of the second target cache data and the sent time of the previous buffer of the second target cache data being sent to the surface flinger.

[0195] Furthermore, when the surface view determines that the second difference is less than 1 second / target consumption rate * m, the surface view may not send the second target cached data to the surface flinger in the current cycle. In a possible implementation, if the surface view determines that it can refrain from sending the second target cached data to the surface flinger in the current cycle, the surface view may continue to determine whether to send the second target cached data to the surface flinger when the next Vsync signal arrives.

[0196] Alternatively, when the surface view determines that the second difference is greater than or equal to 1 second / target consumption rate * m, the surface flinger can send the second target cached data to the surface flinger and set the last sending time of the layer to the current system time.

[0197] Understandably, the surface view can perform frame stabilization processing on the target cached data based on the target consumption rate. For example, it can determine whether to send the second target cached data to the surface flinger within the vsync signal cycle. In this way, when the surface view performs frame stabilization processing on each frame in the image sequence, it can ensure that the rendered image can be output to the surface flinger as evenly as possible, thereby improving the stuttering effect.

[0198] S1110, the surface view sends the second target cache data (consumption rate identifier, layer name) to the surface flinger via BBQ.

[0199] S1111, the surface flinger caches the second target cache data in the transaction queue.

[0200] S1112, surface flinger performs layer composition on the second target cached data.

[0201] In a possible implementation, when the camera HAL service returns an image sequence in step S1105, the terminal device can perform steps S1106-S1112 on each image frame in the image sequence, i.e., perform frame stabilization in the SurfaceView, so that the terminal device can display continuous image frames uniformly. For example, when the image sequence returned by the camera HAL includes frame 1, frame 2, and frame 3, the time interval between frame 1 and frame 2 is the first interval, and the time interval between frame 2 and frame 3 is the second interval, with the first and second intervals having different sizes. Accordingly, the SurfaceView can perform rendering processing on frame 1, frame 2, and frame 3, and send the rendered three frames sequentially to the Surface Flinger according to the target consumption rate, so that the terminal device can obtain a stable image.

[0202] Based on this, even if the camera HAL frame output time is uneven, the surface view can stably send image frames to the surface flinger, thereby enabling the surface flinger to perform layer compositing based on a stable consumption rate, thus reducing screen stuttering.

[0203] For example, Figure 12 This is a schematic diagram illustrating yet another type of consumer buffer provided in an embodiment of this application. Figure 12 In the corresponding embodiment, an example is given with a screen refresh rate of 60Hz and a target consumption rate of 30Hz determined by the surface flinger based on the layer. When the target consumption rate is 30Hz and m is 0.8, the surface flinger can determine that the theoretical time interval between two adjacent buffers is 1 / 30Hz * 0.8, which is approximately 27ms. The description of this theoretical time interval and the actual time interval can be found in step S1109.

[0204] like Figure 12 As shown, the camera HAL frame output time is uneven, and the time from image frame input to the application is also uneven. The application can be based on... Figure 11 The method for stabilizing image frames in the corresponding embodiment sends the image frames evenly to the surfaceflinger.

[0205] For example, an application can draw and render frame 1 in Vsync1 cycle, frame 2 in Vsync2 cycle, frame 3 in Vsync4 cycle, frame 4 in Vsync5 cycle, and frame 5 in Vsync8 cycle.

[0206] After rendering, for any frame, such as frame 2, since the actual time interval between the current system time of frame 2 and the time when frame 1 was sent to BBQ (or surface flinger) is less than the theoretical time interval, the application can temporarily refrain from sending frame 2 to BBQ within the Vsync2 cycle. Furthermore, upon receiving Vsync3, the application can determine that the actual time interval between the current system time of frame 2 and the time when frame 1 was sent to BBQ (or surface flinger) is greater than the theoretical time interval; therefore, the application can send frame 2 to BBQ within the Vsync3 cycle.

[0207] Similarly, the method by which the application determines whether to send frames 3, 4, and 5 when the Vsync signal arrives can be found in the description of frame 2, and will not be repeated here.

[0208] See Figure 12 As can be seen, the application can determine the target consumption rate and the time interval between sending two adjacent buffers to the BBQ (or surface flinger), allowing the application to send the buffers relatively evenly to the BBQ and surface flinger, thereby alleviating screen stuttering when displaying consecutive frames. For example, the application can stabilize image frames to ensure that frames 1, 2, 3, 4, and 5 are displayed uniformly.

[0209] It is understood that the method for stabilizing image frames provided in this application embodiment can be used not only for, but also for, other applications. Figure 9 or Figure 11 The scenario shown is a camera displaying a preview image, and it can also be applied to scenarios where the received video is stuttering due to network latency. This application does not limit this aspect.

[0210] The above combination Figures 4-12 The methods provided in the embodiments of this application have been described. The apparatus for executing the above methods, provided in the embodiments of this application, is described below. Figure 13 As shown, Figure 13 This is a schematic diagram of a device for stabilizing image frames provided in an embodiment of this application. The device for stabilizing image frames can be a terminal device in the embodiment of this application, or a chip or chip system within the terminal device.

[0211] like Figure 13As shown, the image frame stabilization device 1300 can be used in communication equipment, circuits, hardware components, or chips. The image frame stabilization device includes a display unit 1301 and a processing unit 1302. The display unit 1301 supports the display steps performed by the image frame stabilization device 1300; the processing unit 1302 supports the information processing steps performed by the image frame stabilization device 1300.

[0212] In a possible implementation, the device 1300 for stabilizing image frames may also include a communication unit 1303. Specifically, the communication unit supports the device 1300 for stabilizing image frames in performing data transmission and data reception steps. The communication unit 1303 may be an input or output interface, pins, or circuitry, etc.

[0213] In a possible embodiment, the device for stabilizing image frames may further include a storage unit 1304. The processing unit 1302 and the storage unit 1304 are connected via a line. The storage unit 1304 may include one or more memories, which may be devices in one or more devices or circuits used for storing programs or data. The storage unit 1304 may exist independently and be connected to the processing unit 1302 of the device for stabilizing image frames via a communication line. Alternatively, the storage unit 1304 may be integrated with the processing unit 1302.

[0214] Storage unit 1304 may store computer-executable instructions for the methods in the terminal device, so that processing unit 1302 executes the methods in the above embodiments. Storage unit 1304 may be a register, cache, or RAM, etc., and storage unit 1304 may be integrated with processing unit 1302. Storage unit 1304 may be a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, and storage unit 1304 may be independent of processing unit 1302.

[0215] Figure 14 This is a schematic diagram of the hardware structure of another terminal device provided in an embodiment of this application, such as... Figure 14 As shown, the terminal device includes a processor 1401, a communication line 1404, and at least one communication interface. Figure 14 (The example described uses communication interface 1403 as an example).

[0216] The processor 1401 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs according to the present application.

[0217] Communication line 1404 may include circuitry for transmitting information between the aforementioned components.

[0218] Communication interface 1403 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0219] The terminal device may also include a memory 1402.

[0220] The memory 1402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via communication line 1404. The memory may also be integrated with the processor.

[0221] The memory 1402 stores computer execution instructions for implementing the scheme of this application, and the processor 1401 controls the execution of these instructions. The processor 1401 executes the computer execution instructions stored in the memory 1402 to implement the method provided in the embodiments of this application.

[0222] It is possible that the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0223] In a specific implementation, as one embodiment, the processor 1401 may include one or more CPUs, for example... Figure 14 CPU0 and CPU1 in the CPU.

[0224] In a specific implementation, as one example, the terminal device may include multiple processors, for example... Figure 14 Processors 1401 and 1405 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0225] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0226] This application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium capable of transferring a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0227] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may also include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers.

[0228] The above combinations should also be included within the scope of computer-readable media. The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image processing method, the method being applied to an electronic device, the electronic device including a screen, the electronic device having a camera application installed, characterized in that, include: The camera application sequentially obtains the first image frame, the second image frame, and the third image frame; Wherein, the time interval between obtaining the first image frame and obtaining the second image frame is the first time interval, and the time interval between obtaining the second image frame and obtaining the third image frame is the second time interval, and the first time interval and the second time interval are of different sizes; During the first vertical synchronization (Vsync) period, the first image frame is processed by the surface flinger display compositing system to obtain the first composite image frame. After completing the synthesis processing of the first image frame, the first synthesized image frame is sent to the display driver of the screen through the surface flinger within the first Vsync cycle. During the second Vsync cycle, the second image frame is synthesized using the surface flinger to obtain the second synthesized image frame; After the synthesis processing of the second image frame is completed, the second synthesized image frame is sent to the display driver through the surface flinger within the second Vsync cycle; During the third Vsync cycle, the third image frame is synthesized by the surface flinger to obtain the third synthesized image frame. After the synthesis processing of the third image frame is completed, the third synthesized image frame is sent to the display driver through the surface flinger during the third Vsync cycle; Wherein, the number of Vsync cycles between the first Vsync cycle and the second Vsync cycle is the first number, and the number of Vsync cycles between the second Vsync cycle and the third Vsync cycle is the second number. The first number and the second number are the same. The time interval corresponding to the first number of Vsync cycles is the third time interval, and the time interval corresponding to the second number of Vsync cycles is the fourth time interval. The first time interval is less than the third time interval, and the second time interval is greater than the fourth time interval. The screen displays the first synthesized image frame during the fourth Vsync cycle; The screen displays the second composite image frame during the fifth Vsync cycle; The screen displays the third composite image frame during the sixth Vsync cycle; Specifically, the fourth Vsync cycle is located after the first Vsync cycle and is adjacent to the first Vsync cycle; the fifth Vsync cycle is located after the second Vsync cycle and is adjacent to the second Vsync cycle; and the sixth Vsync cycle is located after the third Vsync cycle and is adjacent to the third Vsync cycle. The number of Vsync cycles that differs between the fourth Vsync cycle and the fifth Vsync cycle is the third number, and the number of Vsync cycles that differs between the fifth Vsync cycle and the sixth Vsync cycle is the fourth number. The third number is the same as the fourth number.

2. The method as described in claim 1, characterized in that, The method further includes: If the time interval between the seventh Vsync cycle and the first Vsync cycle is less than a preset time interval, the surface flinger does not perform synthesis processing within the seventh Vsync cycle, wherein the seventh Vsync cycle is after the first Vsync cycle and before the second Vsync cycle.

3. The method as described in claim 1, characterized in that, During the second Vsync cycle, the second image frame is synthesized using SurfaceFlinger to obtain the second synthesized image frame, including: If the time interval between the second Vsync period and the first Vsync period is greater than a preset time interval, the second image frame is synthesized by the surface flinger within the second Vsync period to obtain the second synthesized image frame.

4. The method according to any one of claims 1, characterized in that, The method further includes: The first image frame is generated using the camera module; The first image frame is sent to the camera service through the camera hardware abstraction layer service; The first image frame is sent to the camera application via the camera service; The second image frame is generated using the camera module; The second image frame is sent to the camera service through the camera hardware abstraction layer service; The second image frame is sent to the camera application via the camera service; The third image frame is generated by the camera module; The third image frame is sent to the camera service through the camera hardware abstraction layer service; The camera service sends the third image frame to the camera application, wherein the camera module generates the first image frame, the second image frame, and the third image frame based on the first frame rate.

5. The method as described in claim 4, characterized in that, The time interval between sending the first image frame to the camera service and sending the second image frame to the camera service is a fifth time interval, and the time interval between sending the second image frame to the camera service and sending the third image frame to the camera service is a sixth time interval, wherein the fifth time interval and the sixth time interval are of different sizes.

6. The method as described in claim 4, characterized in that, The first time interval is less than the reciprocal of the first frame rate, and the second time interval is greater than the reciprocal of the first frame rate.

7. The method as described in claim 4, characterized in that, The frame rate of the screen refresh image frame is the second frame rate, and the first frame rate is less than the second frame rate.

8. The method as described in claim 7, characterized in that, The second frame rate is 60Hz.

9. The method according to any one of claims 4-8, characterized in that, The first frame rate is 30 Hz.

10. The method according to any one of claims 1-8, characterized in that, The process of synthesizing the first image frame using a vertical synchronous surface flinger during the first Vsync period to obtain a first synthesized image frame includes: During the first Vsync cycle, the buffered data of the first image frame is obtained through the surface flinger, and the first image frame is synthesized to obtain the first synthesized image frame.

11. The method according to any one of claims 1-8, characterized in that, The method is applied to the camera application's photo preview interface or video preview interface.

12. The method according to any one of claims 1-8, characterized in that, Before obtaining the first image frame, the second image frame, and the third image frame sequentially through the camera application, the method further includes: The camera application generates instructions for creating a preview stream.

13. An electronic device, characterized in that, include: One or more processors or memories; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-12.

15. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-12.

Images