A display method, an electronic device, and a computer-readable storage medium

Abstract

The application provides a display method, an electronic device and a computer readable storage medium, and relates to the technical field of terminals. In a case where the electronic device displays a first image frame, a user can trigger a sliding operation for the first image frame. In response to the sliding operation triggered by the user, the electronic device can display a second image frame and a third image frame. The second image frame is an interpolated frame between the first image frame and the third image frame. The second image frame is determined based on a fourth image frame that has been stored in the electronic device before the sliding operation is triggered, so that the electronic device does not need to wait for the third image frame to be generated before generating the second image frame. In this way, the display speed of the second image frame can be effectively improved, and the hand delay can be reduced.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a display method, electronic device and computer-readable storage medium. Background Technology

[0002] Since the screen size of electronic devices is fixed, the display space is limited. Current interface design often utilizes swiping gestures to achieve user interaction. This allows users to switch content displayed on the screen via swiping, avoiding the need to cram too many controls into the already limited display space, thus making the electronic device's interface more concise and aesthetically pleasing.

[0003] Currently, after a user triggers a swipe, the electronic device displays the corresponding screen only after a certain delay, resulting in poor responsiveness and affecting the user's operating experience. Summary of the Invention

[0004] In view of this, this application provides a display method, an electronic device, and a computer-readable storage medium that can pre-draw a portion of an image before displaying a real image frame, and then directly generate the image to be displayed based on the pre-drawn image, thereby improving display speed.

[0005] Firstly, this application provides a display method applied to an electronic device. The electronic device displays a first image frame, and a user can trigger a swipe operation on the first image frame. In response to this swipe operation, the electronic device can display a second image frame and a third image frame to achieve a swipe display effect. Specifically, the swipe operation can trigger the electronic device to display the third image frame, and before displaying the third image frame, the electronic device can display an interpolated frame between the first and third image frames, i.e., display the second image frame. Thus, using the second image frame as an interpolated frame between the first and third image frames for transition makes the swipe display process smoother and effectively eliminates screen jumps, thereby improving the user experience.

[0006] The second image frame is generated based on the fourth image frame, which is stored in the electronic device before receiving the swipe operation. The fourth image frame includes at least the first image frame and first image content, where the first image content is overflow content continuous with the image content of the first image frame. Because the fourth image frame is stored in the electronic device before receiving the swipe operation, the electronic device can load and render the second image frame faster, reducing user waiting time and improving image display response speed. Furthermore, using the stored fourth image frame to generate the second image frame avoids repeatedly loading and rendering the same image content, thus optimizing the resource utilization of the electronic device and reducing its processing load.

[0007] In some of the above implementations, during the sliding display process of the electronic device in response to the user-triggered sliding operation, the time interval between the first image frame and the third image frame is short, and the user's sliding speed will not be too high. Therefore, there are repeated interface elements between the first image frame and the third image frame. Thus, frame interpolation can be performed between the first image frame and the third image frame.

[0008] In some of the other implementations described above, where there are no identical interface elements between the first and third image frames, but there are identical interface elements between the fourth and third image frames, frame interpolation can be performed between the first and third image frames based on the fourth image frame.

[0009] In the above implementation, the first image content is overflow content that is continuous with the image content of the first image frame. The overflow position of the first image content is related to the direction of the trigger operation received before the electronic device displays the first image frame. In some examples, the overflow direction of the first image content is opposite to the direction of the swipe operation triggered by the user on the original frame before the electronic device displays the first image frame.

[0010] In one possible implementation of the first aspect, the third image frame includes the second image content, which is generated based on the fourth image frame.

[0011] Generating the second image content using an existing fourth image frame avoids repeatedly loading and rendering the same image data, thus optimizing the resource utilization of electronic devices. Furthermore, using the fourth image frame increases the generation speed of the third image frame, thereby improving the response speed of swiping operations and reducing user waiting time.

[0012] In one possible implementation of the first aspect, the method further includes, before displaying the second image frame:

[0013] In response to a swipe operation, obtain the first motion vector of the same interface element between the first image frame and the second image frame;

[0014] The second image frame is extracted from the fourth image frame based on the first motion vector.

[0015] By obtaining the first motion vector of the same interface element between the first and second image frames, the part to be cropped in the fourth image frame can be located more accurately, thereby generating a more accurate second image frame and improving the accuracy of image processing.

[0016] In one possible implementation of the first aspect, extracting the second image frame from the fourth image frame based on the first motion vector includes:

[0017] The first starting position and the first ending position are determined based on the first motion vector;

[0018] The image content from the first start position to the first end position in the fourth image frame is extracted as the second image frame;

[0019] The first motion vector includes a first displacement and a first direction. The distance between the image content corresponding to the first starting position and the image content corresponding to the first position in the fourth image frame is equal to the first displacement. The direction from the first position to the first starting position is opposite to the first direction. The first position is the origin of the fourth image frame in the screen coordinate system. The distance between the first ending position and the first starting position is related to the size of the image frame displayed by the electronic device.

[0020] The first motion vector reflects the direction and displacement of the interface elements between the first and second image frames. Therefore, by determining the start and end positions using the first motion vector, the position information of the image content of the second image frame in the fourth image frame can be accurately determined. Thus, by cropping the second image frame from the first start position to the first end position, the image portion of the second image frame can be accurately obtained, discarding irrelevant frames. Furthermore, this process eliminates the need to process the fourth image frame, significantly reducing the amount of data that needs to be processed. This reduction in data volume also lowers the demand for computing and storage resources, thereby reducing the workload.

[0021] In one possible implementation of the first aspect, extracting the image content from the first start position to the first end position in the fourth image frame as the second image frame includes:

[0022] The image content from the first start position to the first end position in the fourth image frame is copied to the storage space of the second image frame to obtain the second image frame.

[0023] By generating a second image frame through duplication, the image content can be prevented from being damaged, thus effectively ensuring the image quality of the second image frame.

[0024] In one possible implementation of the first aspect, the method further includes, before displaying the third image frame:

[0025] In response to a swipe operation, obtain the second motion vector of the same interface element between the first image frame and the third image frame;

[0026] Extract the content of the second image from the fourth image frame based on the second motion vector;

[0027] A third image frame is generated based on the content of the second image.

[0028] By using the existing fourth image frame and second motion vector to generate the third image frame, the repeated loading and rendering of the same image data can be avoided, thereby optimizing the resource utilization of electronic devices.

[0029] In one possible implementation of the first aspect, in response to a sliding operation, obtaining a second motion vector of the same interface element between a first image frame and a third image frame includes:

[0030] In response to a swipe operation, obtain all interface elements that need to be included in the third image frame indicated by the swipe operation, as well as the position of each interface element;

[0031] Obtain all interface elements in the first image frame and the position of each interface element;

[0032] The second motion vector is determined based on the position of the target interface element in the third image frame and the position of the target interface element in the first image frame. The target interface element is any one of the same interface elements in the third image frame and the first image frame.

[0033] By obtaining all interface elements and their positions in the third image frame, as well as all interface elements and their positions in the first image frame, the second motion vector can be accurately calculated. Thus, based on the second motion vector, both the third and second image frames can be generated quickly and accurately. This improves display speed while simultaneously enhancing image accuracy and quality.

[0034] In one possible implementation of the first aspect, extracting the second image content from the fourth image frame based on the second motion vector includes:

[0035] The second starting position and the second ending position are determined based on the second motion vector;

[0036] The image content from the second start position to the second end position in the fourth image frame is extracted as the second image content;

[0037] The second motion vector includes a second displacement and a second direction;

[0038] The distance between the image content corresponding to the second starting position and the image content corresponding to the first position in the fourth image frame is equal to the second displacement, and the direction from the first position to the second starting position is opposite to the second direction; the first position is the origin of the fourth image frame in the screen coordinate system; the distance between the second ending position and the second starting position is related to the size of the image frame displayed by the electronic device;

[0039] Alternatively, the distance between the image content corresponding to the second starting position and the image content corresponding to the first position in the fourth image frame is less than the second displacement. The second starting position is the position where the fourth image frame is at its minimum horizontal and vertical coordinate values ​​in the screen coordinate system. The direction from the first position to the second starting position is opposite to the second direction. The distance between the second ending position and the second position is related to the size of the image frame displayed by the electronic device. The second position is the position obtained by moving from the first position according to the second motion vector in the screen coordinate system.

[0040] The second motion vector reflects the direction and displacement of interface elements between the first and third image frames. Therefore, by determining the start and end positions using the second motion vector, the position of the image content of the third image frame within the fourth image frame can be accurately determined. Thus, by cropping the third image frame from the second start position to the second end position, the image portion of the third image frame can be accurately obtained, discarding irrelevant frames. Furthermore, no processing of the fourth image frame is required during this process, significantly reducing the amount of data that needs to be processed. This reduction in data volume also lowers the demand for computing and storage resources, thereby reducing the workload.

[0041] In one possible implementation of the first aspect, extracting the image content from the second start position to the second end position in the fourth image frame as the second image content includes:

[0042] The image content from the second start position to the second end position in the fourth image frame is copied to the storage space of the second image content to obtain the second image content.

[0043] By generating a second image by copying, the image content can be protected from damage, thus effectively ensuring the image quality of the second image content.

[0044] In one possible implementation of the first aspect, when the size of the second image content is smaller than the size of the image frame displayed by the electronic device, the method further includes:

[0045] Generate the third image content; the interface elements that need to be included in the third image content are not included in the fourth image frame;

[0046] Generating a third image frame based on the content of the second image includes:

[0047] A third image frame is generated based on the content of the second and third images.

[0048] If the size of the second image content is smaller than the size of the image frame displayed by the electronic device, the third image content, which is not included in the fourth image frame, can be generated first, and then the third image frame can be generated based on the second image content and the third image content to ensure the integrity of the third image frame.

[0049] In one possible implementation of the first aspect, if the size of the second image content is equal to the size of the image frame displayed by the electronic device, the second image content can be directly determined based on the fourth image frame, and the third image frame can be generated.

[0050] In one possible implementation of the first aspect, generating the third image content includes:

[0051] Based on the second image content, the third image content is generated by including all interface elements that need to be included in the third image frame indicated by the sliding operation, as well as the position of each interface element.

[0052] Based on the content of the second image, the missing parts in the third image frame can be determined. Then, based on all the interface elements that the third image frame should include and the position of each interface element as indicated by the sliding operation, the content of the third image frame is accurately generated to ensure the integrity of the third image frame.

[0053] In one possible implementation of the first aspect, after the first image frame is drawn and before the sliding operation is received, the method further includes:

[0054] Obtain the third motion vector, which is the predicted motion vector of the same interface element between the first image frame and the second image frame;

[0055] The content of the first image is drawn based on the third motion vector;

[0056] A fourth image frame is generated based on the content of the first image and the first image frame.

[0057] By obtaining the third motion vector, the content of the first image can be accurately drawn, thus ensuring that the second image frame and the content of the second image in the third image frame can be accurately extracted from the fourth image frame. This further ensures the generation speed of the second and third image frames, as well as the efficiency of the electronic device in generating the second and third image frames.

[0058] In a second aspect, this application provides an electronic device comprising a display screen, a memory, and one or more processors; the display screen, the memory, and the processors are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the method as described in the fourth aspect and any possible design thereof.

[0059] Thirdly, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect above and any possible design of the method thereof.

[0060] Fourthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, cause the electronic device to perform the method described in the first aspect above and any possible design of the method thereof.

[0061] Fifthly, this application provides an apparatus included in an electronic device, which has the function of implementing the behavior of the electronic device in any of the above aspects and possible implementations. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function. For example, a display module or unit, a generation module or unit, a drawing module or unit, an acquisition module or unit, and a storage module or unit, etc.

[0062] Sixthly, embodiments of this application provide a chip system including a processor and potentially a memory, for implementing any of the methods provided in the first aspect above. The chip system may be composed of chips or may include chips and other discrete devices.

[0063] Understandably, the electronic device described in the second aspect and any possible design of the above-described device, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to perform the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0064] Figure 1 A schematic diagram of a sliding operation provided in an embodiment of this application;

[0065] Figure 2 A schematic diagram of a sliding display provided for an embodiment of this application;

[0066] Figure 3 A schematic diagram illustrating a follow-up delay provided in an embodiment of this application;

[0067] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0068] Figure 5 A schematic diagram of the hardware and software structure provided in an embodiment of this application;

[0069] Figure 6 A schematic diagram of an image frame provided in an embodiment of this application. Figure 1 ;

[0070] Figure 7 A schematic diagram illustrating a display method provided in an embodiment of this application;

[0071] Figure 8 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0072] Figure 9 This application provides an illustration of frame interpolation and image frame generation. Figure 1 ;

[0073] Figure 10 This application provides an illustration of frame interpolation and image frame generation. Figure 2 ;

[0074] Figure 11 A schematic diagram illustrating motion vector prediction provided in an embodiment of this application;

[0075] Figure 12 A schematic diagram of an image frame provided in an embodiment of this application. Figure 2 ;

[0076] Figure 13 This is a schematic diagram of frame interpolation generation provided in an embodiment of this application. Detailed Implementation

[0077] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships may exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0078] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0079] Currently, most applications on electronic devices can be interacted with via swipe gestures. Examples include shopping apps, news apps, and social media apps installed on electronic devices.

[0080] After the electronic device displays the interface of the aforementioned application on the screen, the user can perform swipe gestures on the screen. In response to the user's swipe gestures, the electronic device can switch the content displayed on the screen.

[0081] Take a mobile phone as an example. Figure 1 As shown, the phone's screen displays the interface of a shopping app. The products fully displayed on this interface include headphones, smartwatches, smart glasses, wired headphones, a desk lamp, and a wireless charger. In response to a user-triggered downward swipe, the fully displayed products on the shopping app interface change to routers, smart locks, headphones, smartwatches, smart glasses, and wired headphones. Then, in response to another downward swipe from the user, the fully displayed products on the shopping app interface change to a smart camera, a power strip, a router, a smart lock, headphones, and a smartwatch.

[0082] Currently, during the screen display process of electronic devices, the display generates vertical sync pulse signals at a fixed frequency and sends these signals to the graphics processing unit (GPU). Each time a vertical sync pulse signal is received, the GPU can draw the image to be displayed in the next frame based on drawing instructions. The GPU then sends the drawn image to the display for display. In this way, the GPU can draw the image to be displayed according to the frequency of the vertical sync pulse signal, and the display can also display the image at that frequency, thus achieving the effect of screen switching. The fixed frequency at which the display generates the vertical sync pulse signal is also called the refresh rate.

[0083] In some embodiments, such as Figure 1 As shown in the sliding display process, each frame of the image to be displayed is drawn directly by the GPU; this rendering method is called traditional frame-by-frame rendering. When using traditional frame-by-frame rendering, the phone sequentially displays the images drawn by the GPU frame by frame on the screen, such as... Figure 2As shown, during the user's swipe operation, the phone sequentially displays Fa, Fb, and Fc, which are drawn frame by frame by the GPU.

[0084] Traditional frame-by-frame rendering ensures the accuracy of each frame, but requires high GPU performance to maintain smoothness; otherwise, frame drops are likely to occur. Furthermore, the higher the refresh rate of electronic devices, the more images the GPU needs to render per second, thus placing higher demands on GPU performance.

[0085] For example, at a refresh rate of 30Hz, the GPU needs to render 30 images per second. At a refresh rate of 60Hz, the GPU needs to render 60 images per second.

[0086] The higher the refresh rate of an electronic device, the more images the GPU needs to render per second. Therefore, the electronic device can display more frames per unit of time, resulting in a smoother display of content.

[0087] If users have high requirements for the smoothness of electronic device displays, then the electronic devices they choose will need to be equipped with high-performance GPUs, and correspondingly, the price of such electronic devices will also increase.

[0088] To balance the high demands on the GPU performance of electronic devices with the smoothness of the displayed content, frame interpolation technology can be used to enable electronic devices to achieve high refresh rate display effects.

[0089] Frame interpolation technology refers to the use of algorithms to insert composite frames between images drawn at the original low frame rate, thereby increasing the number of images that can be displayed per unit time and thus improving the smoothness of the display.

[0090] Understandably, for the aforementioned shopping, news, and social media apps, when users switch between displayed content by swiping, there is a significant amount of repetitive content before and after the switch. Therefore, the interpolation frame technique can be used to maintain the smoothness of the interface display in these applications while reducing the number of rendering frames, thereby reducing the demands on GPU performance and the GPU resource consumption during display. Continuing with... Figure 2 When using frame interpolation techniques, for example... Figure 2 As shown in the diagram, Fb and Fc, the GPU can draw only Fa and Fc, and then synthesize Fb based on the already drawn Fa and Fc. Since the image content to be displayed in Fb overlaps with the image content in Fa and Fc, the GPU only needs to extract and stitch the overlapping image content from Fa and Fc to obtain Fb during the process of generating Fb from Fa and Fc.

[0091] In this context, image frames directly drawn by the GPU under the triggering of the vertical synchronization pulse signal can be called original frames, while image frames synthesized by the GPU from two adjacent original frames can be called interpolated frames (or intermediate frames; subsequent embodiments will use interpolated frames as an example). For example, in the above... Figure 2 In this context, Fa and Fc can also be referred to as the original frame, and Fb can also be referred to as the interpolated frame.

[0092] like Figure 2 As shown, the GPU can extract the content of repeated region 201 from Fa and the content of repeated region 202 from Fc. Then, the contents of repeated region 201 and repeated region 202 are concatenated to obtain Fb. Figure 2 As shown, region 201' in Fb has the same image content as the repeating region 201, and region 202' in Fb has the same image content as the repeating region 202.

[0093] This also reduces repetitive drawing tasks on the GPU, lowering its load. Furthermore, the GPU generates frames (Fb) based on Fa and Fc much faster than it redraws them, which also shortens the image display time and reduces the response time from user interaction to screen update.

[0094] However, the display process using the aforementioned frame interpolation technique still suffers from poor responsiveness.

[0095] Before time T1, Fa has already been plotted, so, as Figure 3 As shown, the phone displays Fa at time T1. If the user triggers a swipe operation at time T1, the GPU starts drawing Fc at time T1 and completes the drawing of Fc at time T2. The duration between time T1 and time T2 is called the inter-frame duration. After time T2, the GPU generates Fb based on Fa and Fc, and completes the generation of Fb at time T3. The duration between time T2 and time T3 is the interpolation frame synthesis duration. Thus, after the user triggers a swipe operation at time T1, Fb can not be displayed until at least time T3, and Fc can only be displayed after Fb is displayed. Therefore, the total waiting time from when the user triggers a swipe operation to when the phone displays Fb is at least the sum of the inter-frame duration and the interpolation frame synthesis duration, also known as the touch latency.

[0096] It is possible Figure 3 The results show that after a user triggers a swipe, the user needs to wait for the GPU to complete the rendering of the frame (Fc) and then wait for the GPU to synthesize the frame (Fb) before the image content of the frame (Fb) can be seen on the phone. This process involves a relatively long wait time and exhibits significant latency.

[0097] Based on this, this application provides a display method in which, when an electronic device displays a first image frame, a user can trigger a swipe operation on the first image frame. In response to the user-triggered swipe operation, the electronic device can display a second image frame and a third image frame. The second image frame is an interpolated frame between the first and third image frames. Furthermore, the second image frame is determined based on a fourth image frame already stored in the electronic device before the swipe operation is triggered; therefore, the electronic device does not need to wait for the third image frame to be generated before generating the second image frame. This effectively improves the display speed of the second image frame and reduces response time.

[0098] To better understand the embodiments of this application, the electronic device provided in the embodiments of this application will first be introduced.

[0099] The display method provided in this application embodiment can be applied to electronic device 100. For example, the electronic device may specifically be a mobile phone, tablet computer, smart screen, laptop computer, in-vehicle device, wearable device (such as a smartwatch), ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), artificial intelligence device, or other terminal device with display capabilities. The operating system installed on the electronic device includes, but is not limited to, those specified below. Alternatively, other operating systems may be used. This application does not limit the specific type of electronic device or the operating system installed.

[0100] Figure 4 A schematic diagram of the structure of the electronic device 100 is shown.

[0101] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identity module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0102] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0103] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, GPU, image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0104] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0105] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0106] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0107] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0108] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0109] Figure 5 This is a schematic diagram of the hardware and software structure involved in updating the user interface of the electronic device provided in the embodiments of this application.

[0110] like Figure 5 As shown, the layered architecture divides the hardware and software structure into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces.

[0111] In some embodiments, the hardware and software architecture is divided into four layers, from top to bottom: application layer, system library, kernel layer, and hardware layer.

[0112] The application layer can include a series of applications. These applications can be third-party applications or system applications. When an electronic device displays an application interface, the user can trigger the electronic device to switch the displayed content of the application interface by performing a swipe operation (such as an up swipe, a down swipe, a left swipe, or a right swipe). In this embodiment, there is duplicate content in the displayed content before and after the switch.

[0113] An application can display content on a screen, and all the content displayed on the screen can be called the user interface.

[0114] In some embodiments, some content in the user interface consists of fixed interface elements, while some content consists of variable interface elements. The fixed interface elements remain in the same position on the screen, while the variable interface elements change their position on the screen as the user interacts with them or as the application runs.

[0115] In some examples, such as Figure 2 As shown, the user interface displayed on the mobile phone screen includes Fa, a search area 203, a navigation area 204, and a status bar 205. The interface elements included in Fa are variable interface elements, while the interface elements in the search area and navigation area are fixed interface elements. It should be noted that the "variable interface elements" and "fixed interface elements" in this example are merely examples; the "variable interface elements" and "fixed interface elements" may differ in different user interfaces, and this embodiment does not impose specific limitations.

[0116] The system library may include a module for obtaining the position of interface elements, a module for obtaining motion vectors, a module for aligning image regions, a module for drawing compensation regions, an image generation module, and a surface flinger.

[0117] The interface element position acquisition module is used to obtain the position of interface elements based on the display content update request when the application issues a display content update request.

[0118] The motion vector acquisition module is used to obtain the motion vector of an interface element between two image frames based on the element's position in those frames.

[0119] The image region alignment module is used to extract image content from an image frame. In some examples, the image region alignment module can extract image content from the image frame based on the motion vectors obtained by the motion vector acquisition module. Specifically, the image region alignment module can extract image content from the image frame by copying.

[0120] The compensation region drawing module is used to draw image content based on the position of interface elements. In some examples, the compensation region drawing module can determine the image content to be drawn based on the motion vectors obtained by the motion vector acquisition module, and determine the position of interface elements based on the motion vectors to achieve the drawing of image content.

[0121] The image generation module is used to generate original frames or interpolated frames based on the image content cropped by the image region alignment module and the image content drawn by the compensation region drawing module.

[0122] The surface synthesizer can perform layer synthesis based on the original frames or interpolated frames generated by the image generation module to obtain the final user interface to be displayed.

[0123] Surface synthesizers can generate interpolated or original frames with invariant layer images, such as... Figure 2 In the example shown, the status bar 205, search area 203, navigation area 204, etc., are composited into a user interface to be displayed. The surface compositer can then send this user interface to the display screen for display.

[0124] In some embodiments, the interface element position acquisition module can obtain the positions of all interface elements in the next image frame (or the next original frame) based on the display content update request issued by the application. Then, the interface element position acquisition module sends the positions of all interface elements in the next image frame and the positions of all interface elements in the current display image frame to the motion vector acquisition module. This allows the motion vector acquisition module to calculate the motion vector of any identical interface element between the current display image frame and the next original frame based on the position of any given interface element in the current display image frame and its position in the next original frame. Furthermore, the motion vector acquisition module can also calculate the motion vector of any given interface element between the current display image frame and each interpolated frame between the current display image frame and the next original frame.

[0125] Once the motion vector acquisition module calculates the motion vector of any identical interface element between the current displayed image frame and an interpolated frame between the current displayed image frame and the next original frame, the image region alignment module can extract the interpolated frame from the buffer frame based on the motion vector calculated by the motion vector acquisition module and send the interpolated frame to the surface compositer. The surface compositer can then synthesize the user interface to be displayed based on the interpolated frame and the unchanged layer image.

[0126] Once the motion vector acquisition module calculates the motion vector of any identical interface element between the current display image frame and the next original frame, the image region alignment module can extract a portion of the image content from the buffer frame based on the motion vector calculated by the motion vector acquisition module. The remaining image content is then drawn by the compensation region drawing module. The image generation module determines the next original frame based on the extracted portion and the remaining image content and sends this next original frame to the surface compositer. The surface compositer can then synthesize the user interface to be displayed based on this original frame and an unchanged layer image.

[0127] In other embodiments, the interface element position acquisition module can also acquire the position of the interface element in the displayed image frame. Then, the motion vector acquisition module can predict the position of the interface element in the next image frame based on the position of the interface element in the displayed image frame, and acquire a predicted motion vector based on the predicted position. The compensation region drawing module can draw the image content of the buffer region based on the predicted motion vector, thereby enabling the image generation module to generate a buffer frame based on the image content of the buffer region and the currently displayed original frame.

[0128] In the above embodiments, as an example, the motion vector acquisition module may include a motion vector calculation unit and a motion vector prediction unit. The motion vector calculation unit and the motion vector prediction unit can acquire motion vectors by calculation and prediction, respectively.

[0129] The compensation region rendering module can include a remaining portion rendering unit and an additional buffer rendering unit. The remaining portion rendering unit and the additional buffer rendering unit can render the image content of different parts of the image.

[0130] For ease of understanding, the buffer frame, buffer area, and remaining area in the above embodiments will be described in more detail in the following embodiments.

[0131] The kernel layer may include a display driver. This display driver is typically a software display engine (SDE) driver.

[0132] The display driver can be used to respond to requests to update the user interface, driving the hardware to update and display the user interface.

[0133] The hardware layer can include GPUs and displays.

[0134] The compensation region rendering module can utilize the GPU to perform tasks such as image rendering. The surface synthesizer can transmit the user interface synthesized by the image generation module to the display driver, which can then transmit the synthesized user interface to the display screen for display.

[0135] The following examples use a mobile phone as an example.

[0136] To facilitate understanding, before specifically introducing the methods provided in this application, the following terms appearing in the subsequent embodiments will be explained:

[0137] F1, F2, F3...Fn-1, Fn, Fn+1 refer to the image frames generated by the mobile phone under the triggering of the vertical synchronization pulse signal, which are referred to as the original frames in the following description.

[0138] I1, I2, I3...In-1, In, In+1 refer to the image frames generated by the mobile phone based on the corresponding original frames, which are referred to as buffer frames in the following description.

[0139] B(n, m) refers to the m-th interpolated frame displayed by the phone after displaying Fn.

[0140] Where n and m are both positive integers.

[0141] There is a one-to-one correspondence between the buffered frame and the original frame, that is, I1 corresponds to F1, I2 corresponds to F2, ..., In corresponds to Fn, and In+1 corresponds to Fn+1.

[0142] The image content of the buffer frame In generated by the mobile phone includes the image content of the corresponding original frame Fn. For example... Figure 6 As shown, taking n as a denoted as 'a', Ia is a buffer frame. Not all image content in Ia will be displayed on the phone's screen. Ia includes, for example,... Figure 6 Some of the image content shown will be displayed on the phone's screen. For example... Figure 6 As shown, the portion of image content displayed on the screen of an electronic device in Ia is called the original frame, such as Fa.

[0143] The image content of the buffer frame In generated by the mobile phone also includes the image content of the buffer area. The buffer frame In is used to generate the interpolated frame B(n, m) after the original frame Fn and the next original frame, such as the original frame Fn+1. For ease of understanding, please refer to the relevant embodiments of the display method described later.

[0144] In the above interpolated frame B(n, m), n is the same as n in the image frame Fn. The value range of m is related to the actual refresh rate of the mobile phone and the actual refresh rate that the mobile phone can achieve after using the interpolation frame technology. The maximum value of m is M, where M is a positive integer.

[0145] In some examples, after the phone displays F8, the interpolated frame is B(8, m). The phone's actual refresh rate is 30Hz, and with interpolation, the achievable refresh rate is 60Hz. Therefore, the value of m ranges from [1, 2]. Thus, after displaying F8, the interpolated frames displayed include B(8, 1) and B(8, 2). Similarly, if the phone's actual refresh rate is 30Hz, and with interpolation, the achievable refresh rate is 90Hz, then the value of m ranges from [1, 3]. In this case, after displaying F8, the phone still needs to display interpolated frames B(8, 1), B(8, 2), and B(8, 3).

[0146] For ease of understanding, in the following embodiments, unless otherwise specified, the number of interpolated frames will be 1.

[0147] At a certain moment, the phone has completed the generation of In-1 and displayed Fn-1 on the screen based on In-1. If the user then triggers a swipe operation on the interface, the phone can perform the following steps to generate the image frame that needs to be displayed on the screen:

[0148] S701, In response to a user-triggered swipe operation, generate a request to update the displayed content.

[0149] Since the screen displays Fn-1 when the user triggers the swipe operation, the content update request includes the positions of all interface elements in Fn after Fn-1, as well as the image content that needs to be drawn in Fn.

[0150] Here, Fn-1 refers to the first image frame in this application, B(n-1, 1) refers to the second image frame in this application, Fn refers to the third image frame in this application, and In-1 refers to the fourth image frame in this application. The number of second image frames in this embodiment can be one or more, and the specific number corresponds to the actual number of interpolated frames, as detailed in the following description.

[0151] When a user triggers a swipe operation, the phone can capture the swipe operation and determine the position of all interface elements in Fn and the image content to be drawn in Fn based on parameters such as the swipe direction and swipe distance corresponding to the swipe operation.

[0152] It should be noted that a user can trigger a swipe action in the following ways: Figure 1 The touch-triggered method shown can also be triggered by other methods such as air gestures, voice control, eye tracking, etc.

[0153] In some embodiments, combined with Figure 5 In response to a user-triggered swipe action, such as Figure 8 As shown, a mobile application can generate a content update request. This application refers to the application belonging to the interface currently displayed on the screen. The application then sends the generated content update request to the phone's interface element location acquisition module, enabling the module to determine the positions of all interface elements in Fn based on the content update request.

[0154] For details on how to generate a display content update request based on a user-triggered swipe action, please refer to relevant technologies, which will not be elaborated here.

[0155] S702. Based on the display content update request, determine the motion vector MV_n1 and the motion vector MV_n2_1.

[0156] Here, motion vector MV_n1 is the motion vector of any common interface element between Fn-1 and Fn. Motion vector MV_n2_1 is the motion vector of any common interface element between Fn-1 and interpolated frame B(n-1, 1). The motion vector includes the displacement and direction of the motion.

[0157] Wherein, motion vector MV_n1 refers to the second motion vector in this application, and motion vector MV_n2_1 refers to the first motion vector in this application.

[0158] It should be noted that, since any interface element moves in the same direction and distance between Fn-1 and Fn during the sliding operation, the motion vector MV_n1 and motion vector MV_n2_1 can be determined based on any interface element that exists in both Fn-1 and Fn.

[0159] The content update request includes the positions of all UI elements in frame Fn. Furthermore, the phone stores the positions of all UI elements in the previously displayed original frame Fn-1. Each UI element has a unique identifier within the same application. Therefore, the phone can, based on the identifier of each UI element, filter from the positions of all UI elements in Fn and all UI elements in Fn-1 to determine the position of the same UI element in both Fn and Fn-1.

[0160] For example, if an interface element is located at (x1, y1) in Fn-1 and at (x2, y2) in Fn, then the determined motion vector MV_n1 is (x2-x1, y2-y1).

[0161] Since interpolated frame B(n-1, 1) has not been synthesized when the display content update request is received, the positions of the interface elements in interpolated frame B(n-1, 1) are also unknown. In this embodiment, during the process of the mobile phone responding to the user's swipe operation to perform a swipe display, the time interval between two adjacent original frames Fn-1 and Fn displayed by the mobile phone is short, so the movement of the interface elements between Fn-1 and Fn can be regarded as uniform linear motion. Thus, the motion vector MV_n2_1 of the same interface elements between Fn-1 and interpolated frame B(n-1, 1) can be approximately regarded as half of MV_n1:

[0162]

[0163] It should be noted that in the above embodiment, the number of interpolated frames is 1, and the interpolated frame is B(n-1, 1), so a motion vector MV_n2_1 can be obtained in S702. If the number of interpolated frames is 2, then the interpolated frames should include B(n-1, 1) and B(n-1, 2). In this case, two motion vectors MV_n2_1 and MV_n2_2 should be obtained in S702. MV_n2_1 is the motion vector of any common interface element between Fn-1 and B(n-1, 1), and MV_n2_2 is the motion vector of any common interface element between Fn-1 and B(n-1, 2).

[0164] Similarly, if the number of interpolated frames is M, then in S702, M motion vectors MV_n2_1, MV_n2_2, ..., MV_n2_m should be obtained. Among them, MV_n2_m is the motion vector of any common interface element between Fn-1 and B(n-1, m).

[0165] In some embodiments, when the number of interpolated frames is greater than 1, MV_n2_m can be calculated based on the ratio of the distance the interface element moves from Fn-1 to B(n-1, m) to the distance the interface element moves from Fn-1 to Fn.

[0166] For example, when the number of interpolated frames is 2, the interpolated frames include B(n-1, 1) and B(n-1, 2). The ratio of the distance the interface element moves from Fn-1 to B(n-1, 1) to the distance the interface element moves from Fn-1 to Fn is 1:3, therefore:

[0167]

[0168] The ratio of the distance a UI element travels from Fn-1 to B(n-1, 2) to the distance a UI element travels from Fn-1 to Fn is 2:3, therefore:

[0169]

[0170] For example, when the number of interpolated frames is M, the m-th interpolated frame is B(n-1, m). The ratio of the distance the interface element moves from Fn-1 to B(n-1, m) to the distance the interface element moves from Fn-1 to Fn is m:(M+1), therefore:

[0171]

[0172] In the above embodiments, continue to combine Figure 5 ,like Figure 8As shown, the interface element position acquisition module in the mobile phone can obtain the positions of all interface elements in Fn based on the display content update request. Then, the interface element position acquisition module can send the positions of all interface elements in Fn to the motion vector calculation unit. The motion vector calculation unit can also obtain the positions of the interface elements in Fn-1. Thus, the motion vector calculation unit can determine the motion vectors MV_n1 and MV_n2_1.

[0173] S703. Extract B(n-1, 1) from In-1 based on the motion vector MV_n2_1.

[0174] In subsequent examples, we will use the display screen coordinate system as the origin at the top left corner for illustration. Figure 9 As shown, the top left corner of the display screen is the origin O(0,0), the X-axis is to the right, the Y-axis is downward, and the directions of the X-axis and Y-axis are the directions of pixel increment.

[0175] At this point, the motion vector can be represented as (△x, △y). The absolute value of △x is the distance moved along the X-axis; therefore, a positive △x indicates movement to the right, and a negative △x indicates movement to the left. The absolute value of △y represents the distance moved along the Y-axis; a positive △y indicates movement downwards, and a negative △y indicates movement upwards.

[0176] For example, MV_n2_1 is (0, -2), which means that the same interface element has moved up two units of image content from Fn-1 to B(n-1, 1).

[0177] In some embodiments, the motion direction (referred to as direction 1) and motion displacement (referred to as displacement 1) can be determined based on the motion vector MV_n2_1. Then, the position of the interface element O1 corresponding to the coordinates obtained by moving the origin O by displacement 1 in the opposite direction of direction 1 is used as the starting position for the cutoff. Next, starting from interface element O1 and moving in the opposite direction of direction 1, interface element P1 is found. The distance between interface element P1 and interface element O1 is related to the size of the image frame displayed on the phone. The distance between interface element P1 and interface element O1 is the same as the length of the image frame displayed on the phone in the Y-axis direction. Then, interface element P2, which has the same Y-axis coordinate as interface element P1 and the largest X-axis coordinate value, is found, and the position of interface element P2 can be used as the ending position for the cutoff. Thus, the image content corresponding to the starting position O1 to the ending position P2 in the buffer frame In-1 is the image content of B(n-1, 1).

[0178] Ideally, the distance between interface element P1 and interface element O1 should be the same as the length of the image frame displayed on the mobile phone along the Y-axis. Furthermore, it is necessary to ensure that interface element P1 is within buffer frame In-1. However, since this embodiment guarantees that buffer frame In-1 will always include interface element P1, the alternative case where buffer frame In-1 does not include interface element P1 will not be discussed in detail.

[0179] In the above implementation process, from the perspective of the underlying logic of mobile phone storage, each buffer frame In has a start address and an end address in the phone's storage space. Therefore, when extracting B(n-1, 1) from the buffer frame In-1, the address corresponding to the starting position of the extraction can be obtained as the start address 1, and the address corresponding to the ending position of the extraction can be obtained as the end address 2. In this way, the phone can directly copy the storage content corresponding to the start address 1 to the end address 2 to the storage space used to store B(n-1, 1), thereby realizing the extraction of B(n-1, 1).

[0180] For ease of understanding, we will introduce the concept of the smallest truncation unit, or simply "unit" below. (Combined with...) Figure 9 To provide a more vivid description of the interception process.

[0181] like Figure 9 As shown, buffer frame In-1 is the image already generated by the phone. Taking In-1 as containing ten units of image content, numbered 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 in sequence, as an example. If the image frames displayed on the phone's screen each contain seven units of image content, then Fn-1 within In-1 contains seven units of image content. Figure 9 As shown, the image content of the seven units included in Fn-1 is 1, 2, 3, 4, 5, 6, 7 in sequence.

[0182] After determining that MV_n2_1 is (0, -2), the mobile phone can extract B(n-1, 1) from In-1 based on MV_n2_1. For example... Figure 9 As shown, when the user triggers an upward swipe, the interface elements in In-1 should move upwards. Therefore, the position of interface element O1, corresponding to the coordinates of the origin O in In-1 shifted downwards by 2 units from (0, -2), can be used as the starting position for the cutoff. Next, starting from interface element O1, we find interface element P1 along the Y-axis. Since here... Figure 9The image frame displayed on the phone shown has a Y-axis dimension of 7 units. Therefore, the distance between interface element P1 and interface element O1 is 7 units of image content. Next, find interface element P2 with the same Y-axis coordinate as interface element P1 and the largest X-axis coordinate value, and use the position of interface element P2 as the end position for cropping. Thus, by cropping the image content between the start and end positions, a B(n-1, 1) containing seven units of image content can be captured, as shown. Figure 9 As shown, taking the image content of the seven units included in B(n-1,1) as 3, 4, 5, 6, 7, 8, 9 as an example.

[0183] The following, combined with Figure 10 The steps for extracting B'(n-1, 1) from In-1 based on the motion vector MV_n2_1 when the user triggers a swipe down operation are explained. When the user triggers a swipe down operation, the interface elements in In-1 should move downwards. Therefore, the position of interface element O2 corresponding to the coordinates obtained by moving the origin O in In-1 upwards by 2 units from (0, 2) can be used as the starting position for extraction. Next, starting from interface element O2, interface element P3 is found along the Y-axis. Since... Figure 10 The image frame displayed on the phone shown has a Y-axis dimension of 7 units of image content. Therefore, the distance between interface element P3 and interface element O2 is 7 units of image content. Next, find interface element P4 that has the same Y-axis coordinate as interface element P3 and the largest X-axis coordinate value, and use the position of interface element P4 as the end position for the cropping. Thus, by cropping the image content between the start and end positions, a cropping B'(n-1, 1) containing seven units of image content can be obtained, as shown. Figure 10 As shown, the image content of the seven units included in B'(n-1,1) is 2, 3, 4, 5, 6, 7, 8 in sequence.

[0184] In some embodiments, from the perspective of the underlying logic of mobile phone storage, each buffer frame In has a start address and an end address in the phone's storage space. Therefore, when extracting B(n-1, 1) from the buffer frame In, the address corresponding to the starting position of extraction can be obtained as the start address 1, and the address corresponding to the ending position of extraction can be obtained as the end address 2. In this way, the phone can directly copy the storage content corresponding to the start address 1 to the end address 1 to the storage space used to store B(n-1, 1), thereby realizing the extraction of B(n-1, 1).

[0185] In the above implementation, the position of interface element O1, the position of interface element O3, and the starting address 1 all refer to the first starting position in this application. The position of interface element P2, the position of interface element P4, and the ending address 1 all refer to the first ending position in this application. The first motion vector, including the first displacement and the first direction, refers to the displacement and direction included in MV_n2_1 in this application. The image content corresponding to the first starting position is interface element O1 or interface element O2. The first position is the origin of the fourth image frame in the screen coordinate system. The image content corresponding to the first position in the fourth image frame is interface element O.

[0186] Combination Figure 5 The image region alignment module can extract B(n-1, 1) from In-1 based on the motion vector MV_n2_1. In some examples, combined with Figure 5 ,like Figure 8 As shown, after determining the motion vector MV_n2_1, the motion vector calculation unit sends the motion vector MV_n2_1 to the image region alignment module, so that the image region alignment module can extract B(n-1, 1) from In-1 based on the motion vector MV_n2_1.

[0187] The above embodiments are illustrated using the example where a complete B(n-1, 1) can be extracted from In-1. In other embodiments, if a complete B(n-1, 1) cannot be extracted from In-1, the phone can synthesize B(n-1, 1) based on In-1 and Fn after generating Fn, or synthesize B(n-1, 1) based on Fn-1 and Fn. For example, after the application starts, the I1 corresponding to the initial frame F1 displayed by the phone does not include a buffer area, so the initial frame displayed by the phone only includes seven units of image content. Thus, the phone can first generate F2 based on the display content update request, and then generate B(1, 1) based on F1 and F2. Or, if the sliding distance of the user's swipe operation is large, and a complete B(n-1, 1) cannot be obtained from In-1, then B(n-1, 1) can also be synthesized based on In-1 and Fn after generating Fn, or synthesized based on Fn-1 and Fn.

[0188] S704, Display based on B(n-1, 1).

[0189] After the S703 intercepts B(n-1, 1), the mobile phone can display based on B(n-1, 1).

[0190] In some embodiments, combined with Figure 5 ,like Figure 8As shown, after the image region alignment module generates B(n-1, 1), it can send B(n-1, 1) to the surface compositer. The surface compositer will composite the unchanged layer images, such as the status bar and wallpaper, with B(n-1, 1) to form the user interface to be displayed. Afterwards, the surface compositer can send the user interface to the display screen for display.

[0191] In this way, after the user triggers a swipe, only the motion vector MV_n2_1 needs to be determined based on the content update request. Then, B(n-1, 1) can be directly extracted from In-1 and displayed. This eliminates the need to wait for the GPU to generate Fn. This improves the speed of frame interpolation generation, thereby enhancing responsiveness.

[0192] S705. Determine the reuse region in Fn from In-1 based on the motion vector MV_n1.

[0193] In some embodiments, the motion direction (referred to as direction 2) and motion displacement (referred to as displacement 2) can be determined based on the motion vector MV_n1. Then, the position of interface element O3 corresponding to the coordinates obtained by moving the origin O by displacement 2 in the opposite direction of direction 2 is used as the starting position for the cutoff. Next, starting from interface element O3, interface element P5 is found in the opposite direction of direction 2. Interface element P5 is the interface element in In-1 that is furthest from interface element O3 in the opposite direction of direction 2. After finding interface element P5, interface element P6 with the same Y-axis coordinate and the largest X-axis coordinate value can be found, and the position of interface element P6 can be used as the ending position for the cutoff. Thus, the image content corresponding to the starting position O3 to the ending position P6 in the buffer frame In-1 is the image content of the reused region.

[0194] In practical applications, the distance between interface element P5 and interface element O3 is less than or equal to the length of the image frame displayed on the mobile phone along the Y-axis. Furthermore, interface element P5 is located in In-1.

[0195] In some embodiments, if interface element P' is found by starting from interface element O3 and moving in the opposite direction of direction 2.

[0196] If the distance between interface element P' and interface element O3 is the same as the length of the image frame displayed on the phone in the Y-axis direction, and interface element P' is not included in In-1, then the interface element in In-1 that is farthest from interface element O3 in the opposite direction of direction 2 can be taken as the actual P5, and P6 can be determined based on P5, such as... Figure 9 As shown. At this point, the reused region in Fn determined from In-1 based on the motion vector MV_n1 is only a part of Fn.

[0197] If the distance between interface element P' and interface element O3 is the same as the length of the image frame displayed on the phone in the Y-axis direction, then interface element P' is included in In-1. In this case, the interface element in In-1 that is furthest from interface element O3 in the opposite direction of direction 2 can be taken as the actual P5, and P6 can be determined based on P5. In this situation, the captured reused area can be directly used as Fn, and subsequent steps S706 and S707 do not need to be executed.

[0198] like Figure 9 As shown, taking buffer frame In-1 as containing ten units of image content, numbered 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 in sequence, as an example. After determining that MV_n1 is (0, -4), as... Figure 9 As shown, when the user triggers an upward swipe, the interface elements in In-1 should move upwards. Therefore, the origin O in In-1 can be moved downwards along the coordinates (0, -4) to a position O3, which is the starting position for the capture. Next, starting from interface element O3, we find interface element P5 along the Y-axis. Figure 9 The image frame displayed on the phone shown has a Y-axis dimension of 7 units. Therefore, the distance between interface element P5 and interface element O3 is 6 units of image content. Next, find interface element P6 that has the same Y-axis coordinate as interface element P5 and the largest X-axis coordinate value, and use the position of interface element P6 as the end position for cropping. In this way, by cropping the image content between the start and end positions, a reused area of ​​six units can be captured, such as... Figure 9 As shown, the image content of the six units included in the multiplexing region in Fn are 5, 6, 7, 8, 9, and 10, respectively.

[0199] The following, combined with Figure 10 The steps for extracting a reused region from In-1 based on the motion vector MV_n1 when the user triggers a swipe down operation are explained. When the user triggers a swipe down operation, the interface elements in In-1 should move downwards. Therefore, the coordinates of the origin O in In-1 can be moved upwards by 4 units from (0, 4). However, since the interface element corresponding to the coordinates obtained by moving the origin O upwards by 4 units from (0, 4) is not included in In-1, the position of the interface element O4, which is the furthest distance that the origin O can reach by moving its coordinates upwards, can be used as the starting position for extraction. Next, starting from interface element O4, the interface element P7 is found along the Y-axis. Since... Figure 10The image frame displayed on the phone shown has a Y-axis dimension of 7 units. Therefore, the distance between interface element P7 and interface element O4 is 6 units of image content. Next, find interface element P8 that has the same Y-axis coordinate as interface element P7 and the largest X-axis coordinate value, and use the position of interface element P8 as the end position for cropping. In this way, by cropping the image content between the start and end positions, a 6-unit area of ​​image content can be captured as a reused region, such as... Figure 10 As shown, taking the image content of 6 units included in the reused area in Fn' as an example, which are 1, 2, 3, 4, 5, and 6 respectively.

[0200] In some embodiments, from the perspective of the underlying logic of mobile phone storage, each buffer frame In has a start address and an end address in the phone's storage space. Therefore, when extracting a reused region from the buffer frame In, the address corresponding to the starting position of extraction can be obtained as the start address 2, and the address corresponding to the ending position of extraction can be obtained as the end address 2. In this way, the phone can directly copy the storage content corresponding to the start address 2 to the end address 2 to the storage space used to store the image content of the reused region, thereby realizing the extraction of the image content of the reused region.

[0201] In the above implementation, the position of interface element O3, the position of interface element O4, and the starting address 2 all refer to the second starting position in this application. The position of interface element P6, the position of interface element P8, and the ending address 2 all refer to the second ending position in this application. The second motion vector, including the second displacement and the second direction, refers to the displacement 2 and direction 2 included in MV_n2_1 in this application. The image content corresponding to the second starting position is interface element O3 or interface element O4. The first position is the origin of the fourth image frame in the screen coordinate system. The image content corresponding to the first position in the fourth image frame is interface element O.

[0202] In the above Figure 9 In the embodiment shown, O3 is the interface element corresponding to the second position of this application. Figure 10 In the embodiment shown, the interface element corresponding to the second position of this application is not drawn. It is located in the opposite direction of the Y-axis of interface element O, and the distance between it and interface element O is 4.

[0203] At this point, some image content is missing in Fn. The following step, S706, can be executed to complete the missing image content in Fn:

[0204] S706. Determine the remaining areas in Fn that need to be redrawn based on the reused areas in Fn.

[0205] In some embodiments, the mobile phone can determine the remaining areas in Fn that need to be redrawn based on the reused areas reused in Fn. The image content that should be displayed in Fn is known, so after determining the reused areas, the missing image content in the known image content can be determined based on the known image content and the reused areas.

[0206] As in the embodiment of S705 above. Figure 9 It has been determined that the reused area in Fn includes six units of image content 5, 6, 7, 8, 9, and 10. Furthermore, it is known that Fn should include seven units of image content. Therefore, it can be determined that Fn is missing one unit of image content. This missing unit of image content needs to be drawn by the mobile phone. Figure 9 As shown, the remaining area in Fn is image content 11 that is not in In-1, so the mobile phone needs to draw this image content 11.

[0207] As in the embodiment of S705 above. Figure 10 The multiplexed region in Fn has been determined to include six units of image content: 1, 2, 3, 4, 5, and 6. Therefore, it can be determined that Fn is missing one unit of image content. Figure 10 As shown, the remaining area in Fn is image content 0, which is not present in In-1. Therefore, the mobile phone needs to draw this image content 0.

[0208] S707. Draw the image content of the remaining area.

[0209] In some embodiments, since the display content update request includes the positions of all interface elements in Fn and the image content to be drawn in Fn, the positions of all interface elements in the remaining area and the image content to be drawn are known. Therefore, the mobile phone can draw the remaining area based on the display content update request.

[0210] Combination Figure 5 ,like Figure 8 As shown, the remaining drawing unit can determine the reused region in Fn from In-1 based on the motion vector MV_n1. Then, based on the reused region in Fn, the remaining regions in Fn that need to be redrawn are determined. Finally, the image content of the remaining regions is drawn.

[0211] S708. Generate and display Fn based on the image content of the reused area and the image content of the remaining area.

[0212] At this point, the electronic device can determine Fn based on the reused area determined in S705 and the remaining area drawn in S707. After determining Fn, the mobile phone can display Fn on the display screen.

[0213] In the above embodiments, the image content of the reused area refers to the second image content in this application, and the image content of the remaining area refers to the third image content in this application.

[0214] In this process, the mobile phone reuses part of the image content from In-1 to Fn. Only part of the content needs to be drawn by the mobile phone. Compared with the current interpolation frame technology, it is not necessary to draw Fn from beginning to end. This not only improves the generation speed of Fn, but also reduces the drawing load of the mobile phone. Therefore, the method provided by the embodiments of this application can improve the display performance of the mobile phone.

[0215] It should be noted that, although Figure 2 The Fa and Fc shown are called the original frames. Figure 7 In the relevant embodiments, Fn is also referred to as the original frame, but the two are actually different. Specifically, in the introduction as follows... Figure 2 as well as Figure 3 In the relevant embodiments, all contents in Fa and Fc are directly drawn by the GPU. As described in the embodiments related to S705-S707 above, only the image content of the remaining area in Fn is drawn by the GPU, and the image content of the reused area is directly reused from In-1.

[0216] After the mobile phone determines the motion vector MV_n1 in S702, In can be determined through the following S709-S711, which is used to determine Fn+1 to be displayed after Fn is determined, as well as B(n+1, m):

[0217] S709. Based on the positions of all interface elements in a series of consecutive original frames that have been displayed, predict the predicted motion vector MV_n_f of the same interface elements between Fn and B(n,1).

[0218] The number of consecutive original frames can be predetermined by the developers; for example, it can be 5 or 3.

[0219] In some embodiments, the phone stores the positions of all UI elements in the displayed original frame. In some examples, due to a user-triggered swipe, the phone can generate a display content update request, which includes the positions of all UI elements in the next original frame. Therefore, after obtaining the positions of all UI elements in the original frame based on the display content update request, the phone can store the positions of all UI elements in the original frame in the phone's memory.

[0220] The mobile phone can predict the predicted motion vector MV_n_f of the same interface elements between Fn and B(n,1) based on the position of the interface elements in these displayed original frames.

[0221] This can be achieved by using least-squares fitting to predict the positions of interface elements in the interpolated frames based on the positions of interface elements in the already displayed original frames. For example, ... Figure 11 As shown, the positions of the interface elements in Fn-4, Fn-3, Fn-2, Fn-1, and Fn are known. Therefore, we can take the frame order n-4, n-3, n-2, n-1, and n as the independent variable x, and the positions of the interface elements in the original frames as the dependent variable f(x), and express them as a quadratic curve:

[0222] f(x) = ax 2 +bx+c

[0223] Where a, b, and c represent unknown coefficients.

[0224] The error in the least squares fitting process is defined as:

[0225] ε(x i )=D[x i ]-f(x i |i∈0,1,…,n

[0226] Wherein, D[x i [] represents the position of the interface element in the original frame that has been displayed, that is, the position of the interface element in Fn-4, Fn-3, Fn-2, Fn-1, Fn.

[0227] When the optimization objective is to minimize the sum of squared errors, i.e.:

[0228]

[0229] The specific values ​​of a, b, and c can be obtained.

[0230] At this point, the quadratic curve is a known curve. Therefore, based on the known curve, the possible positions of the interface elements in B(n, 1) can be deduced. For example... Figure 11 As shown by point p in the diagram, the possible positions of the interface element in the first interpolated frame B(n, 1) after Fn are:

[0231]

[0232] Where m represents the sequence number of the inserted frame, and M represents the number of frames inserted between the two original frames.

[0233] Thus, given the position of the interface element in Fn and the predicted position of the interface element in the interpolated frame B(n, 1), the predicted motion vector MV_n_f can be obtained. For the specific determination method, please refer to the method for determining the motion vector MV_n1 in the aforementioned embodiments related to S702.

[0234] Combination Figure 5 ,like Figure 8 As shown, the motion vector prediction unit can obtain the position of the interface element in each original frame required for prediction from the interface element position acquisition module, and then predict the predicted motion vector MV_n_f of the same interface element between Fn and B(n,1) based on the position of any interface element among multiple consecutive displayed original frames.

[0235] S710. Draw the image content of the buffer region based on the predicted motion vector MV_n_f.

[0236] After obtaining the predicted motion vector MV_n_f, the drawing height of the buffer area image content can be determined so that the buffer area image content can include the positions where interface elements may move. Then, the image content of the buffer area can be drawn based on the positions where interface elements may move.

[0237] The image content in the buffer area refers to the first image content in this application. The image content in the buffer area is continuous with the image content in the original frame. Since the image content in the buffer area is not displayed on the screen, it can be referred to as overflow content that is continuous with the image content of the original frame.

[0238] In this case, the overflow direction of the image content in the buffer area is opposite to the direction of the swipe operation triggered by the user on the original frame before Fn-1 is displayed on the electronic device. For example, as Figure 9 As shown, when the user swipes up (opposite direction of the Y-axis), the overflow direction of the image content in the buffer area is downward (positive direction of the Y-axis). Figure 10 As shown, when the user swipes down (positive Y-axis direction), the overflow direction of the image content in the buffer area is upward (opposite Y-axis direction).

[0239] Combination Figure 5 ,like Figure 8 As shown, the additional buffer rendering unit can obtain the predicted motion vector MV_n_f from the motion vector prediction unit and render the image content of the buffer area based on the predicted motion vector MV_n_f.

[0240] In some embodiments, the height of the buffer region image content can be determined as:

[0241]

[0242] Where k represents the redundancy coefficient.

[0243] The redundancy factor ensures that the height of the image content in the buffer region can cover the motion distance of the subsequent interpolated frames B(n, m) of the current frame Fn. The redundancy factor is related to the curve fitting error, the predicted motion vector size, and the amount of data used for prediction. Developers can select an empirical value as the redundancy factor based on multiple experimental data. For example, an empirical value of 1.6 can be used as the redundancy factor. Generally, the redundancy factor can range from 1 to 2.

[0244] S711, Determine the image content based on Fn and the buffer area.

[0245] like Figure 12 As shown, Fn and the image content of the buffer area are synthesized to obtain In. Thus, during subsequent sliding display, the mobile phone can extract the interpolated frame B(n, 1) after obtaining Fn from In.

[0246] The above embodiments described the process of generating In. The process of generating In-1 is the same as that of generating In, and will not be elaborated further in this embodiment. Figure 12 As shown, In includes Fn and the image content of the buffer area. Here, if Fn is regarded as the first image frame in this application, then the image content of the buffer area is the first image content in this application, and In is the fourth image frame in this application.

[0247] In the above process, the use of a buffer area can improve the speed of frame interpolation and the generation of original frames, thus improving the display speed of the mobile phone and thereby improving the response speed of the mobile phone.

[0248] In some scenarios, after a user triggers a swipe gesture and their hand is removed from the phone's display, the phone can continue to show the swipe animation. In this case, the phone generates the image frames to be displayed after the user's hand is separated from the screen, based on a pre-configured motion curve profile.

[0249] For example, at a certain moment, the phone has completed the generation of In-1 and displays Fn-1 on the screen based on In-1. If the user triggers a swipe operation on the interface and at that moment the user's hand is separated from the phone's screen, then the phone can determine the position of the interface element in Fn and the position of the interface element in B(n-1, 1) based on the pre-configured motion curve configuration file, thereby determining the motion vector MV_n1 and motion vector MV_n2_1. Thus, it can be used to... Figure 7 In the relevant embodiments, Fn and B(n-1, 1) are determined in the same manner as in S703-S708.

[0250] Afterwards, the mobile phone can also determine In through the following Sp01-Sp03, which is used to determine Fn+1 to be displayed after Fn, as well as B(n,m).

[0251] Sp01. Determine the motion vector MV_n_a of any identical interface element between Fn and B(n,1) from the motion curve configuration file.

[0252] The motion curve configuration file is pre-configured in the phone and is used to represent the relationship between the movement speed of interface elements and time. After Fn, B(n,1) is displayed according to the motion curve configuration file. Therefore, the phone can determine the motion vector MV_n_a of any identical element between Fn and B(n,1) based on the motion curve configuration file.

[0253] In some embodiments, the motion curve configuration file includes motion curves representing the velocity-time relationship of interface element movement. When the user triggers a swipe operation and their hand separates from the phone's display, the phone can obtain the initial velocity of the user's hand at the moment of separation. Then, the phone integrates the initial velocity and the motion curve to obtain the displacement in the motion vector. Furthermore, the phone can predict the direction of motion in the motion vector based on the direction of the user's swipe operation. Thus, after obtaining the displacement and direction of motion, it is possible to predict the motion vector MV_n_a of the same interface elements between Fn and B(n, 1).

[0254] Since the motion curve configuration file is pre-configured, it indicates the speed at which interface elements move over time during the sliding display process. Given that the movement time of any identical interface element between Fn and B(n, 1) is determined, the motion vector MV_n_a of any identical interface element between Fn and B(n, 1) can be determined based on the motion curve configuration file.

[0255] Sp02, Draw the image content of the buffer area based on the motion vector MV_n_a.

[0256] Sp03, based on the determined Fn and the buffer region, determines In.

[0257] In the above embodiment, since the motion vector MV_n_a is the actual motion vector of any identical interface element between Fn and B(n, 1), the image content of the buffer area drawn by the mobile phone through Sp02 is exactly the image content of the 10th unit missing from B(n, 1) compared to Fn, such as... Figure 13As shown. Therefore, after determining In based on Sp01-Sp03 above, the mobile phone can extract B(n, 1) from In for display, and In will not include any redundant images. In this way, the performance waste problem in the process of drawing In can be avoided.

[0258] In other embodiments, the mobile phone may also be configured with a fixed predicted motion vector MV_n_a. In this way, if the predicted motion vector MV_n_a cannot be predicted based on the method in the above embodiments, the mobile phone can directly draw the image content of the buffer area based on the fixed predicted motion vector MV_n_a.

[0259] In some examples, when the number of raw frames already displayed on the phone is less than a preset threshold, the phone can directly draw the image content of the buffer area based on a predicted motion vector MV_n_a with a fixed value. For example, after the application starts, the initial frame F1 displayed by the phone does not include the buffer area in I1. In this case, the number of raw frames already displayed is too small to draw the image content of the buffer area based on a predicted motion vector MV_n_a with a fixed value. Figure 7 The embodiment shown provides a method for determining the predicted motion vector MV_n_a.

[0260] In other examples, even if the phone does not have a corresponding motion curve file configured, the phone can directly draw the image content of the buffer area based on the predicted motion vector MV_n_a with a fixed value to ensure the speed of subsequent frame interpolation and original frame generation.

[0261] It should be noted that in the display method provided in this application embodiment, the sliding operation triggered by the user is a sliding operation in the same direction.

[0262] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform the various functions or steps described in the method embodiments.

[0263] This application also provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the various functions or steps described in the above method embodiments.

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0265] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0266] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0267] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0268] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0269] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A display method, characterized in that, Applied to electronic devices, the method includes: Display the first image frame; In response to a user's swipe operation on the first image frame, a second image frame and a third image frame are displayed. The swipe operation is used to trigger the display of the third image frame. The second image frame is an interpolated frame between the first image frame and the third image frame. The second image frame is generated based on the fourth image frame, which is stored in the electronic device before receiving the sliding operation. The fourth image frame includes at least the first image frame and first image content, where the first image content is overflow content that is continuous with the image content of the first image frame.

2. The method according to claim 1, characterized in that, The third image frame includes the second image content, which is generated based on the fourth image frame.

3. The method according to any one of claims 1-2, characterized in that, Before displaying the second image frame, the method further includes: In response to the sliding operation, the first motion vector of the same interface element between the first image frame and the second image frame is obtained; The second image frame is extracted from the fourth image frame based on the first motion vector.

4. The method according to claim 3, characterized in that, Extracting the second image frame from the fourth image frame based on the first motion vector includes: The first starting position and the first ending position are determined based on the first motion vector; The image content from the first start position to the first end position in the fourth image frame is extracted as the second image frame; The first motion vector includes a first displacement and a first direction. The distance between the image content corresponding to the first starting position and the image content corresponding to the first position in the fourth image frame is equal to the first displacement. The direction from the first position to the first starting position is opposite to the first direction. The first position is the origin of the fourth image frame in the screen coordinate system. The distance between the first ending position and the first starting position is related to the size of the image frame displayed by the electronic device.

5. The method according to claim 4, characterized in that, The step of extracting the image content from the first start position to the first end position in the fourth image frame as the second image frame includes: The image content from the first start position to the first end position in the fourth image frame is copied to the storage space of the second image frame to obtain the second image frame.

6. The method according to any one of claims 3-5, characterized in that, Before displaying the third image frame, the method further includes: In response to the sliding operation, a second motion vector of the same interface element between the first image frame and the third image frame is obtained; The second image content is extracted from the fourth image frame based on the second motion vector; The third image frame is generated based on the content of the second image.

7. The method according to claim 6, characterized in that, The step of obtaining a second motion vector of the same interface element between the first image frame and the third image frame in response to the sliding operation includes: In response to the sliding operation, obtain all interface elements that need to be included in the third image frame indicated by the sliding operation, as well as the position of each interface element; Obtain all interface elements in the first image frame and the position of each interface element; The second motion vector is determined based on the position of the target interface element in the third image frame and the position of the target interface element in the first image frame, wherein the target interface element is any one of the same interface elements in the third image frame and the first image frame.

8. The method according to claim 6 or 7, characterized in that, The step of extracting the second image content from the fourth image frame based on the second motion vector includes: The second starting position and the second ending position are determined based on the second motion vector; The image content from the second start position to the second end position in the fourth image frame is extracted as the second image content; The second motion vector includes a second displacement and a second direction; The distance between the image content corresponding to the second starting position and the image content corresponding to the first position in the fourth image frame is equal to the second displacement, and the direction from the first position to the second starting position is opposite to the second direction; the first position is the origin of the fourth image frame in the screen coordinate system; the distance between the second ending position and the second starting position is related to the size of the image frame displayed by the electronic device; Alternatively, the distance between the image content corresponding to the second starting position and the image content corresponding to the first position in the fourth image frame is less than the second displacement. The second starting position is the position where the fourth image frame is at its minimum horizontal and vertical coordinate values ​​in the screen coordinate system. The direction from the first position to the second starting position is opposite to the second direction. The distance between the second ending position and the second position is related to the size of the image frame displayed by the electronic device. The second position is the position obtained by moving the first position according to the second motion vector in the screen coordinate system.

9. The method according to any one of claims 5-8, characterized in that, When the size of the second image content is smaller than the size of the image frame displayed by the electronic device, the method further includes: Generate third image content; the interface elements to be included in the third image content are not included in the fourth image frame; The process of generating the third image frame based on the content of the second image includes: The third image frame is generated based on the content of the second image and the content of the third image.

10. The method according to claim 9, characterized in that, The generation of the third image content includes: Based on the second image content, the third image content is generated by including all interface elements that need to be included in the third image frame as indicated by the sliding operation, as well as the position of each interface element.

11. The method according to any one of claims 1-10, characterized in that, After the first image frame is drawn and before the sliding operation is received, the method further includes: Obtain a third motion vector, which is the predicted motion vector of the same interface element between the first image frame and the second image frame; The content of the first image is drawn based on the third motion vector; The fourth image frame is generated based on the content of the first image and the first image frame.

12. An electronic device, characterized in that, The electronic device includes a display screen, a memory, and one or more processors; the display screen, the memory, and the processors are coupled; the display screen is used to display an image generated by the processor, the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-11.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1-11.

Images