Image processing method, device, computer equipment and storage medium

Abstract

Embodiments of the present application disclose image processing method, device, computer equipment and storage medium, the image processing method comprises: rendering UI image frame to the frame buffer, put video image frame into the cache queue;According to the timestamp of the video image frame, generate image frame control result;According to the image frame control result, determine the video image frame put into the image additional layer;According to the UI image frame in the frame buffer, the video image frame in the image additional layer, generate display image frame. By setting the image additional layer and the frame buffer in the same level, parallel processing UI image frame and video image frame, and displaying video image frame according to image frame control result, effectively avoid the problem that video data processing lags behind caused by single thread processing, which leads to the problem that video image frame cannot be rendered and displayed within the required display time, so as to realize high frame rate image video display effect and improve user experience.

Description

Technical Field

[0001] The present invention relates to the field of image display, and in particular to an image processing method, device, computer equipment and storage medium. Background Art

[0002] As the performance of computer devices continues to improve, more and more applications provide the function of playing images and videos, so as to display playable images and videos according to received operation instructions. Most current applications are developed based on native frameworks and use native frameworks to play images and videos.

[0003] When an application plays an image or video, it needs to render the video frames of the image or video in a thread, and also needs to render the UI (User Interface) frames in the thread to achieve image refresh of the video. However, many computer devices, such as those based on the Android system, use a single-threaded native framework to render and display images and videos. For high-frame-rate playback scenarios, affected by factors such as UI graphics rendering, when a single thread processes multiple rendering instructions, it will cause thread jams and image rendering lags, which in turn makes it impossible to obtain high-frame-rate image and video display effects. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to overcome the deficiencies in the prior art and to provide an image processing method, apparatus, computer equipment and storage medium to solve the problem that the native framework cannot obtain high frame rate image and video display effects.

[0005] In a first aspect, the present application provides an image processing method, applied to an image processing device including an image additional layer and a frame buffer, the method comprising:

[0006] Rendering the UI image frame to the frame buffer, and placing the video image frame into a buffer queue, wherein the video image frame includes a timestamp;

[0007] generating an image frame control result according to the timestamp of the video image frame;

[0008] Determining, according to the image frame control result, a video image frame to be placed in the image additional layer;

[0009] A display image frame is generated according to the UI image frame in the frame buffer and the video image frame in the image additional layer.

[0010] With reference to the first aspect, in a first possible implementation manner, the image frame control result includes deleting a frame, repeating a frame, and updating a frame, and determining, based on the image frame control result, the video image frame to be placed in the image additional layer includes:

[0011] If the image frame control result is to delete the frame, remove the head video image frame of the cache queue, and repeat the step of generating the image frame control result according to the timestamp of the video image frame;

[0012] If the image frame control result is to update the video image frame, the head video image frame of the cache queue is placed in the image additional layer.

[0013] In combination with the first aspect, in a second possible implementation manner, generating an image frame control result according to the timestamp of the video image frame includes:

[0014] Get the current display resolution and the first time of the current display;

[0015] An image frame control result is generated according to the timestamp of the video image frame, the display resolution and the first time.

[0016] With reference to the first aspect, in a third possible implementation, placing the video image frame into a cache queue includes:

[0017] Determining whether the cache queue overflows;

[0018] If the cache queue overflows, discarding the video image frame;

[0019] If the cache queue does not overflow, the video image frame is placed into the cache queue.

[0020] With reference to the first aspect, in a fourth possible implementation, placing the video image frame into a cache queue includes:

[0021] The display order of the video image frames is determined according to the time stamps of the video image frames, and the video image frames are sequentially placed into a cache queue.

[0022] In combination with the first aspect, in a fifth possible implementation manner, before rendering the UI image frame to the frame buffer and placing the video image frame into the buffer queue, the method further includes:

[0023] Determine the type of image data obtained;

[0024] If the acquired image data is UI data, encapsulating the UI data into a UI image frame;

[0025] If the acquired image data is video data, a video image frame including a timestamp is generated according to the video data and the acquired second time.

[0026] With reference to the first aspect, in a sixth possible implementation manner, generating an image frame control result according to the timestamp of the video image frame includes:

[0027] Querying whether the buffer queue contains a video image frame;

[0028] If the buffer queue contains a video image frame, an image frame control result is generated according to the timestamp of the video image frame.

[0029] In a second aspect, the present application provides an image processing device, the image processing device comprising an image additional layer and a frame buffer, the device further comprising:

[0030] A frame rendering module, configured to render UI image frames to the frame buffer and place video image frames into a buffer queue, wherein the video image frames include a timestamp;

[0031] A frame control module, configured to generate an image frame control result according to a timestamp of the video image frame;

[0032] A frame processing module, configured to determine the video image frame to be placed in the image additional layer according to the image frame control result, and update the image additional layer;

[0033] The frame display module is used to generate a display image frame according to the UI image frame in the frame buffer and the video image frame in the image additional layer.

[0034] In a third aspect, the present application provides a computer device comprising a processor and a memory, wherein the memory stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the image processing method of the first aspect are implemented.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium having a program or instruction stored thereon, which, when executed by a processor, implements the steps of the image processing method of the first aspect.

[0036] The present application provides an image processing method, which is applied to an image processing device including an image attachment layer and a frame buffer. The method includes: rendering a UI image frame to the frame buffer and placing a video image frame into a cache queue, wherein the video image frame includes a timestamp; generating an image frame control result based on the timestamp of the video image frame; determining the video image frame to be placed into the image attachment layer based on the image frame control result; and generating a display image frame based on the UI image frame in the frame buffer and the video image frame in the image attachment layer. By setting an image attachment layer and a frame buffer at the same level, processing the UI image frame and the video image frame in parallel, and displaying the video image frame based on the image frame control result, the problem of video data processing lag caused by single-threaded processing, which results in the video image frame not being able to be rendered and displayed within the required display time, is effectively avoided, thereby achieving a high frame rate image and video display effect and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0038] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention is shown;

[0039] Figure 2 A flowchart showing image display of a single-threaded native framework is shown;

[0040] Figure 3 shows a flowchart of image display provided by an embodiment of the present invention;

[0041] Figure 4 A schematic structural diagram of an image processing device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0043] The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the figures is not intended to limit the scope of the claimed invention, but rather merely represents selected embodiments of the present invention. All other embodiments derived by those skilled in the art based on the embodiments of the present invention without inventive effort are intended to be within the scope of protection of the present invention.

[0044] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or the possibility of adding one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0045] Furthermore, the terms “first,” “second,” “third,” etc., are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0046] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present invention pertain. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present invention.

[0047] Example 1

[0048] See also Figure 1 , Figure 1 The flowchart of the image processing method provided by the embodiment of the present invention is shown. Figure 1 The image processing method is applied to a computer device including an image additional layer, the computer device including an image additional layer and a frame buffer, and the image processing method includes the following steps:

[0049] Step 110: Render the UI image frame to the frame buffer, and put the video image frame into the buffer queue, wherein the video image frame includes a timestamp.

[0050] Please also refer to Figure 2 , Figure 2 The following is a flowchart of image display in a single-threaded native framework. For example, the single-threaded native framework of the Android system is used as an example. The native framework of the Android system renders and displays images through a single thread. The UI image frame and the video image frame are both in one thread and are rendered to the frame buffer in sequence. The display device obtains the image to be displayed by refreshing the frame buffer. In high-frame-rate playback scenarios, the processing of video image frames is affected by factors such as UI graphics rendering.

[0051] In this application example, the image additional layer and the frame cache are at the same level to provide parallel processing asynchronous process, effectively avoiding thread jamming and image rendering lag. Among them, the image additional layer can be implemented by a memory, or it can be any device for storing video image frames, which is not limited here.

[0052] As an example, placing the video image frame into the cache queue includes:

[0053] The display order of the video image frames is determined according to the time stamps of the video image frames, and the video image frames are sequentially placed into a cache queue.

[0054] It should be understood that the timestamp is used to mark the video image frame, indicating that the video image frame existed at a specific point in time and its integrity can be verified. According to the timestamps of the video image frames, the obtained video image frames are sorted to obtain the display order of the video image frames. According to the obtained display order, the video image frames are sequentially placed in the cache queue. Specifically, the video image frame with the earliest timestamp is placed at the head of the cache queue, and the video image frame with the latest timestamp is placed at the end of the cache queue. The video image frames are displayed using the timestamps so that the display device displays the video image frames in order, obtaining an image displayed at a high frame rate.

[0055] As an example, before rendering the UI image frame to the frame buffer and placing the video image frame into the buffer queue, the process further includes:

[0056] Determine the type of image data obtained;

[0057] If the acquired image data is UI data, encapsulating the UI data into a UI image frame;

[0058] If the acquired image data is video data, a video image frame including a timestamp is generated according to the video data and the acquired second time.

[0059] UI data is the overall design data for the application's human-computer interaction, operational logic, and aesthetically pleasing interface. Users can provide UI instructions through their computer devices. Video data can be obtained through a decoder and can be composed of multiple image data, without limitation.

[0060] Please also refer to Figure 3 , Figure 3A flowchart of image display provided by an embodiment of the present invention is shown. The present application first decodes the original image data through a decoder to obtain UI data and video data, encapsulates the UI data into UI image frames through a first thread, and renders the UI image frames to the frame cache in sequence. The display order of multiple image data in the video data is obtained through a second thread, the order of the generated video image frames is determined, and according to the order of the video image frames, the video data is encapsulated into video image frames in sequence, and the timestamps of the video image frames are set according to the obtained second time to obtain video image frames including timestamps, and the video image frames including timestamps are placed into the cache queue in sequence. The asynchronous processing of UI images and video images is completed by the parallel processing of the first thread and the second thread, which effectively avoids the problem of video data processing lag caused by single-thread processing, resulting in the video image frame not being able to complete the rendering display within the required display time, thereby obtaining a high frame rate image and video display effect and improving the user experience.

[0061] It should be understood that the second time is the current time of encapsulating the video data into the video image frame. As an example, the step of placing the video image frame into the cache queue includes:

[0062] Determine whether the frame buffer queue (Frame Buffer Queue) overflows;

[0063] If the cache queue overflows, discarding the video image frame;

[0064] If the cache queue does not overflow, the video image frame is placed into the cache queue.

[0065] After obtaining the video image frame including the timestamp based on the video data and the current time obtained, determine whether the cache queue overflows. If the cache queue overflows, it is determined that the cache queue cannot store more video image frames, the video image frames are discarded, and the video data is obtained again through the decoder. If the cache queue does not overflow, the video image frame is placed in the cache queue. It should be understood that the cache queue is used to store video image frames, and according to the first-in-first-out principle, the video image frames in the cache queue are placed in the image additional layer in turn to ensure the synchronous display of the UI image frame and the video image frame.

[0066] Step 120: Generate an image frame control result according to the timestamp of the video image frame.

[0067] According to the timestamp of the video image frame, a video image frame to be displayed is determined, and an image frame control result is generated. According to the image frame control result, a refresh rate of the video image frame of the display device is determined, so as to ensure the synchronous display of the UI image frame and the video image frame. The display device can be any device for displaying images arranged on the computer device, and the image frame control result includes a delete frame, a repeat frame and an update frame, which are not limited herein. According to the timestamp of the video image frame, the serial number of the video image frame at the head of the cache queue is determined. According to the serial number of the video image frame at the head, the image frame control result is generated. Specifically, taking the video image frame with the serial number of the second frame to be displayed by the display device as an example. If the serial number of the video image frame at the head of the cache queue is the first frame, the generated image frame control result is a delete frame. If the serial number of the video image frame at the head of the cache queue is the second frame, the generated image frame control result is an update frame. If the serial number of the video image frame at the head of the cache queue is the third frame, the generated image frame control result is a repeat frame.

[0068] As an example, the generating the image frame control result according to the timestamp of the video image frame comprises:

[0069] obtaining a current display resolution and a first time of current display;

[0070] generating the image frame control result according to the timestamp of the video image frame, the display resolution and the first time.

[0071] It should be understood that the first time is the current time of displaying the image by the display device. For a high-resolution application scenario, a frame rate of 48HZ or more is usually required. Specifically, if the obtained current display resolution is 1920*1080P50HZ, it is confirmed that the frame rate is 50HZ. Therefore, it is confirmed that the display device needs to display 50 frames of video image frames per second, i.e. it is required to refresh the video image frame once every 20 milliseconds. Specifically, when the time is greater than or equal to 0 milliseconds and less than 20 milliseconds, the display device needs to display the video image frame with the serial number of the first frame; when the time is greater than or equal to 20 milliseconds and less than 40 milliseconds, the display device needs to display the video image frame with the serial number of the second frame. The display device refreshes the video image frame every 20 milliseconds, thereby realizing the display effect with the frame rate of 50HZ.

[0072] Taking the time greater than or equal to 20 milliseconds and less than 40 milliseconds as an example, the display device needs to display the video image frame with the serial number of the second frame. If the serial number of the video image frame at the head of the cache queue is the first frame, it is determined that the time for displaying the video image frame of the first frame has been exceeded, and the generated image frame control result is a deleted frame. If the serial number of the video image frame at the head of the cache queue is the second frame, it is determined that the video image frame of the second frame needs to be placed in the image additional layer, and the generated image frame control result is an updated frame. If the serial number of the video image frame at the head of the cache queue is the third frame, it is determined that the time for displaying the video image frame of the third frame has not been reached, and the display device needs to repeatedly display the video image frame of the second frame, and the generated image frame control result is a repeated frame.

[0073] As an example, generating an image frame control result according to the timestamp of the video image frame includes:

[0074] Querying whether the buffer queue contains a video image frame;

[0075] If the buffer queue contains a video image frame, an image frame control result is generated according to the timestamp of the video image frame.

[0076] The video data is encapsulated into a limited number of video image frames, and a cache queue is used to store the video image frames to be displayed. A third thread queries the cache queue to determine whether a video image frame exists. If a video image frame exists in the cache queue, an image frame control result is generated based on the timestamp of the video image frame. If a video image frame does not exist in the cache queue, the step of querying the cache queue is repeated to wait for the second thread to place the video image frame in the cache queue. Furthermore, if the step of querying the cache queue is repeated a predetermined number of times and still no video image frame exists, it can be determined that all video image frames have been displayed.

[0077] Step 130: Determine the video image frame to be placed in the image additional layer according to the image frame control result.

[0078] Typically, a display device requires video frames to be refreshed at a preset interval, such as every 20 milliseconds. Based on the frame control results, the rate at which video frames are added to the image layer is determined to achieve a frame rate that matches the current display resolution, thereby achieving a high-frame-rate video playback effect.

[0079] As an example, the image frame control result includes deleting a frame, repeating a frame, and updating a frame, and determining the video image frame to be placed in the image additional layer according to the image frame control result includes:

[0080] If the image frame control result is to delete the frame, remove the head video image frame of the cache queue, and repeat the step of generating the image frame control result according to the timestamp of the video image frame;

[0081] If the image frame control result is to update the video image frame, the head video image frame of the cache queue is placed in the image additional layer.

[0082] If the image control result is an update frame, it is determined that the display device needs to display the head video image frame in the cache queue at this time, and the head video image frame in the cache queue is placed in the image additional layer. If the image control result is a delete frame, it is determined that the time range of the head video image frame in the display device's cache queue has exceeded, and the head video image frame in the cache queue is removed.

[0083] After removing the head video frame from the cache queue, or placing it in the image appending layer, the next-highest video frame in the cache queue is converted to the new head video frame. Based on the timestamp of the new head video frame, a new image control result is generated, thereby processing all video frames in the cache queue.

[0084] It should be understood that if the image control result is a repeated frame, the display device needs to repeatedly display the video image frame being displayed, without removing the head video image frame of the cache queue or placing the head video image frame of the cache queue into the image additional layer, until the image control result is changed from a repeated frame to a deleted frame or a repeated frame.

[0085] Step 140 : Generate a display image frame according to the UI image frame in the frame buffer and the video image frame in the image additional layer.

[0086] The UI image frames in the asynchronously processed frame buffer and the video image frames in the image additional layer are mixed and processed to obtain display image frames. It should be understood that generating display image frames means that the display device finally displays the image.

[0087] Based on a preset transparency, the UI image frames in the frame buffer are blended with the video image frames in the image appending layer to generate a display image frame. This often requires providing the visual effect of a video display beneath the UI. Based on a preset transparency, the UI image frames in the frame buffer are blended with the video image frames in the image appending layer to generate a display image frame. This preset transparency allows the UI displayed on the display device to have higher contrast and clarity, improving the user's visual experience.

[0088] The present application provides an image processing method, which is applied to an image processing device, wherein the image processing device includes an image attachment layer and a frame buffer, and the method includes: rendering a UI image frame to the frame buffer, and placing a video image frame into a cache queue, wherein the video image frame includes a timestamp; generating an image frame control result based on the timestamp of the video image frame; determining the video image frame to be placed into the image attachment layer based on the image frame control result; and generating a display image frame based on the UI image frame in the frame buffer and the video image frame in the image attachment layer. By setting an image attachment layer and a frame buffer at the same level, processing the UI image frame and the video image frame in parallel, and displaying the video image frame based on the image frame control result, thread jamming and image rendering lag are effectively avoided, and a high frame rate video display effect is achieved.

[0089] Example 2

[0090] See also Figure 4 , Figure 4 FIG2 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. The image processing device 200 includes:

[0091] A frame rendering module 210 is configured to render the UI image frame to the frame buffer and place the video image frame into a buffer queue, wherein the video image frame includes a timestamp;

[0092] A frame control module 220, configured to generate an image frame control result according to the timestamp of the video image frame;

[0093] A frame processing module 230 is configured to determine the video image frame to be placed in the image additional layer according to the image frame control result, and update the image additional layer;

[0094] The frame display module 240 is configured to generate a display image frame according to the UI image frame in the frame buffer and the video image frame in the image additional layer.

[0095] As an example, the image frame control result includes deleting a frame, repeating a frame, and updating a frame. The frame processing module 230 includes:

[0096] a frame deletion submodule, configured to remove the head video image frame of the cache queue if the image frame control result is to delete the frame, and repeatedly execute the step of generating the image frame control result according to the timestamp of the video image frame;

[0097] The frame updating submodule is configured to place the head video image frame of the cache queue into the image additional layer if the image frame control result is to update the video image frame.

[0098] As an example, the frame control module 220 includes:

[0099] The first time acquisition submodule is used to obtain the current display resolution and the first time of the current display;

[0100] The result generating submodule is configured to generate an image frame control result according to the timestamp of the video image frame, the display resolution and the first time.

[0101] As an example, the frame rendering module 210 includes:

[0102] An overflow judgment submodule, used to judge whether the cache queue overflows;

[0103] a frame discarding submodule, configured to discard the video image frame if the buffer queue overflows;

[0104] The frame placing submodule is used to place the video image frame into the cache queue if the cache queue has not overflowed.

[0105] As an example, the frame rendering module 210 is further configured to determine the display order of the video image frames according to the timestamps of the video image frames, and sequentially place the video image frames into a cache queue.

[0106] As an example, the image processing apparatus 200 further includes:

[0107] An image data determination module, used to determine the type of acquired image data;

[0108] A UI data encapsulation module, configured to encapsulate the acquired image data into a UI image frame if the acquired image data is UI data;

[0109] The video data encapsulation module is configured to generate a video image frame including a timestamp according to the video data and the acquired second time if the acquired image data is video data.

[0110] As an example, the frame control module 220 includes:

[0111] A queue query submodule, configured to query whether a video image frame exists in the cache queue;

[0112] The control result submodule is configured to generate an image frame control result according to a timestamp of the video image frame if there is a video image frame in the cache queue.

[0113] The image processing device 200 also includes an image additional layer and a frame buffer. The image processing device 200 is used to execute the corresponding steps in the above-mentioned image processing method. The specific implementation of each function will not be described one by one here. The optional examples in Example 1 are also applicable to the image processing device 200 in Example 2.

[0114] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores a program or instruction, and the memory stores a program or instruction, which, when executed by the processor, implements the steps of the above-mentioned image processing method.

[0115] An embodiment of the present application further provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above-mentioned image processing method are implemented.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0117] In addition, the functional modules or units in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0118] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: Applied to an image processing device including an image additional layer and a frame buffer, the method comprises: Rendering the UI image frame to the frame buffer through a first thread, and placing the video image frame into a buffer queue through a second thread, wherein the video image frame includes a timestamp; generating an image frame control result according to the timestamp of the video image frame; Determining, according to the image frame control result, a video image frame to be placed in the image additional layer; Generate a display image frame according to the UI image frame in the frame buffer and the video image frame in the image additional layer; Generating a display image frame according to the UI image frame in the frame buffer and the video image frame in the image additional layer includes: mixing the UI image frame in the frame buffer and the video image frame in the image additional layer to generate a display image frame; Generating the image frame control result according to the timestamp of the video image frame includes: obtaining a current display resolution and a first time currently displayed; generating the image frame control result according to the timestamp of the video image frame, the display resolution and the first time; The image additional layer and the frame buffer are at the same level to provide a parallel processing asynchronous process, and the asynchronous processing of the UI image frame and the video image frame is completed by the first thread and the second thread processed in parallel; The image frame control result includes a deleted frame, a repeated frame, and an updated frame. The determining of the video image frame to be placed in the image additional layer according to the image frame control result includes: If the image frame control result is to delete the frame, remove the head video image frame of the cache queue, and repeat the step of generating the image frame control result according to the timestamp of the video image frame; If the image frame control result is to update the video image frame, placing the head video image frame of the cache queue into the image additional layer; If the image frame control result is a repeated frame, the video image frame is repeatedly displayed without removing the head video image frame of the cache queue or placing the head video image frame of the cache queue into an additional layer.

2. The image processing method according to claim 1, wherein: Putting the video image frame into the cache queue includes: Determining whether the cache queue overflows; If the cache queue overflows, discarding the video image frame; If the cache queue does not overflow, the video image frame is placed into the cache queue.

3. The image processing method according to claim 1, wherein: Putting the video image frame into the cache queue includes: The display order of the video image frames is determined according to the time stamps of the video image frames, and the video image frames are sequentially placed into a cache queue.

4. The image processing method according to claim 1, wherein: Before rendering the UI image frame to the frame buffer by the first thread and placing the video image frame into the buffer queue by the second thread, the method further includes: Determine the type of image data obtained; If the acquired image data is UI data, encapsulating the UI data into a UI image frame; If the acquired image data is video data, a video image frame including a timestamp is generated according to the video data and the acquired second time.

5. The image processing method according to claim 1, wherein: Generating an image frame control result according to the timestamp of the video image frame includes: Querying whether the buffer queue contains a video image frame; If the buffer queue contains a video image frame, an image frame control result is generated according to the timestamp of the video image frame.

6. An image processing device, characterized in that The image processing device includes an image additional layer and a frame buffer, and the device further includes: A frame rendering module, configured to render the UI image frame to the frame buffer via a first thread, and to place the video image frame into a buffer queue via a second thread, wherein the video image frame includes a timestamp; A frame control module, configured to generate an image frame control result according to a timestamp of the video image frame; A frame processing module, configured to determine the video image frame to be placed in the image additional layer according to the image frame control result, and update the image additional layer; a frame display module, configured to generate a display image frame according to the UI image frame in the frame buffer and the video image frame in the image additional layer; Generating a display image frame according to the UI image frame in the frame buffer and the video image frame in the image additional layer includes: mixing the UI image frame in the frame buffer and the video image frame in the image additional layer to generate a display image frame; Generating the image frame control result according to the timestamp of the video image frame includes: obtaining a current display resolution and a first time currently displayed; generating the image frame control result according to the timestamp of the video image frame, the display resolution and the first time; The image additional layer and the frame buffer are at the same level to provide a parallel processing asynchronous process; the asynchronous processing of the UI image frame and the video image frame is completed by the first thread and the second thread processed in parallel; The image frame control results include deleted frames, repeated frames and updated frames, and the frame processing module includes: a frame deletion submodule, configured to remove the head video image frame of the cache queue if the image frame control result is to delete the frame, and repeatedly execute the step of generating the image frame control result according to the timestamp of the video image frame; an updating frame submodule, configured to place the head video image frame of the cache queue into the image additional layer if the image frame control result is to update the video image frame; The frame processing module is further configured to repeatedly display the video image frame if the image frame control result is a repeated frame, without removing the head video image frame of the cache queue or placing the head video image frame of the cache queue into an additional layer.

7. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

Images