Image data processing method and device, equipment, storage medium and program product

Abstract

The invention provides an image data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: responding to a first display interrupt signal arriving according to a preset period and a data capture request, determining a state of a coding operation of first image data triggered based on a second display interrupt signal arriving according to a preset period, the second display interrupt signal arriving earlier than the first display interrupt signal, and the first image data being image data of a first frame stored in a frame buffer queue before the second display interrupt signal arrives; triggering a third display interrupt signal in response to determining that the encoding operation of the state representation for the first image data is completed; and based on the third display interrupt signal, obtaining second image data currently stored in the first frame from the frame buffer queue, and processing the second image data. According to the method, the display interruption can be dynamically adjusted, so that the coding efficiency can be improved, and the frame loss ratio can be reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer graphics processing and video coding, in particular to an image data processing method and device, equipment, storage medium and program product. BACKGROUND

[0002] In application fields such as remote office, online education and game live broadcast, the content of an operating system desktop and an application program can be rendered to a frame buffer by a graphics processing unit (GPU), the frame buffer serving as a source of display buffering, screen grabbing (or desktop grabbing) operation, and the data in the frame buffer will be submitted to a display for display when a next display interrupt signal arrives, and will also be copied to another grabbing buffer for subsequent processing (such as data format conversion and encoding), and the process of the subsequent processing needs to be completed before the arrival of the next display interrupt signal. Currently, the display interrupt signal is usually generated at a fixed time interval, and if the aforementioned subsequent processing process appears a large delay state and cannot complete the processing of the current frame in time, a display interrupt will be missed, frame loss will occur, and the fluency and visual effect of display output will be affected, and the encoding efficiency will also be affected.

[0003] Therefore, how to improve the encoding efficiency and reduce the proportion of frame loss becomes a technical problem to be solved. SUMMARY

[0004] Embodiments of the present disclosure provide an image data processing method, an image data processing device, an electronic device, a computer readable storage medium and a computer program product.

[0005] In a first aspect, the embodiments of the present disclosure provide an image data processing method, comprising: in response to detecting a first display interrupt signal arriving at a preset period and a data grabbing request, determining a state of an encoding operation of first image data triggered based on a second display interrupt signal arriving at the preset period, wherein the second display interrupt signal arrives earlier than the first display interrupt signal, and the first image data is image data of a first frame stored in a frame buffer queue before the arrival of the second display interrupt signal; in response to determining that the state represents that the encoding operation for the first image data has been completed, triggering a third display interrupt signal; based on the third display interrupt signal, obtaining second image data currently stored in the first frame from the frame buffer queue, and processing the second image data.

[0006] In some optional implementations of the embodiments of the present disclosure, the triggering of the third display interrupt signal in response to the determination that the state indicates that the encoding operation for the first image data is completed includes: in response to the determination that the state indicates that the encoding operation for the first image data is completed and no new display interrupt signal that arrives at a preset period is detected, triggering the third display interrupt signal.

[0007] In some optional implementations of the embodiments of the present disclosure, the image data processing method further includes: in response to the determination that the state indicates that the encoding operation for the first image data is not completed and the detection that there is third image data to be rendered into the frame buffer queue, determining a data storage state of the frame buffer queue; and in response to the determination that the data storage state indicates that the frame buffer queue is full of image data, generating a rendering frame loss identifier corresponding to the third image data.

[0008] In some optional implementations of the embodiments of the present disclosure, the triggering of the third display interrupt signal in response to the determination that the state indicates that the encoding operation for the first image data is completed includes: in response to the determination that the state indicates that the encoding operation for the first image data is completed, triggering the third display interrupt signal in a case where the rendering frame loss identifier is detected.

[0009] In some optional implementations of the embodiments of the present disclosure, the processing of the second image data includes: converting the second image data from an RGB format into a YUV format to obtain fourth image data; and performing an encoding operation on the fourth image data to obtain the code stream data.

[0010] In some optional implementations of the embodiments of the present disclosure, the performing of the encoding operation on the fourth image data includes: performing the encoding operation on the fourth image data based on preconfigured encoding attribute information, where the encoding attribute information includes at least one of the following: a code control mode, a code rate, and a buffer for storing the code stream data.

[0011] In some optional implementations of the embodiments of the present disclosure, the image data processing method further includes: determining a rendering timestamp corresponding to the second image data; and associating the code stream data with the rendering timestamp and sending the code stream data and the rendering timestamp to an upper-layer application, where the upper-layer application is configured to decode the code stream data based on the rendering timestamp.

[0012] In a second aspect, the embodiments of the present disclosure provide an image data processing apparatus, comprising: a determination module, a triggering module and a processing module. The determination module is configured to determine a state of an encoding operation of first image data triggered based on a second display interrupt signal arriving at a preset period, in response to detecting the first display interrupt signal arriving at the preset period and a data grabbing request, wherein the second display interrupt signal arrives earlier than the first display interrupt signal, and the first image data is image data of a first frame stored in a frame buffer queue before the second display interrupt signal arrives; the triggering module is configured to trigger a third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed; and the processing module is configured to obtain second image data of the current first frame stored in the frame buffer queue based on the third display interrupt signal, and process the second image data.

[0013] In some optional implementations of the embodiments of the present disclosure, the triggering module is further configured to trigger the third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed and no new display interrupt signal arriving at the preset period is detected.

[0014] In some optional implementations of the embodiments of the present disclosure, the determination module is further configured to determine a data storage state of the frame buffer queue in response to determining that the state represents that the encoding operation for the first image data has not been completed and there is third image data to be rendered into the frame buffer queue; and the processing module is further configured to generate a rendering frame loss identification corresponding to the third image data in response to determining that the data storage state represents that the frame buffer queue is full of image data.

[0015] In some optional implementations of the embodiments of the present disclosure, the triggering module is further configured to trigger the third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed, in a case where the rendering frame loss identification is detected.

[0016] In some optional implementations of the embodiments of the present disclosure, the processing module is further configured to convert the second image data from an RGB format into a YUV format to obtain fourth image data; and perform an encoding operation on the fourth image data to obtain code stream data.

[0017] In some optional implementations of the embodiments of the present disclosure, the processing module is further configured to perform the encoding operation on the fourth image data based on preconfigured encoding attribute information, wherein the encoding attribute information comprises at least one of: a code control mode, a code rate, and a buffer for storing the code stream data.

[0018] In some optional implementations of the embodiments of this disclosure, the above-mentioned processing module is further configured to: determine the rendering timestamp corresponding to the second image data; associate the bitstream data and the rendering timestamp, and send them to the upper-layer application, wherein the upper-layer application is used to decode the bitstream data based on the rendering timestamp.

[0019] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the image data processing method described in any of the implementations of the first aspect above.

[0020] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to perform an image data processing method as described in any implementation of the first aspect.

[0021] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the image data processing method as described in any implementation of the first aspect.

[0022] According to the technical solution provided in this disclosure, when a first display interrupt signal arriving at a preset period is detected, and there is a data capture requirement (i.e., a data capture request is detected) to achieve desktop capture (or screen capture), the state of the encoding operation for the first image data that has already been triggered can be determined. Based on this state, if it is determined that the encoding operation for the first image data has been completed, a temporary third display interrupt signal can be triggered in a timely manner to trigger the processing of image data for the new frame in real time. Optionally, based on the third display interrupt signal, the image data currently stored in the first frame (i.e., the second image data) can be retrieved from the frame buffer queue and the processing of the second image data can be triggered, thereby timely releasing the buffer space in the frame buffer queue to save memory. In this way, dynamic adjustment of the display interrupt is achieved. Compared to the method of triggering image data encoding and other processing with display interrupt signals arriving at fixed time intervals, the encoding efficiency can be effectively improved, and the frame drop rate can be reduced by increasing the encoding speed.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0024] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of an exemplary system architecture from the relevant disclosures; Figure 2 A flowchart of an image data processing method provided in this disclosure embodiment; Figure 3 A schematic diagram illustrating the arrival timing of a display interrupt signal, provided as an embodiment of this disclosure; Figure 4 A flowchart of another image data processing method provided in this disclosure embodiment; Figure 5 This is a schematic diagram of the architecture of an image data processing system in an application scenario provided by an embodiment of the present disclosure; Figure 6 A structural block diagram of an image data processing apparatus provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device suitable for performing an image data processing method, provided as an embodiment of the present disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0026] It should be noted that the collection, acquisition, storage, processing, transmission, provision, disclosure, and application of user personal information (such as user login application verification information) involved in the technical solution disclosed herein are all carried out with the user's knowledge and explicit authorization, comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0027] Figure 1 This is a schematic diagram of an exemplary system architecture from the relevant disclosures. For example... Figure 1As shown, the system architecture 100 may include a frame buffer queue 101, a display 102, a capture buffer queue 103, a GPU color space converter 104, a GPU hardware encoder 105, and a bitstream processing module 106. For example, in applications such as desktop capture, live streaming, and remote desktop (e.g., for remote conferencing screen sharing), the display content corresponding to the operating system desktop or application can be processed through the GPU rendering pipeline to perform graphics calculations, and the rendering results can be written to the frame buffer queue 101 in video memory, i.e., drawn to the frame buffer by the GPU. For example, the frame buffer queue 101 may include 1-3 frame buffers, each of which can be used to store one frame of image data. This frame buffer queue can serve as a display source, waiting for a display interrupt signal to trigger display, and as a buffer source for desktop capture.

[0028] When a display interrupt (or interrupt request (IR) signal, such as a Vertical Synchronization (VSync) interrupt signal) arrives, it can trigger the submission of one frame of image data from FrameBuffer queue 101 to display 102 for scanning and display. Simultaneously, one frame of image data from FrameBuffer queue 101 is copied to the buffer of capture buffer queue 103 to achieve desktop capture. The freed-up buffer space in FrameBuffer queue 101 can then be used to render a new frame of image data. The FrameBuffer and capture buffer store RGB (Red, Green, Medium, High) data. Blue represents the colors of the red, green, and blue channels respectively. It is an additive color mode that obtains various colors by changing the red (R), green (G), and blue (B) color channels and superimposing them. In order to achieve encoding, the RGB format image data in the capture buffer needs to be converted into YUV format image data by the GPU color space converter 104 (hardware) to facilitate hardware video encoding. Y represents the luminance (or luminance or chromaticity) component (that is, grayscale value), and U and V represent the two chromaticity (or chromaticity) components (which can include hue and saturation) in a color space converter 104.

[0029] When implementing hardware video encoding, the RGB to YUV format conversion can be achieved through GPU hardware, that is, converting RGB format image data into YUV format image data that can be used for encoding. For example, the dedicated encoding circuit of the GPU hardware encoder 105 can perform H.264 / H.265 compression algorithms on the YUV format image data according to configuration parameters to generate highly compressed video stream data (referred to as compressed stream data or stream data). Furthermore, the stream processing module 106, acting as an output processing unit, can distribute the stream according to application requirements. For example, the stream data can be encapsulated and sent to a network interface for network streaming, or it can be stored locally; for example, it can be written to a storage device and saved as a video file.

[0030] The aforementioned hardware video encoding needs to be completed before the arrival of the next display interrupt signal, i.e., a new display interrupt signal; otherwise, congestion will occur. Currently, the aforementioned display interrupt signals are generated at fixed time intervals. If the subsequent processing experiences significant delays and fails to complete the processing of the current frame's image data on time, a display interrupt will be missed, resulting in a dropped frame in the entire system. This affects the smoothness and visual experience of the display output, as well as the encoding efficiency.

[0031] Therefore, an image data processing solution is needed that can improve coding efficiency and reduce the proportion of dropped frames.

[0032] The following description of several optional embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0033] Please refer to Figure 2 , Figure 2 A flowchart of an image data processing method provided in this disclosure embodiment, the image data processing method being executed by an execution entity processor (such as a GPU), wherein process 200 includes the following steps: Step 201: In response to the detection of a first display interrupt signal arriving at a preset period and a data capture request, determine the state of the encoding operation of the first image data triggered by a second display interrupt signal arriving at a preset period.

[0034] This step aims to allow the execution entity (e.g., GPU) of the image data processing method to determine the status of a previously triggered encoding operation on the first image data when it detects a first display interrupt signal arriving at a preset period and there is a data capture requirement to achieve desktop capture (or screen capture). Specifically, the encoding operation on the first image data is triggered by a second display interrupt signal arriving at the preset period before the first display interrupt signal, and the first image data is the image data of the first frame (i.e., the first frame's image data) stored in the frame buffer queue before the arrival of the second display interrupt signal. In other words, in this embodiment, the second display interrupt signal arrives earlier than the first display interrupt signal; for example, as shown... Figure 3 As shown, the second display interrupt signal arrives at time t0 and the first display interrupt signal arrives at time t1. The time interval between time t1 and t0 is the aforementioned preset period, and time t0 is earlier than time t1 in the time sequence direction.

[0035] In some optional implementations of the embodiments of this disclosure, the aforementioned preset period is a fixed time interval, also known as the display interruption interval duration. Its value can depend on the refresh rate of the display and can be expressed as: Preset period = 1 / refresh rate. For example, the refresh rate of the display may include 60Hz, 75Hz, 120Hz, 144Hz, 240Hz, 360Hz, etc., and the corresponding display interruption interval duration may include 1 / 60 millisecond, 1 / 75 millisecond, 1 / 120 millisecond, 1 / 144 millisecond, 1 / 240 millisecond, 1 / 360 millisecond, etc.

[0036] In some optional implementations of the embodiments of this disclosure, the first display interrupt signal and the second display interrupt signal mentioned above include, but are not limited to, the VSync interrupt signal.

[0037] In some optional implementations of the embodiments of this disclosure, the frame buffer queue includes one or more FrameBuffers, such as 1 to 3 FrameBuffers. Each FrameBuffer can be used to store one frame of image data. That is, the first image data can be one frame of image data.

[0038] In some optional implementations of the embodiments of this disclosure, the first image data may be rendered to the frame buffer queue by the GPU and, upon the arrival of the second display interrupt signal and the detection of a corresponding data fetching request, be triggered to perform encoding operations on it.

[0039] Step 202: In response to determining that the encoding operation for the first image data has been completed, a third display interrupt signal is triggered.

[0040] Based on the state of the encoding operation for the first image data determined in step 201 above, where the state can be used to characterize whether the encoding operation for the first image data is completed, if it is further determined based on the state that the encoding operation for the first image data has been completed, a temporary third display interrupt signal can be triggered in a timely manner to trigger the processing of the image data of the new frame in the frame buffer queue in real time.

[0041] Step 203: Based on the third display interrupt signal, retrieve the second image data currently stored in the first frame from the frame buffer queue, and process the second image data.

[0042] In this embodiment, since the above-mentioned data capture request has been detected first, the image data (i.e., the second image data) currently stored in the first frame (i.e. the first frame) can be retrieved from the frame buffer queue based on the third display interrupt signal that is temporarily triggered, and further processing of the second image data can be triggered.

[0043] In some optional implementations of the embodiments of this disclosure, the second image data may be a frame of image data adjacent to the first image data, or it may be image data separated by one frame.

[0044] According to the image data processing method provided in this disclosure, when a first display interrupt signal arriving at a preset period is detected, and there is a data capture requirement (i.e., a data capture request is detected) to achieve desktop capture (or screen capture), the state of the encoding operation for the first image data that has already been triggered can be determined. Based on this state, if it is determined that the encoding operation for the first image data has been completed, a temporary third display interrupt signal can be triggered in a timely manner to trigger the processing of image data for the new frame in real time. Optionally, based on the third display interrupt signal, the image data currently stored in the first frame (i.e., the second image data) can be retrieved from the frame buffer queue and the processing of the second image data can be triggered, thereby timely releasing the buffer space in the frame buffer queue to save memory. In this way, dynamic adjustment of the display interrupt is achieved. Compared to the method of triggering image data encoding and other processing with display interrupt signals arriving at fixed time intervals, the encoding efficiency can be effectively improved, and the frame drop rate can be reduced by increasing the encoding speed.

[0045] In some optional implementations of the embodiments of this disclosure, in the above... Figure 2 Based on the corresponding embodiment, step 202 above can be implemented as follows: in response to determining that the state characterization of the encoding operation for the first image data has been completed and no new display interruption signal arriving at a preset period, a third display interruption signal is triggered.

[0046] In this embodiment, to further improve encoding efficiency, especially when the frame buffer queue is nearing or has already reached full image data storage, a temporary third display interrupt signal can be triggered in a timely manner if it is determined that the encoding operation for the first image data, which was triggered before the arrival of the first display interrupt signal, has been completed and no next display interrupt signal arriving at the aforementioned preset period has been detected. This is used to trigger the processing of the new frame of image data in the frame buffer queue in real time, thereby helping to improve encoding efficiency. For example, as shown... Figure 3 As shown, if it is determined that the encoding operation of the first image data triggered by the second display interrupt signal is completed and no new display interrupt signal that should arrive at time t3 according to the preset period is detected, a third display interrupt signal can be triggered temporarily at time t2 in advance. Based on the third display interrupt signal, the image data currently stored in the first frame (i.e., the first frame) can be retrieved from the frame buffer queue for processing, thereby releasing the buffer space in the frame buffer queue in a timely manner and saving memory more efficiently.

[0047] In some optional implementations of the embodiments of this disclosure, in the above... Figure 2 Based on the corresponding embodiments, the above image data processing method may further include the following: in response to determining that the state characterization of the encoding operation for the first image data is incomplete and detecting the existence of third image data to be rendered to the frame buffer queue, determining the data storage state of the frame buffer queue; in response to determining that the data storage state characterization of the frame buffer queue is full of image data, generating a rendering frame loss identifier corresponding to the third image data.

[0048] In this embodiment, if it is determined that the encoding operation for the first image data, which was triggered before the arrival of the first display interrupt signal, has not been completed, and third image data awaiting rendering to the frame buffer queue is detected, the data storage status of the frame buffer queue can be determined first. If the data storage status indicates that the frame buffer queue is full of image data, then the third image data awaiting rendering cannot be rendered normally to the frame buffer queue. This also indicates that the previous image data encoding operation was not completed on time, causing a delay. A rendering frame loss flag corresponding to the third image data that cannot be rendered to the frame buffer queue can then be generated. This rendering frame loss flag can be used to trigger operations related to reducing frame loss. Thus, by accurately identifying the situation where image data cannot be successfully rendered to the frame buffer queue, subsequent processing related to this situation can be effectively performed.

[0049] In some optional implementations of the embodiments of this disclosure, the third image data may be image data of one frame.

[0050] In some optional implementations of the embodiments of this disclosure, in the above... Figure 2 Based on the corresponding embodiment, if it is determined that the encoding operation for the first image data that was triggered before the arrival of the first display interrupt signal has not been completed, and if third image data to be rendered to the frame buffer queue is detected, the above step 202 can be implemented as follows: if a rendering frame loss flag is detected, in response to determining that the state characterizing the encoding operation for the first image data has been completed, a third display interrupt signal is triggered.

[0051] In this embodiment, if the above-mentioned rendering frame dropping flag is detected, that is, if the rendering frame dropping flag effectively identifies that there is a situation where image data cannot be successfully rendered into the frame buffer queue, it indicates that the frame buffer queue has overflowed due to the delay of the previous encoding operation, resulting in frame dropping. If it is determined that the encoding operation for the first image data that was triggered before the arrival of the first display interrupt signal has been completed, a temporary third display interrupt signal can be triggered in time to trigger the processing of the image data of the new frame in the frame buffer queue in real time.

[0052] In some other optional implementations of the embodiments of this disclosure, in the above... Figure 2 Based on the corresponding embodiment, if it is determined that the encoding operation for the first image data, which was triggered before the arrival of the first display interrupt signal, has not been completed, and if third image data to be rendered into the frame buffer queue is detected, the third image data associated with the aforementioned rendering frame loss flag can be cached in another buffer area different from the frame buffer queue. In response to the detection of free buffer space in the frame buffer queue, the third image data can be rendered from the other buffer area into the frame buffer queue based on the rendering frame loss flag. Furthermore, processing of the third image data can be triggered based on the first arriving display interrupt signal, wherein the first arriving display interrupt signal can include a display interrupt signal arriving at a preset period or a display interrupt signal triggered based on the rendering frame loss flag. It should be noted that this implementation can be carried out when the storage length of the frame buffer queue is sufficiently long to avoid unnecessary delays.

[0053] In some optional implementations of the embodiments of this disclosure, the above-mentioned response to determining that the encoding operation for the first image data has been completed and detecting a rendering frame drop flag, triggering a third display interrupt signal, can be implemented as follows: The response to determining that the encoding operation for the first image data has been completed, detecting a rendering frame drop flag, and not detecting a new display interrupt signal arriving at a preset period, triggering a third display interrupt signal. That is, in order to further improve encoding efficiency, when it is determined that the frame buffer queue is full of image data, a temporary third display interrupt signal can be triggered in a timely manner if no next display interrupt signal arriving at the preset period (i.e., a new display interrupt signal, which can be understood as a new display interrupt signal to arrive adjacent to the first display interrupt signal) is detected. This is used to trigger the processing of the new frame of image data in the frame buffer queue in real time, thereby helping to improve encoding efficiency.

[0054] In some optional implementations of the embodiments of this disclosure, in the above... Figure 2 Based on the corresponding embodiment, the operation of processing the second image data in step 203 above can be implemented as follows: converting the second image data from RGB format to YUV format to obtain the fourth image data; performing an encoding operation on the fourth image data to obtain the bitstream data.

[0055] In this embodiment, considering that the image data in the frame buffer queue is stored in RGB format, in order to ensure successful encoding and guarantee the stability and reliability of the system, the second image data needs to be converted into a YUV format suitable for encoding. Then, the fourth image data in YUV format is encoded to obtain the corresponding bitstream data. In some optional implementations of this disclosure, the H.264 / H.265 compression algorithm can be used to encode the fourth image data in YUV format into highly compressed video bitstream data.

[0056] In some optional implementations of the embodiments of this disclosure, the above-mentioned encoding operation on the fourth image data can be implemented as follows: the encoding operation on the fourth image data is performed based on pre-configured encoding attribute information, wherein the encoding attribute information includes at least one of the following: code control method, code rate, and buffer for storing code stream data.

[0057] In this embodiment, to ensure efficiency and accuracy for YUV format image data, encoding attribute information corresponding to the encoding operation needs to be pre-configured. This encoding attribute information includes, but is not limited to, one or more of the following: bitrate control method, bitrate, and a buffer for storing the bitstream data. The bitrate control method can include, but is not limited to, Constant Bit Rate (CBR), Variable Bit Rate (VBR), Constant Rate Factor (CRF), and Average Bit Rate (ABR). Bitrate refers to the amount of data per second after video encoding. The bitrate selection can be determined based on the target resolution, frame rate, the dynamic nature of the content (such as a static lecture or a dynamic game), and transmission bandwidth / storage limitations. The buffer for storing the bitstream data, also known as the bitrate-controlled buffer, is a buffer area within the encoder used to smooth the output bitstream.

[0058] In some optional implementations of the embodiments of this disclosure, the operations of converting the second image data from RGB format to YUV format and the encoding operations performed on the fourth image data in YUV format can both be completed simultaneously by a GPU hardware encoder. This eliminates the need to first store the RGB format second image data in a capture buffer and then perform format conversion using a GPU color space converter. This saves memory on the FrameBuffer (the capture buffer) used to store the RGB format image data and the Buffer used to store the YUV format image data. Furthermore, since a redundant GPU color space converter is eliminated, hardware resource consumption is saved, costs are reduced, and software control of hardware operations is reduced, thereby reducing user-mode to kernel-mode calls and lowering memory usage.

[0059] Please refer to Figure 4 , Figure 4 A flowchart of another image data processing method provided in this disclosure embodiment, the image data processing method being executed by an execution entity processor (such as a GPU), wherein process 400 includes the following steps: Step 401: In response to detecting a first display interrupt signal arriving at a preset period and a data capture request, determine the state of the encoding operation of the first image data triggered by a second display interrupt signal arriving at a preset period.

[0060] In some optional implementations of the embodiments of this disclosure, the second display interrupt signal arrives earlier than the first display interrupt signal, and the first image data is the image data of the first frame stored in the frame buffer queue before the second display interrupt signal arrives.

[0061] Step 402: In response to determining that the encoding operation for the first image data has been completed, a third display interrupt signal is triggered.

[0062] Step 403: Based on the third display interrupt signal, retrieve the second image data currently stored in the first frame from the frame buffer queue, and process the second image data.

[0063] Steps 401-403 above are the same as described above. Figure 2 The steps 201-203 described in the corresponding embodiments are the same. For the same parts, please refer to the corresponding parts of the previous embodiment, which will not be repeated here.

[0064] Step 404: Determine the rendering timestamp corresponding to the second image data.

[0065] Step 405: Associate the bitstream data with the rendering timestamp and send it to the upper-layer application, whereby the upper-layer application decodes the bitstream data based on the rendering timestamp.

[0066] In this embodiment, when a first display interrupt signal arriving at a preset period is detected, and there is a data capture requirement (i.e., a data capture request is detected) to achieve desktop capture (or screen capture), a temporary third display interrupt signal can be triggered promptly after determining that the encoding operation for the first image data has been completed. In response to this third display interrupt signal, the image data (i.e., the second image data) currently stored in the first frame (i.e., the first frame) is retrieved from the frame buffer queue, and processing of the second image data is triggered to obtain the corresponding bitstream data. The rendering timestamp corresponding to the second image data can then be associated with the processed bitstream data and provided together to the upper-layer application. The rendering timestamp corresponding to the second image data is recorded when the second image data is rendered to the frame buffer queue. Thus, the upper-layer application can use the rendering timestamp to correctly decode the bitstream data and achieve correct display.

[0067] To enhance understanding, this disclosure also provides a specific implementation scheme based on a particular application scenario. Please refer to the example below. Figure 5The system architecture 500 shown includes at least a frame buffer queue 501, a GPU color space conversion and encoder 503, and a bitstream processing module 505. For example, in applications such as desktop capture, live streaming, and remote desktop (e.g., for remote conference screen sharing), the display content corresponding to the operating system desktop or application can be processed through the GPU rendering pipeline to perform graphics calculations, and the rendering results can be written to the frame buffer queue 501 in video memory. That is, the GPU draws the data into the frame buffer, and the timestamp corresponding to the rendering operation (including the rendering timestamp in the above embodiment) can be recorded. For example, the frame buffer queue 501 can include 1-3 frame buffers, each of which can be used to store one frame of image data. The frame buffer queue 501 can serve as a display source or a buffer source for desktop capture (or screen capture).

[0068] In this embodiment, the encoding attributes required for encoding by the GPU color space converter and encoder 503 (corresponding to the encoding information in the above embodiments) can be pre-configured, such as code control mode, bit rate, and buffer required for the encoding process. Driven by a display interrupt signal (which typically arrives at fixed time intervals (corresponding to the preset period in the above embodiments), the GPU color space converter and encoder 503 can be triggered to process the image data cached in the FrameBuffer queue 501 based on these encoding attributes. The GPU color space converter and encoder 503 can directly process RGB format image data (including the first image data, second image data, and third image data in the above embodiments), i.e., it can convert the RGB format to YUV format before encoding. This saves memory on the FrameBuffer (i.e., the aforementioned capture buffer) used to store RGB format image data and the Buffer used to store YUV format image data. Furthermore, since a redundant GPU color space converter is not required, hardware resource overhead is saved, thereby reducing costs. It also reduces software control of hardware operations, thus reducing user-mode to kernel-mode calls and lowering memory usage. Optionally, H.264 / H.265 compression algorithms can be used to generate compressed video stream data. After encoding, the stream data and the timestamp recorded when the corresponding image data was rendered can be sent to the user space of the GPU through the stream processing module 505, which is the output processing unit, so that it can be pushed to the upper layer application.

[0069] In this embodiment, if the current FrameBuffer queue 501 is full of image data, and there is RGB data (corresponding to the third image data in the above embodiment) that needs to be rendered into the FrameBuffer queue 501, and there is a display interrupt signal (corresponding to the first display interrupt signal in the above embodiment) that arrives at the above fixed time interval, then the RGB data cannot be rendered into the FrameBuffer queue 501, that is, frame dropping occurs. In this case, dynamic adjustment of the display interrupt can be started, and a rendering frame dropping mark can be recorded.

[0070] Optionally, when the FrameBuffer queue 501 is full and a periodic display interrupt signal arrives, if the encoding operation for the previous frame's image data (corresponding to the first image data) is completed, a temporary display interrupt signal (corresponding to the third display interrupt signal) is triggered for encoding and rendering of the new frame's image data (corresponding to the second image data). Further, optionally, when the next periodic display interrupt signal (i.e., a new display interrupt signal) arrives, and the encoding and rendering of the new frame's image data have been completed, the operation of retrieving a new frame of data from the FrameBuffer queue 501 and rendering the new frame's image data to the FrameBuffer queue 501 can continue in response to the new display interrupt signal, thus achieving FrameBuffer swapping. This can be understood as improving encoding efficiency and resolving congestion caused by encoding delays through the triggering of the temporary display interrupt signal. Encoding and rendering can then continue to be triggered at fixed time intervals, avoiding subsequent frame drops caused by congestion and reducing the proportion of dropped frames. Furthermore, by triggering the aforementioned temporary display interruption signal, the encoding of image data for the new frame is initiated, reducing frame drops. This is equivalent to dynamically adjusting the frame rate (i.e., the number of image frames displayed per second). Thus, under the condition that performance allows, the fixed frame rate that is preset and used in the case of triggering display interruption at the aforementioned fixed time interval without frame drops can be achieved as much as possible, thereby ensuring the smoothness of the display.

[0071] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an image data processing apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices (such as GPUs).

[0072] like Figure 6 As shown, the image data processing apparatus 600 of this embodiment may include: a determining module 601, a triggering module 602, and a processing module 603.

[0073] The determining module 601 is configured to, in response to detecting a first display interrupt signal arriving at a preset period and a data capture request, determine the state of the encoding operation of the first image data triggered by a second display interrupt signal arriving at a preset period, wherein the second display interrupt signal arrives earlier than the first display interrupt signal, and the first image data is the image data of the first frame stored in the frame buffer queue before the arrival of the second display interrupt signal; the triggering module 602 is configured to, in response to determining that the state indicates that the encoding operation for the first image data has been completed, trigger a third display interrupt signal; the processing module 603 is configured to, based on the third display interrupt signal, retrieve the second image data currently stored in the first frame from the frame buffer queue and process the second image data.

[0074] In this embodiment of the disclosure, the image data processing apparatus 600 includes: the determining module 601, the triggering module 602, and the processing module 603. The specific processing of the determining module 601 and its resulting technical effects can be found in the following references: Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiments will not be repeated here.

[0075] In some optional implementations of the embodiments of this disclosure, the triggering module 602 is further configured to trigger a third display interrupt signal in response to determining that the state characterization of the encoding operation for the first image data has been completed and no new display interrupt signal arriving at a preset period.

[0076] In some optional implementations of the embodiments of this disclosure, the determining module 601 is further configured to: determine the data storage state of the frame buffer queue in response to determining that the state characterizes that the encoding operation for the first image data has not been completed and that there is third image data to be rendered to the frame buffer queue; the processing module 603 is further configured to: generate a rendering frame loss identifier corresponding to the third image data in response to determining that the data storage state characterizes that the frame buffer queue is full of image data.

[0077] In some optional implementations of the embodiments of this disclosure, the triggering module 602 is further configured to: trigger a third display interrupt signal in response to determining that the encoding operation for the first image data has been completed when a rendering frame loss flag is detected.

[0078] In some optional implementations of the embodiments of this disclosure, the processing module 603 is further configured to: convert the second image data from RGB format to YUV format to obtain the fourth image data; and perform an encoding operation on the fourth image data to obtain the bitstream data.

[0079] In some optional implementations of the embodiments of this disclosure, the processing module 603 is further configured to: perform encoding operations on the fourth image data based on pre-configured encoding attribute information, wherein the encoding attribute information includes at least one of the following: code control method, code rate, and buffer for storing code stream data.

[0080] In some optional implementations of the embodiments of this disclosure, the processing module 603 is further configured to: determine the rendering timestamp corresponding to the second image data; associate the bitstream data with the rendering timestamp and send it to the upper-layer application, wherein the upper-layer application is used to decode the bitstream data based on the rendering timestamp.

[0081] This embodiment exists as a device embodiment corresponding to the above method embodiment. When a first display interrupt signal arriving at a preset period is detected, and there is a data capture requirement (i.e., a data capture request is detected) to achieve desktop capture (or screen capture), the state of the encoding operation for the first image data that has already been triggered can be determined. Based on this state, if it is determined that the encoding operation for the first image data has been completed, a temporary third display interrupt signal can be triggered in a timely manner to trigger the processing of image data for the new frame in real time. Optionally, based on the third display interrupt signal, the image data currently stored in the first frame (i.e., the second image data) can be retrieved from the frame buffer queue and the processing of the second image data can be triggered, thereby timely releasing the buffer space in the frame buffer queue to save memory. In this way, dynamic adjustment of the display interrupt is achieved. Compared to the method of triggering image data encoding and other processing with display interrupt signals arriving at fixed time intervals, the encoding efficiency can be effectively improved, and the frame drop rate can be reduced by increasing the encoding speed.

[0082] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the image data processing method described in any of the above embodiments.

[0083] According to embodiments of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions that enable a computer to perform the image data processing method described in any of the above embodiments.

[0084] According to embodiments of this disclosure, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the image data processing method described in any of the above embodiments.

[0085] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0086] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0087] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0088] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as image data processing methods. For example, in some embodiments, the image data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the image data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the image data processing method by any other suitable means (e.g., by means of firmware).

[0089] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0091] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0092] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0093] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0094] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology. Cloud servers, also known as cloud computing servers or cloud hosts, are a hosting product within the cloud computing service ecosystem, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) services, such as high management difficulty and weak business scalability.

[0095] According to the technical solution provided in this disclosure, when a first display interrupt signal arriving at a preset period is detected, and there is a data capture requirement (i.e., a data capture request is detected) to achieve desktop capture (or screen capture), the state of the encoding operation for the first image data that has already been triggered can be determined. Based on this state, if it is determined that the encoding operation for the first image data has been completed, a temporary third display interrupt signal can be triggered in a timely manner to trigger the processing of image data for the new frame in real time. Optionally, based on the third display interrupt signal, the image data currently stored in the first frame (i.e., the second image data) can be retrieved from the frame buffer queue and the processing of the second image data can be triggered, thereby timely releasing the buffer space in the frame buffer queue to save memory. In this way, dynamic adjustment of the display interrupt is achieved. Compared to the method of triggering image data encoding and other processing with display interrupt signals arriving at fixed time intervals, the encoding efficiency can be effectively improved, and the frame drop rate can be reduced by increasing the encoding speed.

[0096] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for processing image data, comprising: determining a state of an encoding operation of first image data triggered based on a second display interrupt signal arriving at a preset period in response to detecting the first display interrupt signal arriving at the preset period and a data grabbing request, wherein the second display interrupt signal arrives earlier than the first display interrupt signal, and the first image data is image data of a first frame stored in a frame buffer queue before the second display interrupt signal arrives; triggering a third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed; acquiring second image data of the first frame currently stored in the frame buffer queue based on the third display interrupt signal, and processing the second image data.

2. The method of claim 1, wherein, The triggering of the third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed comprises: triggering the third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed and no new display interrupt signal arriving at the preset period is detected.

3. The method of claim 1, wherein, The method further comprises: determining a data storage state of the frame buffer queue in response to determining that the state represents that the encoding operation for the first image data has not been completed and that there is third image data to be rendered into the frame buffer queue; generating a rendering frame loss identification corresponding to the third image data in response to determining that the data storage state represents that the frame buffer queue is full of image data.

4. The method of claim 3, wherein, The triggering of the third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed comprises: triggering the third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed in a case that the rendering frame loss identification is detected.

5. The method of any one of claims 1 to 4, wherein, The processing of the second image data comprises: converting the second image data from an RGB format to a YUV format to obtain fourth image data; performing an encoding operation on the fourth image data to obtain code stream data.

6. The method of claim 5, wherein, The performing of the encoding operation on the fourth image data comprises: performing the encoding operation on the fourth image data based on preconfigured encoding attribute information, wherein the encoding attribute information comprises at least one of the following: a code control mode, a code rate, and a buffer for storing the code stream data.

7. The method of claim 5, wherein, The method further comprises: determining a rendering timestamp corresponding to the second image data; associating the code stream data with the rendering timestamp and sending to an upper layer application, wherein the upper layer application is configured to decode the code stream data based on the rendering timestamp.

8. An apparatus for processing image data, comprising: The determining module is configured to determine a state of an encoding operation of first image data triggered based on a second display interrupt signal arriving at a preset period in response to detecting the first display interrupt signal arriving at the preset period and the data grabbing request, wherein the second display interrupt signal arrives earlier than the first display interrupt signal, and the first image data is image data of a first frame stored in a frame buffer queue before the second display interrupt signal arrives. The triggering module is configured to trigger a third display interrupt signal in response to determining that the state represents that the encoding operation for the first image data has been completed. The processing module is configured to acquire second image data of a current first frame stored in the frame buffer queue based on the third display interrupt signal and process the second image data. 9.An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image data processing method of any one of claims 1-7.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the image data processing method of any one of claims 1-7.

11. A computer program product comprising a computer program, wherein, The computer program is executed by the processor to implement the image data processing method according to any one of claims 1-7.

Images