Screen picture acquisition method, device and equipment

Abstract

The embodiment of the invention discloses a screen picture acquisition method, device and equipment. According to the technical scheme provided by the embodiment of the invention, the cache frame obtained by calling the graphic processing unit by the screen acquisition tool to perform image acquisition on the screen display image is acquired from the frame pool in the callback mode, and the cache frame is converted into the first texture data; converting a color space of the first texture data into a set color space to obtain second texture data, and performing image coding processing on the second texture data through a hardware encoder to obtain screen acquisition data; the graphic processing unit is directly called through the screen collection tool to collect the cache frame, the cache frame is obtained from the frame pool in a callback mode, the collection and processing efficiency of the cache frame is effectively improved, and the collection speed and collection efficiency of the cache frame can be effectively improved through a hardware collection mode of the graphic processing unit. And the screen picture acquisition efficiency is effectively improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing technologies, and in particular, to a method, apparatus, and device for screen image acquisition. Background Art

[0002] Currently, screen sharing technology mainly realizes the process of screen acquisition, local encoding, sending screen data, remote decoding, and remote display. Screen sharing technology is widely used in scenarios such as video conferencing and instant messaging tools to share the local screen image for remote personnel to view and interact.

[0003] Traditional screen sharing technology usually relies on software for screen acquisition, encoding, and data transmission, which requires a large amount of data processing in the central processing unit (CPU, Central Processing Unit), occupies a relatively high performance of the central processing unit, and has a relatively low screen image acquisition efficiency. Summary of the Invention

[0004] Embodiments of the present application provide a method, apparatus, device, and storage medium for screen image acquisition to solve the technical problems in related technologies that screen image acquisition occupies a relatively high performance of the central processing unit and has a relatively low screen image acquisition efficiency, effectively reducing the performance occupation of the central processing unit by screen image acquisition and improving the screen image acquisition efficiency.

[0005] In a first aspect, embodiments of the present application provide a method for screen image acquisition, including:

[0006] Obtaining a cached frame from a frame pool in a callback manner, where the cached frame is obtained by a screen acquisition tool invoking a graphics processing unit to perform screen image acquisition on a screen display image;

[0007] Converting the cached frame into first texture data;

[0008] Converting the color space of the first texture data to a set color space to obtain second texture data;

[0009] Performing image encoding processing on the second texture data through a hardware encoder to obtain screen acquisition data.

[0010] In a second aspect, embodiments of the present application provide a screen image acquisition apparatus, including an image acquisition module, a data conversion module, a color conversion module, and an image encoding module, where:

[0011] The image acquisition module is configured to obtain a cached frame from a frame pool in a callback manner, where the cached frame is obtained by a screen acquisition tool invoking a graphics processing unit to perform screen image acquisition on a screen display image;

[0012] The data conversion module is used to convert the cache frame into first texture data;

[0013] The color conversion module is used to convert the color space of the first texture data into a set color space to obtain second texture data;

[0014] The image encoding module is used to perform image encoding processing on the second texture data through a hardware encoder to obtain screen acquisition data.

[0015] In a third aspect, an embodiment of the present application provides a screen image acquisition device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the screen image acquisition method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute the screen image acquisition method as described in the first aspect.

[0019] The embodiment of the present application obtains the cached frame obtained by the screen acquisition tool calling the graphics processing unit to acquire the screen display picture from the frame pool through a callback method, converts the cached frame into first texture data, and converts the color space of the first texture data into a set color space to obtain second texture data, performs image encoding processing on the second texture data through a hardware encoder to obtain screen acquisition data, and directly calls the graphics processing unit through the screen acquisition tool to acquire the cached frame, without the need to use software to acquire the screen display picture, thereby reducing the performance occupation of the central processing unit, and acquiring the cached frame from the frame pool through the callback method, without the need for the central processing unit to participate in the acquisition process of the cached frame, effectively improving the acquisition and processing efficiency of the cached frame, and the hardware acquisition method of the graphics processing unit can effectively improve the acquisition speed and acquisition efficiency of the cached frame, effectively improving the screen picture acquisition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a screen image acquisition method provided by an embodiment of the present application;

[0021] Figure 2 is a flow chart of another screen image acquisition method provided by an embodiment of the present application;

[0022] Figure 3 It is a structural schematic diagram of a screen image acquisition device provided in an embodiment of the present application;

[0023] Figure 4 It is a schematic structural diagram of a screen image acquisition device provided by an embodiment of the present application. Detailed implementation manners

[0024] In order to make the objectives, technical solutions, and advantages of the present application clearer, the following further describes the specific embodiments of the present application in detail with reference to the accompanying drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the convenience of description, only parts related to the present application are shown in the drawings, rather than all the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. When the operations are completed, the above process can be terminated, but there can also be additional steps not included in the drawings. The above process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0025] The screen image acquisition method provided by the present application can be applied to scenarios such as video conferencing, instant messaging, and screen recording. The aim is to directly call the graphics processing unit to acquire buffer frames through a screen acquisition tool, and obtain buffer frames from the frame pool through a callback method, reducing the performance occupancy of the central processing unit, improving the acquisition and processing efficiency of buffer frames, and improving the screen image acquisition efficiency through hardware acquisition.

[0026] In the existing screen image acquisition solutions, generally, software is relied on for screen acquisition, encoding, and data transmission. To a certain extent, this will consume a large amount of central processing unit computing resources, thus affecting the overall performance of the system, having a relatively high occupancy of the central processing unit performance, and a relatively low screen image acquisition efficiency. And due to the time-consuming of acquisition and encoding, and traditional screen image acquisition solutions often cannot achieve high-frame-rate screen sharing, the screen image acquisition quality is relatively low, which will affect the user experience in scenarios that require real-time interaction. Based on this, an embodiment of the present application provides a screen image acquisition method to solve the technical problems of relatively high occupancy of the central processing unit performance, relatively low screen image acquisition efficiency, and relatively low screen image acquisition quality in the existing screen image acquisition solutions.

[0027] Figure 1 The flowchart of a screen image acquisition method provided by an embodiment of the present application is given. The screen image acquisition method provided by the embodiment of the present application can be executed by a screen image acquisition device, and the screen image acquisition device can be implemented in a hardware and / or software manner and integrated in a screen image acquisition device.

[0028] The following describes the screen image acquisition method executed by a screen image acquisition device as an example. Refer to Figure 1 , the screen image acquisition method includes:

[0029] S110: Obtain a cached frame from a frame pool in a callback manner. The cached frame is obtained by a screen acquisition tool calling a graphics processing unit to perform screen image acquisition on the screen display image.

[0030] Exemplarily, a cached frame is obtained from the frame pool in a callback manner. Among them, the application layer in the screen image acquisition device registers a callback for the cached frame in the frame pool. The cached frame is called by the underlying layer in the screen image acquisition device to call the screen acquisition tool. The screen acquisition tool calls a graphics processing unit (GPU, Graphics Processing Unit) to perform screen image acquisition on the screen display image to obtain a cached frame and save it to the frame pool. When a new cached frame is added to the frame pool, a callback is performed on the newly added cached frame, and the cached frame is called back to the application layer. The application layer obtains the cached frame from the frame pool frame for subsequent data processing.

[0031] The screen acquisition tool provided by this solution can be WGC (Windows Graphics Capture, desktop / window image acquisition), DXGI (DirectX Graphics Infrastructure, graphics device interface), etc. This solution describes WGC as the screen acquisition tool as an example. Among them, WGC can implement the function of filtering windows and can more flexibly acquire the screen display image. In one embodiment, the screen acquisition tool calls a graphics processing unit to perform screen image acquisition on the screen display image to obtain a cached frame, and saves the cached frame to a preset frame pool. By using hardware acquisition technology to directly obtain the screen image from the graphics processing unit, the participation of the central processing unit in screen image acquisition is reduced, and the acquisition efficiency of the screen display image can be effectively improved. Optionally, the screen acquisition tool can call a graphics processing unit to perform screen image acquisition on the screen display image based on a set image resolution (such as 720P, 1080P, 2K, 4K, 8K, etc.) to obtain a cached frame with the corresponding resolution.

[0032] Among them, compared with the screen capture method of traditional screen capture tools that capture the screen display image based on a set frame rate, in this solution, by registering a callback for the frame pool, when the screen capture tool calls the graphics processing unit to capture the screen display image, the frame rate of capturing the screen display image is not restricted. When a cached frame is captured and placed in the frame pool, the application layer is notified for callback to obtain the cached frame for subsequent data processing. By adjusting the traditional active capture method of capturing the screen display image based on a set frame rate to obtaining a cached frame from the frame pool, and calling back the cached frame from the frame pool whenever a new cached frame is generated, there is no need to restrict the frame rate. The cached frame is copied by the graphics processing unit, reducing the time-consuming for obtaining the cached frame. In the screen display image capture scenario of high-resolution (such as 4K) screen sharing, the capture efficiency of the screen display image will be better than that of actively obtaining frames, meeting the high-frame-rate screen capture requirements of mid- to low-performance integrated graphics machines.

[0033] S120: Convert the cached frame into first texture data.

[0034] Exemplarily, after obtaining the cached frame by calling back from the frame pool, convert the data type of the cached frame into the texture data type to obtain the first texture data. In this solution, by converting the cached frame into the first texture data, the texture data captured by the graphics processing unit does not need to be copied between the central processing unit and the graphics processing unit, reducing the interoperation between the central processing unit and the graphics processing unit for screen capture data (such as 4K data), and avoiding the time-consuming of mapping graphics processing unit data (locking the data in the graphics processing unit and performing data transmission between the graphics processing unit and the central processing unit). When capturing a 4K cached frame, compared with the data capture time-consuming of the traditional software capture method that requires interoperation between the central processing unit and the graphics processing unit for the captured data, the average time-consuming can be effectively reduced by 30 - 60 ms, and when the central processing unit load is relatively high, the time-consuming can be reduced by more than 100 ms, effectively improving the screen image capture efficiency.

[0035] S130: Convert the color space of the first texture data to a set color space to obtain second texture data.

[0036] Exemplarily, the color space of the first texture data is converted to a set color space to obtain second texture data. Optionally, the set color space may be the YUV color space, and the color format corresponding to the second texture data is the YUV format. Among them, the conversion of the color format of the first texture data can be understood as the conversion of the color space of the first texture data to the set color space (such as the mutual conversion between color formats such as ARGB, NV12, I420, etc.). For example, when the color format of the captured buffer frame is ARGB, the color format of the corresponding first texture data is ARGB, and the color format of the first texture data can be converted to the YUV format (such as the NV12 format or I420 format in the YUV format) to obtain the second texture data in the YUV format. By converting the color space of the first texture data, the image quality and the ability to adapt to encoding parameters can be effectively optimized, ensuring the normal progress of image encoding processing and the acquisition effect of the screen image.

[0037] In a possible embodiment, after the screen image acquisition method provided by this solution converts the color space of the first texture data to a set color space to obtain second texture data, it further includes: performing pre-processing on the second texture data, and the pre-processing includes one or a combination of color correction, cropping, scaling, upsampling, downsampling, rotation, and equal ratio alignment. By pre-processing the second texture data, the image quality and the ability to adapt to encoding parameters can be effectively optimized. Among them, when performing color space conversion and / or pre-processing on the texture data, the color space conversion and / or pre-processing of the texture data can be performed by means of hardware processing of the graphics processing unit. When performing basic pre-processing on the second texture data, color correction, cropping, scaling, upsampling, downsampling, rotation, and equal ratio alignment processing can be performed on the second texture data based on set color standards, image sizes, resolutions, display angles, etc. In an embodiment, after converting the color space of the second texture data to a set color space to obtain the second texture data, set algorithm processing can also be performed on the second texture data, and the set algorithm processing of the second texture data can be set according to the processing requirements for the second texture data. By configuring the set algorithm processing for the second texture data, this solution provides an interface for adjusting the second texture data, realizes flexible processing of the texture data, and improves the flexibility of screen image acquisition.

[0038] Through pre-processing of the first texture data such as color format conversion, basic pre-processing, and set algorithm processing, this solution effectively optimizes the image quality and the ability to adapt to encoding parameters, is compatible with data inputs in different formats, and ensures the quality of screen image acquisition. Different from traditional software downsampling and color space conversion, this solution realizes downsampling and color space conversion of texture data based on hardware (graphics processing unit) to improve the conversion efficiency of the entire video link and effectively improve the efficiency of screen image acquisition.

[0039] S140: Image-encode the second texture data through a hardware encoder to obtain screen capture data.

[0040] Exemplarily, input the second texture data into a hardware encoder (such as an Intel encoder), and image-encode the second texture data through the hardware encoder to obtain screen capture data. Optionally, after obtaining the screen capture data, the screen capture data can be sent to a screen sharing receiver, and the screen sharing receiver decodes and renders the screen capture data, and displays the screen image captured by the screen image capture device on the screen sharing receiver. Optionally, after obtaining the screen capture data, the screen capture data can also be saved at a set storage location for subsequent viewing of the screen capture data.

[0041] This solution is based on the screen capture hardware acceleration mechanism of the screen capture tool WGC and the encoder, gives full play to the collaborative processing capabilities of the graphics processing unit and the central processing unit, solves the frame rate bottleneck of the traditional method of capturing screen display images based on a fixed frame rate, adjusts the video link, and the adjustment of the video link involves desktop / window capture, basic pre-processing, encoding, etc., and performs hardware acceleration on the entire video pipeline link to improve the screen image capture efficiency and frame rate.

[0042] In one embodiment, disaster recovery processing in case of hardware anomalies can be configured. For example, when the graphics processing unit has an anomaly, convert the texture data into YUV format data and perform software encoding using the central processing unit to ensure the normal progress of screen image capture.

[0043] As described above, obtain the cached frame captured by the screen capture tool calling the graphics processing unit for the screen display image from the frame pool through a callback method, convert the cached frame into the first texture data, and convert the color space of the first texture data to a set color space to obtain the second texture data. Image-encode the second texture data through a hardware encoder to obtain screen capture data. Directly call the graphics processing unit by the screen capture tool to capture the cached frame, without using software to capture the screen display image, reducing the performance occupancy of the central processing unit. And obtain the cached frame from the frame pool through a callback method, without the central processing unit participating in the process of capturing and obtaining the cached frame, effectively improving the capture and processing efficiency of the cached frame. And through the hardware capture method of the graphics processing unit, the capture speed and efficiency of the cached frame can be effectively improved, and the screen image capture efficiency can be effectively improved.

[0044] Based on the above embodiments, Figure 2 The flowchart of another screen image capture method provided by the embodiment of the present application is given. This screen image capture method is a specific implementation of the above screen image capture method. Refer to Figure 2, the screen image acquisition method includes:

[0045] S210: Invoke the interface of the screen capture tool, and through the screen capture tool, invoke the graphics processing unit to perform screen image capture on the screen display image, and save the captured buffer frame to the frame pool.

[0046] S220: Callback the buffer frame from the frame pool.

[0047] Among them, the buffer frame provided by this solution is obtained by invoking the graphics processing unit through the screen capture tool to perform screen image capture on the screen display image. Exemplarily, when it is necessary to capture screen image information for screen image capture, the interface of the screen capture tool (such as the WGC API) is invoked, so as to invoke the graphics processing unit through the screen capture tool to perform screen image capture on the screen display image to obtain a buffer frame, and save the buffer frame captured by the graphics processing unit to the frame pool. After saving the buffer frame to the frame pool, the application layer in the screen image capture device is notified by means of callback to obtain the buffer frame from the frame pool for subsequent processing. This solution invokes the screen capture tool through the call interface provided by the screen capture tool. The screen capture tool invokes the graphics processing unit to perform screen image capture on the screen display image to obtain a buffer frame, and obtains the buffer frame from the frame pool by means of callback, without restricting the capture frame rate, and can flexibly control the output frame rate according to the callback buffer frame, realizing high-frame-rate screen image capture and sharing, improving the flexibility of frame rate control for screen image capture, and by using the graphics processing unit to perform screen image capture on the screen display image, there is no need to use software to capture the screen display image, reducing the performance occupation of the central processing unit, effectively improving the capture speed and efficiency of the buffer frame, and effectively improving the screen image capture efficiency.

[0048] S230: Convert the buffer frame into first texture data, create a texture with a shared texture handle, and copy the first texture data into the texture.

[0049] Exemplarily, convert the cached frame obtained by the callback (IDirect3DSurface data) into first texture data (ID3D11Texture2D data), create a texture (IDirect3DSurface) with a shared texture handle, determine the shared texture handle (handle) corresponding to the texture, and write the first texture data obtained by converting the cached frame into the created texture. In this solution, the data type of the cached frame is converted from IDirect3DSurface data to ID3D11Texture2D data (texture data) through the texture IDirect3DSurface. Among them, a texture can be understood as a structured collection for storing texel data. A texel represents the smallest unit of texture data that can be read by or written to the pipeline. The texture is generated in a shared manner, and the generated texture has a shared texture handle. When transmitting texture data (including first texture data and second texture data), it can be achieved by transmitting the shared texture handle corresponding to the texture data, without the need to copy the texture data. When it is necessary to obtain the texture data, the corresponding texture data can be retrieved from the corresponding texture based on the shared texture handle. This solution transmits texture data through the texture shared handle, which can transfer texture data from one graphics processing unit context to another graphics processing unit context without copying the texture data, effectively improving the data processing efficiency and the screen image acquisition efficiency.

[0050] S240: Convert the color space of the first texture data to a set color space to obtain second texture data.

[0051] S250: Send the shared texture handle corresponding to the second texture data to the hardware encoder, and the hardware encoder obtains the second texture data corresponding to the shared texture handle in the way of texture handle sharing, and performs image encoding processing based on the second texture data to obtain screen acquisition data.

[0052] Exemplarily, after preprocessing the first texture data to obtain second texture data and copying the second texture data to a texture with a shared texture handle, the shared texture handle corresponding to the second texture data (the same as the shared texture handle of the corresponding first texture data) can be sent to the hardware encoder (intel encoder). After receiving the shared texture handle, the hardware encoder obtains the second texture data corresponding to the shared texture handle in the way of texture handle sharing, and performs image encoding processing on the second texture data to obtain screen acquisition data.

[0053] In one embodiment, the underlying layer of the Windows desktop sharing corresponding to the screen image acquisition device is implemented based on Microsoft DirectX, and the acquisition output is D3D11 texture (second texture data). Before the D3D11 texture data enters the hardware encoder, the video memory data (i.e., the second texture data) is taken out in the way of shared texture. When using the video memory surface as the input, the hardware acceleration device and the memory allocator of the external frame can be specified to ensure the normal progress of the image encoding process.

[0054] This solution transmits the second texture data to the hardware encoder by sharing the texture handle, transfers the texture data to the hardware encoder without copying the data, improves the image encoding efficiency of the second texture data, and improves the screen image acquisition efficiency.

[0055] In a possible embodiment, the data input structure of the hardware encoder provided by this solution includes a texture handle. Based on this, when the hardware encoder provided by this solution obtains the second texture data in the way of sharing the texture handle and performs image encoding processing based on the second texture data to obtain the screen acquisition data, it includes: the hardware encoder obtains the second texture data in the way of sharing the texture handle; the hardware encoder converts the structure body of the second texture data into a set encoding structure body, and performs image encoding processing on the second texture data of the set encoding structure body to obtain the screen acquisition data.

[0056] In one embodiment, the data input structure EncodeData of the hardware encoder is extended (for example, the data input structure body of the Intel media sdk corresponding to the hardware encoder is extended, and the process of receiving the shared texture handle, taking out the texture data based on the shared texture handle, and performing image encoding processing according to the texture data is implemented inside the hardware encoder), so that the hardware encoder supports inputting the texture handle. At the same time, the texture data is implemented inside the hardware encoder. For example, after the D3D11 texture data is input into the hardware encoder, the hardware encoder can use the graphics processing unit to perform image encoding processing on the texture data, so that after the texture handle of the texture data is input from the upper-layer application to the hardware encoder, the hardware encoder can take out the texture data according to the texture handle for image encoding processing.

[0057] Exemplarily, after preprocessing the first texture data to obtain the second texture data and copying the second texture data to a texture with a shared texture handle, the shared texture handle corresponding to the second texture data can be sent to the hardware encoder. The hardware encoder obtains the second texture data corresponding to the shared texture handle in a texture handle sharing manner, converts the structure of the second texture data into a set encoding structure (such as the mfxSurface structure inside the intel hardware encoder), and performs image encoding processing on the second texture data of the set encoding structure to obtain screen capture data.

[0058] In this solution, by adding a texture handle to the data input structure of the hardware encoder and implementing the conversion of the texture data structure inside the hardware encoder, it ensures the correct progress of the image encoding process for the second texture data. The hardware encoder no longer only supports the input of image formats (such as the YUV format), and can quickly obtain texture data based on the externally passed shared texture handle for encoding processing, improving the image encoding efficiency. Without copying the data, the texture data is transferred to the hardware encoder, improving the image encoding efficiency of the second texture data and the screen capture efficiency of the screen image.

[0059] In a possible embodiment, after converting the color space of the first texture data to a set color space to obtain the second texture data, the screen capture method provided by this solution further includes: sending the second texture data to the renderer and rendering the second texture data through the renderer.

[0060] Exemplarily, after preprocessing the first texture data to obtain the second texture data, the second texture data can be sent to the renderer, the second texture data is rendered through the renderer, and the rendering result is displayed on the display module, so as to display the image obtained by capturing the screen display image during the screen capture process. Optionally, the rendering process of the second texture data may include operations such as color space conversion and downsampling of the second texture data to adapt to the display capabilities of the display module.

[0061] In an embodiment, the display module for displaying the rendering result of the second texture data can be the display screen configured on the screen capture device (such as displaying the rendering result of the second texture data in a set area or window of the display screen), or an additional display screen configured to display the rendering result of the second texture data. In this solution, the renderer renders the second texture data and displays the rendering result of the second texture data, allowing the user to view the capture effect of the screen display image in real time and optimizing the user experience.

[0062] In one embodiment, when the screen image acquisition method provided by this solution sends the second texture data to the renderer and the renderer renders the second texture data, it may send the shared texture handle corresponding to the second texture data to the renderer, and the renderer obtains the second texture data in the way of texture handle sharing and renders the second texture data.

[0063] Exemplarily, after preprocessing the first texture data to obtain the second texture data and copying the second texture data to a texture with a shared texture handle, the shared texture handle corresponding to the second texture data can be sent to the renderer. After receiving the shared texture handle, the renderer obtains the corresponding second texture data in the way of texture handle sharing and renders the second texture data.

[0064] Among them, this solution acquires the screen display image through the video hardware acceleration link. The acquired texture data is D3D11 Texture data. The local renderer can use the D3D11 renderer to render the second texture data to improve the rendering effect of the second texture data. The data finally input to the renderer in the media stream is the texture handle corresponding to the second texture data in NV12 or I420 format. The D3D11 renderer extracts the second texture data corresponding to the shared texture handle, extracts the Y component and the UV component in the second texture data, and binds them to the shader resource view. The shader converts the color format of the second texture data from NV12 or I420 format to ARGB format, and the components of the rendering pipeline work together to render the rendering result (i.e., the image obtained by acquiring the screen display image). This solution transfers the second texture data to the renderer in the way of texture handle sharing, transfers the texture data to the renderer without copying the data, and improves the rendering efficiency of the second texture data. At the same time, the screen acquisition tool WGC is used for hardware acquisition of the screen display image. Compared with the traditional software acquisition methods (such as magnifying glass, GDI (Graphics Device Interface)), hardware acquisition can directly obtain the screen image from the graphics processing unit, effectively avoiding the participation of the central processing unit, improving the screen image acquisition efficiency. At the same time, different from the conventional use of the screen acquisition tool WGC, this solution improves the screen image acquisition frame rate and optimizes the screen image acquisition effect by means of passive callback and removing the copy step of the graphics processing unit data to the central processing unit.

[0065] As described above, the cached frame is obtained from the frame pool in a callback manner. The cached frame is obtained by the screen capture tool calling the graphics processing unit to capture the screen display image. The cached frame is converted into first texture data, and the color space of the first texture data is converted to a set color space to obtain second texture data. The second texture data is subjected to image encoding processing by a hardware encoder to obtain screen capture data. The screen capture tool directly calls the graphics processing unit to capture the cached frame, without using software to capture the screen display image, reducing the performance occupancy of the central processing unit. And by obtaining the cached frame from the frame pool in a callback manner, the central processing unit does not need to participate in the process of capturing and obtaining the cached frame, effectively improving the capture and processing efficiency of the cached frame. And through the hardware capture method of the graphics processing unit, the capture speed and efficiency of the cached frame can be effectively improved, and the screen image capture efficiency can be effectively improved. At the same time, by transmitting texture data in the way of texture sharing handle, the texture data can be passed from one graphics processing unit context to another graphics processing unit context without copying the texture data, effectively improving the data processing efficiency and the screen image capture efficiency.

[0066] Figure 3 The structural schematic diagram of a screen image capture device provided by an embodiment of the present application is given. Refer to Figure 3 , the screen image capture device includes an image capture module 31, a data conversion module 32, a color conversion module 33, and an image encoding module 34.

[0067] Among them, the image capture module 31 is used to obtain a cached frame from the frame pool in a callback manner. The cached frame is obtained by the screen capture tool calling the graphics processing unit to capture the screen display image; the data conversion module 32 is used to convert the cached frame into first texture data; the color conversion module 33 is used to convert the color space of the first texture data to a set color space to obtain second texture data; the image encoding module 34 is used to perform image encoding processing on the second texture data by a hardware encoder to obtain screen capture data.

[0068] As described above, the cached frame obtained from the frame pool in a callback manner, which is the screen capture of the screen display image by the screen capture tool calling the graphics processing unit, is converted into first texture data, and the color space of the first texture data is converted to a set color space to obtain second texture data. The second texture data is subjected to image encoding processing by a hardware encoder to obtain screen capture data. By directly calling the graphics processing unit by the screen capture tool to capture the cached frame, it is not necessary to capture the screen display image in a software manner, reducing the performance occupancy of the central processing unit. And by obtaining the cached frame from the frame pool in a callback manner, it is not necessary for the central processing unit to participate in the process of capturing and obtaining the cached frame, effectively improving the capture and processing efficiency of the cached frame. And through the hardware capture method of the graphics processing unit, the capture speed and efficiency of the cached frame can be effectively improved, and the screen capture efficiency can be effectively improved.

[0069] In a possible embodiment, when the image capture module 31 obtains the cached frame from the frame pool in a callback manner, it includes:

[0070] Call the interface of the screen capture tool, and call the graphics processing unit by the screen capture tool to capture the screen display image, and save the captured cached frame to the frame pool;

[0071] Callback the cached frame from the frame pool.

[0072] In a possible embodiment, when the data conversion module 32 converts the cached frame into first texture data, it includes:

[0073] Convert the cached frame into first texture data;

[0074] Create a texture with a shared texture handle, and copy the first texture data into the texture.

[0075] In a possible embodiment, when the image encoding module 34 performs image encoding processing on the second texture data by a hardware encoder to obtain screen capture data, it includes:

[0076] Send the shared texture handle corresponding to the second texture data to the hardware encoder, obtain the second texture data corresponding to the shared texture handle in a shared texture handle manner by the hardware encoder, and perform image encoding processing based on the second texture data to obtain screen capture data.

[0077] In a possible embodiment, the data input structure of the hardware encoder includes a texture handle. The hardware encoder obtains the second texture data in a shared texture handle manner and performs image encoding processing based on the second texture data to obtain screen capture data, including:

[0078] The hardware encoder obtains the second texture data in a shared texture handle manner;

[0079] The hardware encoder converts the structure of the second texture data into a set encoding structure, and performs image encoding processing on the second texture data of the set encoding structure to obtain screen capture data.

[0080] In a possible embodiment, the screen capture device further includes an image rendering module, and the image rendering module is used to send the second texture data to a renderer to render the second texture data through the renderer.

[0081] In a possible embodiment, when the image rendering module sends the second texture data to a renderer to render the second texture data through the renderer, it includes: sending the shared texture handle corresponding to the second texture data to the renderer, and obtaining the second texture data in a texture handle sharing manner through the renderer, and rendering the second texture data.

[0082] In a possible embodiment, the screen capture device further includes a preprocessing module, and the preprocessing module is used to perform preprocessing on the second texture data, and the preprocessing includes one or a combination of more of color correction, cropping, scaling, upsampling, downsampling, rotation, and equal ratio alignment.

[0083] It should be noted that in the embodiments of the above screen capture device, the included respective units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the respective functional units are only for the convenience of mutual distinction and are not used to limit the protection scope of the embodiments of the present application.

[0084] The embodiments of the present application further provide a screen capture device, and the screen capture device can integrate the screen capture device provided by the embodiments of the present application. Figure 4 is a schematic structural diagram of a screen capture device provided by the embodiments of the present application. Refer to Figure 4 , the screen capture device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the screen capture method provided by the above embodiments. Wherein, the input device 43, the output device 44, the memory 42, and the processor 41 can be connected through a bus or other means, Figure 4 and taking the connection through the bus as an example.

[0085] The memory 42, as a computing device readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the screen image acquisition method provided in any embodiment of the present application (for example, the image acquisition module 31, the data conversion module 32, the color conversion module 33 and the image encoding module 34 in the screen image acquisition device). The memory 42 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0086] The input device 43 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 44 may include a display device such as a display screen.

[0087] The processor 41 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 42, that is, realizes the above-mentioned screen image acquisition method.

[0088] The screen image acquisition device, equipment and computer provided above can be used to execute the screen image acquisition method provided by any of the above embodiments, and have corresponding functions and beneficial effects.

[0089] An embodiment of the present application also provides a storage medium storing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to execute the screen image acquisition method provided in the above embodiment. The screen image acquisition method includes: obtaining a cached frame from a frame pool through a callback method, and the cached frame is obtained by calling a graphics processing unit through a screen acquisition tool to perform image acquisition on a screen display; converting the cached frame into first texture data; converting the color space of the first texture data to a set color space to obtain second texture data; performing image encoding processing on the second texture data through a hardware encoder to obtain screen acquisition data.

[0090] Storage medium - Any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media such as CD-ROMs, floppy disks or magnetic tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. Additionally, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system that is connected to the first computer system via a network (such as the Internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (such as in different computer systems connected via a network). The storage medium may store program instructions executable by one or more processors (e.g., embodied as a computer program).

[0091] Of course, for a storage medium provided in an embodiment of the present application that stores computer-executable instructions, the computer-executable instructions are not limited to the screen capture method provided above, and may also execute related operations in the screen capture method provided in any embodiment of the present application.

[0092] The screen capture device, equipment, and storage medium provided in the above embodiments can execute the screen capture method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, reference can be made to the screen capture method provided in any embodiment of the present application.

[0093] The above is only a preferred embodiment of the present application and the technical principles applied. The present application is not limited to the specific embodiments provided here. Various obvious changes, re-adjustments, and substitutions that can be made by those skilled in the art will not depart from the protection scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments. Without departing from the concept of the present application, it may also include more other equivalent embodiments, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method for capturing a screen image, characterized in that, comprising: obtaining a cached frame from a frame pool in a callback manner, where the cached frame is obtained by a screen capture tool invoking a graphics processing unit to capture a screen display image; converting the cached frame into first texture data; converting the color space of the first texture data to a set color space to obtain second texture data; performing image encoding processing on the second texture data through a hardware encoder to obtain screen capture data.

2. The method for capturing a screen image according to claim 1, characterized in that, the obtaining a cached frame from a frame pool in a callback manner comprises: invoking an interface of a screen capture tool, and through the screen capture tool invoking a graphics processing unit to capture a screen display image, and saving the captured cached frame to the frame pool; calling back the cached frame from the frame pool.

3. The method for capturing a screen image according to claim 1, characterized in that, the converting the cached frame into first texture data comprises: converting the cached frame into first texture data; creating a texture with a shared texture handle, and copying the first texture data into the texture.

4. The method for capturing a screen image according to claim 1, characterized in that, the performing image encoding processing on the second texture data through a hardware encoder to obtain screen capture data comprises: sending the shared texture handle corresponding to the second texture data to the hardware encoder, and through the hardware encoder obtaining the second texture data corresponding to the shared texture handle in a shared texture handle manner, and performing image encoding processing based on the second texture data to obtain screen capture data.

5. The method for capturing a screen image according to claim 4, characterized in that, the data input structure of the hardware encoder includes a texture handle, and the hardware encoder obtaining the second texture data in a shared texture handle manner and performing image encoding processing based on the second texture data to obtain screen capture data comprises: the hardware encoder obtaining the second texture data in a shared texture handle manner; the hardware encoder converting the structure body of the second texture data into a set encoding structure body, and performing image encoding processing on the second texture data of the set encoding structure body to obtain screen capture data.

6. The method for capturing a screen image according to claim 1, characterized in that, after the converting the color space of the first texture data to a set color space to obtain second texture data, further comprising: sending the second texture data to a renderer, and rendering the second texture data through the renderer.

7. The method for capturing a screen image according to claim 6, characterized in that, the sending the second texture data to a renderer and rendering the second texture data through the renderer comprises: sending the shared texture handle corresponding to the second texture data to the renderer, and through the renderer obtaining the second texture data in a shared texture handle manner and rendering the second texture data.

8. The screen image acquisition method according to any one of claims 1-7, characterized in that, after converting the color space of the first texture data to a set color space to obtain second texture data, the method further includes: performing pre-processing on the second texture data, where the pre-processing includes one or a combination of more than one of color correction, cropping, scaling, upsampling, downsampling, rotation, and equal ratio alignment.

9. A screen image acquisition device, characterized in that, it includes an image acquisition module, a data conversion module, a color conversion module, and an image encoding module, where: the image acquisition module is configured to obtain a cached frame from a frame pool in a callback manner, and the cached frame is obtained by a screen acquisition tool invoking a graphics processing unit to perform screen image acquisition on a screen display image; the data conversion module is configured to convert the cached frame into first texture data; the color conversion module is configured to convert the color space of the first texture data to a set color space to obtain second texture data; the image encoding module is configured to perform image encoding processing on the second texture data through a hardware encoder to obtain screen acquisition data.

10. A screen image acquisition device, characterized in that, it includes: a memory and one or more processors; the memory is configured to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the screen image acquisition method according to any one of claims 1-8.

Images