Desktop Image Acquisition and Encoding Method and Device

Abstract

This invention relates to a desktop image acquisition and encoding method and apparatus, belonging to the field of cloud desktop technology. The specific technical solution is as follows: An independent acquisition and encoding thread is created; within the acquisition and encoding thread, the pre-configuration of the graphics card for acquisition and encoding is completed; within the acquisition and encoding thread, a cyclic acquisition and encoding process is started, sequentially executing acquisition parameter configuration, calling the graphics card acquisition interface to complete desktop image acquisition, acquisition result processing, image resource registration to the graphics card encoder, calling the graphics card encoding interface to complete image encoding, encoding result processing, and bitstream extraction and transmission operations; after the acquisition and encoding task is completed, the cyclic acquisition and encoding process is terminated, the acquisition session and encoder session are destroyed, the corresponding resources are released, and the acquisition and encoding thread is reclaimed. This disclosure can improve the desktop rendering frame rate and image acquisition frame rate.

Description

Technical Field

[0001] This invention relates to the field of cloud desktop technology, and in particular to a desktop image acquisition and encoding method and apparatus. Background Technology

[0002] The application of cloud desktop systems is becoming increasingly widespread, and their source environments are mainly divided into two categories: Windows operating systems and Linux operating systems. Among them, the source environments of Linux operating systems cover both bare metal physical machines and virtual machines.

[0003] Most Linux systems connect to Xcb objects through the X Window System to obtain the changing region Delta and the desktop image. In this approach, key operations such as desktop image rendering calculation, identification and extraction of changing regions, and reading the frame buffer of the complete desktop image must be completed independently by the CPU. This causes the CPU to simultaneously undertake the dual tasks of system scheduling and graphics data processing, consuming a large amount of computing resources and resulting in low desktop rendering frame rate and image acquisition frame rate. Summary of the Invention

[0004] The purpose of this application is to provide a desktop image acquisition and encoding method and apparatus that can improve the desktop rendering frame rate and image acquisition frame rate.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a desktop image acquisition and encoding method, which is applied to a cloud desktop system under a Linux system, including: Create a separate acquisition and encoding thread; Within the acquisition and encoding thread, the pre-configuration of the graphics card for acquisition and encoding is completed. The pre-configuration includes: configuring acquisition and encoding environment variables, obtaining the acquisition-related function interface list of the graphics card, obtaining the encoding-related function interface list of the graphics card, initializing the acquisition session, initializing the encoder session, obtaining the interface for destroying the acquisition session, and obtaining the interface for destroying the encoder session. Within the acquisition and encoding thread, a loop acquisition and encoding process is started, sequentially executing acquisition parameter configuration, calling the graphics card acquisition interface to complete desktop image acquisition, acquisition result processing, image resource registration to the graphics card encoder, calling the graphics card encoding interface to complete image encoding, encoding result processing, and bitstream extraction and transmission operations. After the acquisition and encoding task is completed, the cyclic acquisition and encoding process is terminated, the acquisition session and encoder session are destroyed, the corresponding resources are released, and the acquisition and encoding thread is reclaimed.

[0006] This disclosure creates an independent acquisition and encoding thread, integrates the entire acquisition and encoding process of the graphics card, and achieves resource isolation between the acquisition and encoding task and other system tasks. It can fully utilize the computing power of the graphics card GPU to complete the acquisition and encoding work, significantly reduce CPU resource utilization, improve the overall frame rate of acquisition and encoding, and at the same time ensure process stability and avoid frame rate fluctuations caused by resource contention.

[0007] Optionally, the configuration for collecting and encoding environment variables includes: Set the environment variables for the graphics card device path, VA-API driver name, built-in encoding parameters, and desktop capture display domain.

[0008] This disclosure clearly defines the four types of core environment variable configurations required for data acquisition and encoding, ensuring accurate matching of parameters for graphics card drivers, device paths, encoding modes, and the target desktop.

[0009] Optionally, the list of acquisition-related function interfaces of the graphics card includes: Load the encoding dynamic library corresponding to the graphics card; The pointer to the instance creation function is retrieved from the encoded dynamic library through the dynamic symbol resolution interface. By creating a pointer to the instance function, the acquisition encoding API function list is obtained and initialized to obtain the acquisition-related function interface list of the graphics card.

[0010] This disclosure uses a standardized dynamic library loading and symbol resolution process to accurately obtain the list of GPU-related function interfaces for image acquisition, providing reliable interface support for subsequent GPU-based acquisition of desktop images and ensuring the accurate execution of acquisition commands.

[0011] Optionally, the step of obtaining the list of encoding-related function interfaces for the graphics card includes: Load the encoding dynamic library corresponding to the graphics card and establish a mapping association between the current process and the encoding dynamic library; Using a dynamic symbol resolution interface, the encoding interface instance creation function is retrieved from the encoding dynamic library, and a pointer object of the encoding interface instance creation function is obtained; Based on the pointer object, the list of encoding-related functions is constructed and initialized, resulting in a list of directly callable encoding-related function interfaces.

[0012] This disclosure uses dynamic link library mapping and function pointer retrieval to construct an encoding interface set, thereby adapting the encoding function to the graphics card hardware acceleration capability, ensuring that encoding instructions directly call the GPU computing power, avoiding CPU intermediate processing, significantly improving encoding efficiency, and shortening the encoding time of a single frame image.

[0013] Optionally, the initialization of the acquisition session includes: Call the graphics card acquisition function interface to create an acquisition program handle; Obtain the graphics card acquisition status and the acquisition size and starting position information of the monitor; Create a data acquisition session and configure the acquisition range and data transmission type to direct rendering manager mode; Set the format of the image data returned by the acquisition.

[0014] This disclosure establishes and configures the acquisition session step by step, clearly defining the acquisition scope, data transmission type, and image format. This ensures precise connection between the acquisition process and the graphics card hardware, enabling direct acquisition and format standardization of desktop images. It provides unified input data for subsequent encoding processes, reducing resource consumption caused by format conversion.

[0015] Optionally, the initialization of the encoder session includes: Call the handle creation interface of the encoding function interface set, configure the encoding frame rate, the encoding resolution matching the acquisition size, and the image source type of the acquisition output to generate the encoder handle; The encoder status query interface, combined with the encoder handle, is used to detect whether the encoder is in normal working condition. Call the encoding session creation interface to create an encoding session based on a valid encoder handle, and set the encoder device type to the video hardware acceleration interface type; Call the encoding preset configuration query interface, select H265 or H264 encoding format, configure low latency high-definition preset image quality, and generate a basic encoding preset configuration object; Configure the color space format, constant bit rate control mode, and bitstream threshold parameters; Instantiate and populate the encoder initialization parameters, associate the configured color space format, constant bit rate control mode and bit stream threshold parameters, and configure the encoding core parameters; Call the encoder initialization interface, pass in the encoder handle and the initialization parameters, and complete the overall encoder initialization; The code stream buffer creation interface is invoked to generate a code stream output pointer object through the encoder handle, and the storage and recording of the code stream output pointer object are completed.

[0016] This disclosure enables precise setting of parameters such as encoding frame rate, resolution, format, and bitrate control mode through multi-dimensional encoder session initialization configuration, so as to leverage the hardware encoding advantages of the graphics card, support the encoding requirements of low latency and high definition, and generate an independent bitstream buffer to ensure the orderly storage and fast reading of the encoded bitstream data.

[0017] Optionally, the processing of the acquisition results includes: If the collected result is a graphics card frame buffer timeout error or a graphics card frame buffer interruption error, then proceed directly to the next loop. If the acquisition result is an error state indicating that the graphics card frame buffer was not successfully executed, then the acquisition encoding process will be reset and the next loop will begin. If the acquisition result indicates that the graphics card frame buffer has been successfully executed, then the image resource registration process begins.

[0018] This disclosure implements differentiated processing strategies based on different states of the collected results, skipping negligible exceptions directly and triggering process reset for erroneous states, thereby improving the fault tolerance of the collection process, avoiding the interruption of the overall process due to a single collection exception, and ensuring the continuity and stability of the collection encoding.

[0019] Optionally, the acquisition encoding process reset operation includes: sequentially destroying the encoder session, destroying the acquisition session, reinitializing the acquisition session, and reinitializing the encoder session.

[0020] This disclosure solves the problem of fault recovery in the acquisition and encoding process by quickly completing session destruction and reconstruction in abnormal state through standardized process reset operation, ensuring that the system can quickly resume normal operation after an error occurs and reducing the impact of abnormal state on the overall frame rate.

[0021] Optionally, the encoding result processing includes: If the encoding result is an error status indicating that the graphics card encoding was not successfully executed, the acquisition encoding process will be reset and the next loop will begin. If the encoding result indicates that the graphics card encoding was successfully executed, then proceed to the bit stream extraction and transmission operation; The bitstream extraction and transmission operation includes: calling the graphics card encoding bitstream locking interface to lock the bitstream buffer, copying the bitstream data to memory through the memory copy interface, and entering the next loop after completing the bitstream data transmission.

[0022] This disclosure performs differentiated processing on the encoding results, triggers process reset in error state, and efficiently completes bitstream extraction and transmission in success state, realizing the rapid export of encoded bitstream data, ensuring the timeliness of bitstream transmission, and further improving the fault tolerance of the encoding process through the process reset mechanism to ensure the stable operation of the overall process.

[0023] Secondly, this application provides a desktop image acquisition and encoding device, which is applied to a cloud desktop system under a Linux system, comprising: Create a module for creating independent acquisition and encoding threads; The configuration module is used to complete the pre-configuration of the graphics card for acquisition and encoding within the acquisition and encoding thread. The pre-configuration includes: configuring acquisition and encoding environment variables, obtaining the acquisition-related function interface list of the graphics card, obtaining the encoding-related function interface list of the graphics card, initializing the acquisition session, initializing the encoder session, obtaining the interface for destroying the acquisition session, and obtaining the interface for destroying the encoder session. The encoding module is used to start a loop acquisition and encoding process within the acquisition and encoding thread, and sequentially execute the following operations: configuring acquisition parameters, calling the graphics card acquisition interface to complete desktop image acquisition, processing acquisition results, registering image resources to the graphics card encoder, calling the graphics card encoding interface to complete image encoding, processing encoding results, and extracting and transmitting the bitstream. The termination module is used to terminate the cyclic acquisition and encoding process after the acquisition and encoding task is completed, destroy the acquisition session and encoder session, release the corresponding resources and reclaim the acquisition and encoding thread. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a desktop image acquisition and encoding method provided in an embodiment of this application. Figure 1 ; Figure 2 A flowchart illustrating a desktop image acquisition and encoding method provided in an embodiment of this application. Figure 2 ; Figure 3 A flowchart illustrating a desktop image acquisition and encoding method provided in an embodiment of this application. Figure 3 ; Figure 4 A flowchart illustrating a desktop image acquisition and encoding method provided in an embodiment of this application. Figure 4 ; Figure 5 A flowchart illustrating a desktop image acquisition and encoding method provided in an embodiment of this application. Figure 5 ; Figure 6 This is a schematic diagram of the functional modules of a desktop image acquisition and encoding device provided in another embodiment of this application.

[0026] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0028] Figure 1 This is a flowchart illustrating a desktop image acquisition and encoding method according to an embodiment of this application, as shown below. Figure 1 As shown, the method is applied to a cloud desktop system under Linux, and includes the following steps S101-S104: S101. Create an independent acquisition and encoding thread.

[0029] Isolate the data acquisition and encoding tasks from other system tasks to avoid resource contention between tasks.

[0030] This thread provides an independent runtime space for subsequent environment configuration, session initialization, and cyclic acquisition and encoding. This thread independence ensures the stability of the acquisition and encoding process, preventing frame rate fluctuations caused by interference from other system processes.

[0031] Before performing step S101, you can first install the graphics card driver and turn on the monitor.

[0032] S102. Complete the pre-configuration of the graphics card for acquisition and encoding within the acquisition and encoding thread. The pre-configuration includes: configuring the acquisition and encoding environment variables, obtaining the list of acquisition-related function interfaces of the graphics card, obtaining the list of encoding-related function interfaces of the graphics card, initializing the acquisition session, initializing the encoder session, obtaining the interface for destroying the acquisition session, and obtaining the interface for destroying the encoder session.

[0033] In one embodiment, configuring the collection encoding environment variables includes: Set the environment variables for the graphics card device path, VA-API driver name, built-in encoding parameters, and desktop capture display domain.

[0034] For example, setting the graphics card device path environment variable includes: Get and set the graphics card's JMFBC_ADAPTER environment variable: The command `lspci -nn -d 0731:* | grep -E "92|11|96" | awk '{print $1}'` retrieves the PCI device: 02:00.0. The command `sudo / dev / dri / by-path / $(ls -l / dev / dri / by-path | grep 02:00.0 | awk ' {print $11}' | grep card)` returns ` / dev / dri / card0`. Set the JMFBC_ADAPTER environment variable: setenv("JMFBC_ADAPTER", " / dev / dri / card0",1).

[0035] For example, setting the VA-API driver name environment variable includes: Set the LIBVA_DRIVER_NAME environment variable to name the graphics card driver: Set the LIBVA_DRIVER_NAME environment variable: setenv("LIBVA_DRIVER_NAME", "mwv207d",1).

[0036] For example, setting the built-in encoding parameter environment variable includes: Set the built-in encoding JMFBC_PP environment variable; Set the JMFBC_PP environment variable: setenv("JMFBC_PP", "0", 1).

[0037] For example, setting the desktop capture display domain environment variables includes: Set the environment variable for collecting data from a specified desktop display: Set the DISPLAY environment variable: setenv("DISPLAY", "0", 1).

[0038] DISPLAY is a core environment variable of the Linux graphical interface system. Setting it to 0 means capturing the default main desktop display output of the system, ensuring that the encoding module can correctly capture the screen data of the specified desktop.

[0039] In one embodiment, such as Figure 2 As shown, the process of obtaining the list of acquisition-related function interfaces for the graphics card includes the following sub-steps A1-A3: A1. Load the encoding dynamic library corresponding to the graphics card; A2. Retrieve and obtain pointers to instance creation functions from the encoded dynamic library through the dynamic symbol resolution interface; A3. Create a pointer to the function through the instance, obtain and initialize the list of acquisition encoding API functions, so as to obtain the list of acquisition-related function interfaces of the graphics card.

[0040] Specifically, the libjmfbc.so library is dynamically linked by calling the dlsym interface with the parameter "jmFBCCreateInstance", which obtains the PJMFBCCREATEINSTANCE object. Then, the JMFBC_API_FUNCTION_LIST object pFbcFn is obtained based on the PJMFBCCREATEINSTANCE object.

[0041] The libjmfbc.so shared library file is opened via the system's dynamic link interface, establishing a mapping between the process and the library file to obtain a handle to the library. Based on the library handle obtained in the previous step, the dlsym dynamic symbol resolution interface is called, specifying the resolved symbol name as jmFBCCreateInstance. The memory address of the function is located in the loaded libjmfbc.so library. After resolution, the returned function address is assigned to a PJMFBCCREATEINSTANCE type object. This object is a function pointer type, used to point to the core instance creation function within the library, and serves as the entry point for obtaining a complete list of function interfaces.

[0042] Using the obtained PJMFBCCREATEINSTANCE function pointer object, the corresponding instance creation function is called to initialize the JMFBC_API_FUNCTION_LIST type object pFbcFn. pFbcFn is a standard API function set structure for the encoding module, encapsulating all callable acquisition, encoding, and control interfaces within the library. Subsequent business logic can directly call the corresponding hardware encoding functions through this object without repeatedly parsing library symbols.

[0043] In one embodiment, obtaining the list of encoding-related function interfaces for the graphics card includes the following sub-steps B1-B3: B1. Load the encoding dynamic library corresponding to the graphics card and establish a mapping association between the current process and the encoding dynamic library; B2. Using a dynamic symbol resolution interface, retrieve the encoding interface instance creation function from the encoding dynamic library and obtain the pointer object of the encoding interface instance creation function; B3. Based on the pointer object, complete the construction and initialization of the list of encoding-related functions to obtain a list of encoding-related function interfaces that can be directly called.

[0044] Specifically, the libjmfbc.so encoding dynamic link library is loaded; the jmEncodeAPICreateInstance symbol in the library is resolved through the dlsym interface to obtain the PJMENCAPICREATEINSTANCE function pointer; the function pointer is called to obtain and initialize the JM_ENCODE_API_FUNCTION_LIST function list, resulting in a pEncFn interface collection object that can be directly called.

[0045] In one embodiment, the initialization of the acquisition session includes the following sub-steps C1-C4: C1. Call the graphics card acquisition function interface to create an acquisition program handle; C2. Obtain the graphics card acquisition status and the acquisition size and starting position information of the monitor; C3. Create a data acquisition session and configure the acquisition range and data transmission type to direct rendering manager mode; C4. Set the format of the image data returned by the acquisition.

[0046] The purpose of this step is to write the initialization capture session interface InitializeCapture(), specifically: 1. Create a data acquisition program handle by calling the API interface jmFBCCreateHandle() in pFbcFn, and obtain the JMFBC_SESSION_HANDLE object fbc_handle.

[0047] 2. Obtain the acquisition program status by calling the API interface jmFBCGetStatus() in pFbcFn and the parameter fbc_handle, including whether the graphics card supports desktop acquisition and the information of the monitor to be acquired: such as the acquisition size length & width, acquisition start position, etc.

[0048] 3. Create a capture session by calling the API interface jmFBCCreateCaptureSession() in pFbcFn and the parameter fbc_handle. Based on the information obtained in step 2, you can set the specified display to be captured, the capture range, and the capture type (JMFBC_CAPTURE_TO_DRM under Linux system), etc.

[0049] 4. Set the data format returned by the acquisition by calling the API interface jmFBCSetUp() in pFbcFn and the parameter fbc_handle. For example, JMFBC_BUFFER_FORMAT_BGRA indicates that the BGRA image type is returned.

[0050] In one embodiment, initializing the encoder session includes the following sub-steps D1-D8: D1. Call the handle creation interface of the encoding function interface set, configure the encoding frame rate, the encoding resolution matching the acquisition size, and the image source type of the acquisition output to generate the encoder handle; D2. Detect whether the encoder is in normal working condition by using the encoder status query interface in conjunction with the encoder handle; D3. Call the encoding session creation interface to create an encoding session based on a valid encoder handle, and set the encoder device type to the video hardware acceleration interface type; D4. Call the encoding preset configuration query interface, select H265 or H264 encoding format, configure low latency high-definition preset image quality, and generate a basic encoding preset configuration object. D5. Configure color space format, constant bit rate control mode, and bitstream threshold parameters; D6. Instantiate and populate the encoder initialization parameters, associate the configured color space format, constant bit rate control mode and bit stream threshold parameters, and configure the encoding core parameters. D7. Call the encoder initialization interface, pass in the encoder handle and initialization parameters, and complete the overall encoder initialization; D8. Call the bitstream buffer creation interface, generate a bitstream output pointer object through the encoder handle, and complete the storage and recording of the bitstream output pointer object.

[0051] The purpose of this step is to write the initialization encoder session interface InitializeEncoder(), specifically: 1. By calling the API interface jmEncCreateHandle() in pEncFn, you can set parameters such as encoding frame rate, encoding size (corresponding to the acquisition size), image source type JM_ENC_INPUT_RESOURCE_TYPE_DRM (corresponding to the acquisition output type JMFBC_CAPTURE_TO_DRM) to create an encoder handle and obtain a void* object enc_handle.

[0052] 2. Obtain the encoder status by calling the API interface jmEncGetStatus() in pEncFn and the parameter enc_handle to determine whether the encoder is working properly. 3. Create an encoding session by calling the API interface jmEncOpenEncodeSession() in pEncFn and the parameter enc_handle, and set the encoder device to JM_ENC_DEVICE_TYPE_VAAPI.

[0053] 4. By calling the API interface jmEncGetEncodePresetConfig() in pEncFn and the parameter enc_handle, set the encoding type to JM_ENC_CODEC_HEVC_GUID (H265 bitstream) or JM_ENC_CODEC_H264_GUID (H264 bitstream), and set the preset quality to, for example, JM_ENC_PRESET_LOW_LATENCY_HP_GUID (low latency high definition type), to obtain a set of default encoding parameters and get the JM_ENC_PRESET_CONFIG object presetConfig.

[0054] 5. Improve the JM_ENC_PRESET_CONFIG object presetConfig and fill in various parameters to be encoded, such as: color space format: JM_ENC_HEVC_PROFILE_MAIN_444_GUID (YUV444 of H265), rate control mode: CBR (constant bit rate mode), maximum bit rate, average bit rate, etc.

[0055] 6. Create a JM_ENC_INITIALIZE_PARAMS object initParams, complete the encoder parameters, set presetConfig to initParams, and set the encoding size, H265 / H264, encoding quality, etc.

[0056] 7. Initialize the encoder by calling the API interface jmEncInitializeEncoder() in pEncFn and the parameters enc_handle and initParams.

[0057] 8. By calling the API interface jmEncCreateBitstreamBuffer() in pEncFn and the parameter enc_handle, obtain the JM_ENC_CREATE_BITSTREAM_BUFFER object, get the pointer enc_bitstream_ptr of type JM_ENC_OUTPUT_PTR for the bitstream output, and record and save it.

[0058] In one embodiment, the above method further includes: writing a destruction interface UnInitializeCapture(), specifically including: Destroy the capture session by creating a JMFBC_DESTROY_CAPTURE_SESSION_PARAMS object destroyCaptureParams, and calling the system interface memset(), which in turn calls the API interface jmFBCDestroyCaptureSession() in pFbcFn with parameters fbc_handle and destroyCaptureParams to destroy the capture session.

[0059] Destroy the acquisition handle by creating a JMFBC_DESTROY_HANDLE_PARAMS object named destroyHandleParams, and calling the system interface memset(), which in turn calls the API interface jmFBCDestroyHandle() in pFbcFn with parameters fbc_handle and destroyHandleParams to destroy the acquisition handle.

[0060] In one embodiment, the above method further includes: writing the encoder session destruction interface UnInitializeEncoder(), specifically including: Release the bitstream buffer by calling the API interface jmEncDestroyBitstreamBuffer() in pEncFn, along with the parameters enc_handle and the pointer enc_bitstream_ptr of type JM_ENC_OUTPUT_PTR recorded in step 6.8.

[0061] Destroy the encoder by calling the API interface jmEncDestroyEncoder() in pEncFn and the parameter enc_handle.

[0062] Destroy the encoder handle by creating a JM_ENC_DESTROY_HANDLE_PARAMS object destroyEncHandleParams, and calling the API interface jmEncDestroyHandle() in pFbcFn with parameters fbc_handle and destroyEncHandleParams to destroy the acquisition handle.

[0063] S103. Start the loop acquisition and encoding process in the acquisition and encoding thread, and execute the acquisition parameter configuration, call the graphics card acquisition interface to complete desktop image acquisition, acquisition result processing, image resource registration to the graphics card encoder, call the graphics card encoding interface to complete image encoding, encoding result processing, and bit stream extraction and transmission operations in sequence.

[0064] In one embodiment, the configuration of the acquisition parameters includes: Create JMFBC_FRAME_GRAB_INFO acquisition information and JMFBC_GRAB_FRAME_PARAMS acquisition parameter objects, and set the specified parameters. Set the acquisition type to JMFBC_TOSYS_GRAB_FLAGS_NOFLAGS (it will only return success when there is a change in the screen, reducing resource consumption in static desktops or low frame rate scenes). The code is as follows: JMFBC_FRAME_GRAB_INFO pciInfo; JMFBC_GRAB_FRAME_PARAMS grabParams; memset(&grabParams, 0, sizeof(grabParams)); memset(&pciInfo, 0, sizeof(pciInfo)); grabParams.version = JMFBC_GRAB_FRAME_PARAMS_VER; grabParams.pFrameGrabInfo = &pciInfo; grabParams.timeoutMs = 30; grabParams.flags = JMFBC_TOSYS_GRAB_FLAGS_NOFLAGS.

[0065] In one embodiment, calling the graphics card acquisition interface to complete desktop image acquisition includes: A desktop image is captured by calling the API interface jmFBCGrabFrame() in pFbcFn along with the parameters fbc_handle and grabParams.

[0066] In one embodiment, the processing of the acquisition results includes: If the collected result is a graphics card frame buffer timeout error or a graphics card frame buffer interruption error, then proceed directly to the next loop. If the acquisition result is an error state indicating that the graphics card frame buffer was not successfully executed, then the acquisition encoding process will be reset and the next loop will begin. If the acquisition result indicates that the graphics card frame buffer has been successfully executed, then proceed to the image resource registration stage; The acquisition and encoding process reset operation includes: sequentially destroying the encoder session, destroying the acquisition session, reinitializing the acquisition session, and reinitializing the encoder session.

[0067] Specifically, determine the return value of the interface jmFBCGrabFrame(), which is of type JMFBCSTATUS, and obtain the specific return value fbcStatus.

[0068] Check the return value fbcStatus. If the return value is JMFBC_ERR_TIMEOUT (timeout) or JMFBC_ERR_INTERRUPT (interrupted), continue and execute the next acquisition action. If the return value is not JMFBC_SUCCESS, it is another error, and the entire acquisition and encoding process is reset directly. If JMFBC_SUCCESS is returned, the image was successfully acquired this time.

[0069] In one embodiment, registering the image resources to the graphics card encoder includes: Register the acquired image with the encoder. Create a JM_ENC_REGISTER_RESOURCE object registerParams, set the pciInfo obtained from the acquisition parameter configuration to registerParams, and set the registered resource type JM_ENC_INPUT_RESOURCE_TYPE_DRM and the acquired image type JM_ENC_BUFFER_FORMAT_ABGR. Then, complete the registration by calling the API interface jmEncRegisterResource() and enc_handle in pEncFn. Partial code is as follows: registerParams.resourceType = JM_ENC_INPUT_RESOURCE_TYPE_DRM; registerParams.bufferFormat = JM_ENC_BUFFER_FORMAT_ABGR; In one embodiment, the encoding result processing includes: If the encoding result is an error status indicating that the graphics card encoding was not successfully executed, the acquisition encoding process will be reset and the next loop will begin. If the encoding result indicates that the graphics card encoding was successfully executed, then proceed to the bit stream extraction and transmission operation; The bitstream extraction and transmission operation includes: calling the graphics card encoding bitstream locking interface to lock the bitstream buffer, copying the bitstream data to memory through the memory copy interface, and entering the next loop after completing the bitstream data transmission.

[0070] Specifically, the registered frame is encoded by creating an object encParams of the encoding format JM_ENC_PIC_PARAMS, setting the image size and other information to encParams, calling the API interface jmEncEncodePicture() in pEncFn with enc_handle and encParams as parameters, and then obtaining the return value encStatus of type JMENCSTATUS.

[0071] Determine the encoding return value encStatus. If it is not JM_ENC_SUCCESS, reset the entire acquisition and encoding process. If it is JM_ENC_SUCCESS, the encoding was successfully completed.

[0072] After encoding is complete, the buffer is locked first. A JM_ENC_LOCK_BITSTREAM type object lockParams is created. The JM_ENC_OUTPUT_PTR type pointer enc_bitstream_ptr recorded in the InitializeEncoder() interface for writing the encoder session is set to lockParams. Then, the API interface jmEncLockBitstream() in pEncFn is called with enc_handle and lockParams as parameters to lock the buffer. Finally, the return value encStatus of type JMENCSTATUS is obtained.

[0073] Check the encoding return value encStatus. If it is not JM_ENC_SUCCESS, continue and proceed to the next acquisition loop. If it is JM_ENC_SUCCESS, it means that the lock was successful. Then call the system interface memcpy() to copy the bitstream pointed to by lockParams.pBitstreamBufferPtr from the encoder to memory. The bitstream size is lockParams.bitstreamSizeInBytes.

[0074] The bitstream copied to memory is transmitted, the acquisition and encoding of this frame ends, and the next loop begins.

[0075] S104. After the acquisition and encoding task is completed, terminate the cyclic acquisition and encoding process, destroy the acquisition session and encoder session, release the corresponding resources and reclaim the acquisition and encoding thread.

[0076] After the overall acquisition and encoding task is completed, the cyclic acquisition and encoding process is terminated, the acquisition session and encoder session are destroyed, the corresponding resources are released, and the acquisition and encoding thread is reclaimed.

[0077] This disclosure creates an independent acquisition and encoding thread, integrates the entire acquisition and encoding process of the graphics card, realizes resource isolation between the acquisition and encoding task and other system tasks, fully utilizes the computing power of the graphics card GPU to complete the acquisition and encoding work, significantly reduces CPU resource utilization, improves the overall frame rate of acquisition and encoding, and at the same time ensures process stability and avoids frame rate fluctuations caused by resource contention.

[0078] Figures 4-5 This is a flowchart illustrating a desktop image acquisition and encoding method according to an embodiment of this application, as shown below. Figure 4-5 As shown, the method is applied to a cloud desktop system under a Linux system, including: Step 1: Install the Jingjia Micro graphics card driver and turn on the monitor.

[0079] Step 2: Write an interface to obtain and set the environment variables required for data acquisition encoding.

[0080] Get and set the graphics card's JMFBC_ADAPTER environment variable: The command `lspci -nn -d 0731:* | grep -E "92|11|96" | awk '{print $1}'` retrieves the PCI device: 02:00.0. The command `sudo / dev / dri / by-path / $(ls -l / dev / dri / by-path | grep 02:00.0 | awk ' {print $11}' | grep card)` returns ` / dev / dri / card0`. Set the JMFBC_ADAPTER environment variable: setenv("JMFBC_ADAPTER", " / dev / dri / card0",1); Set the LIBVA_DRIVER_NAME environment variable to name the graphics card driver: Set the LIBVA_DRIVER_NAME environment variable: setenv("LIBVA_DRIVER_NAME", "mwv207d",1); Set the built-in encoding JMFBC_PP environment variable: Set the JMFBC_PP environment variable: setenv("JMFBC_PP", "0", 1); Set the environment variable for collecting data from a specified desktop display: Set the DISPLAY environment variable: setenv("DISPLAY", "0", 1); Step 3: Obtain and save the JMFBC_API_FUNCTION_LIST object pFbcFn, which will be used later for creating and calling collection-related APIs.

[0081] By calling the dlsym interface to dynamically link the libjmfbc.so library with the parameter "jmFBCCreateInstance", a PJMFBCCREATEINSTANCE object is obtained. Then, based on the PJMFBCCREATEINSTANCE object, a JMFBC_API_FUNCTION_LIST object pFbcFn is obtained.

[0082] Step 4: Obtain and save the JM_ENCODE_API_FUNCTION_LIST object pEncFn, which will be used later to create and call encoder-related APIs.

[0083] By calling the dlsym interface to dynamically link the libjmfbc.so library with the parameter "jmEncodeAPICreateInstance", a PJMENCAPICREATEINSTANCE object is obtained. Then, based on the PJMENCAPICREATEINSTANCE object, a JM_ENCODE_API_FUNCTION_LIST object pEncFn is obtained.

[0084] Step 5: Write the initialization capture session interface InitializeCapture().

[0085] By calling the API interface jmFBCCreateHandle() in pFbcFn, a collection program handle is created, and a JMFBC_SESSION_HANDLE object fbc_handle is obtained.

[0086] The acquisition program status can be obtained by calling the API interface jmFBCGetStatus() in pFbcFn with the parameter fbc_handle. This includes whether the graphics card supports desktop acquisition and the information of the monitor to be acquired, such as the acquisition size (length & width), acquisition start position, etc.

[0087] A capture session is created by calling the API interface jmFBCCreateCaptureSession() in pFbcFn and the parameter fbc_handle. Based on the information obtained in step 2, you can set the specified display to be captured, the capture range, and the capture type (JMFBC_CAPTURE_TO_DRM under Linux system), etc.

[0088] The data format returned by the acquisition can be set by calling the API interface jmFBCSetUp() in pFbcFn and the parameter fbc_handle. For example, JMFBC_BUFFER_FORMAT_BGRA indicates that the BGRA image type is returned.

[0089] Step 6: Write the initialization encoder session interface InitializeEncoder().

[0090] By calling the API interface jmEncCreateHandle() in pEncFn, you can set parameters such as encoding frame rate, encoding size (corresponding to the acquisition size), image source type JM_ENC_INPUT_RESOURCE_TYPE_DRM (corresponding to the acquisition output type JMFBC_CAPTURE_TO_DRM), etc., to create an encoder handle and obtain a void* object enc_handle.

[0091] The encoder status can be obtained by calling the API interface jmEncGetStatus() in pEncFn and the parameter enc_handle to determine whether the encoder is working properly. An encoding session is created by calling the API interface jmEncOpenEncodeSession() in pEncFn with the parameter enc_handle, and the encoder device is set to JM_ENC_DEVICE_TYPE_VAAPI.

[0092] By calling the API interface jmEncGetEncodePresetConfig() in pEncFn and the parameter enc_handle, setting the encoding type to JM_ENC_CODEC_HEVC_GUID (H265 bitrate) or JM_ENC_CODEC_H264_GUID (H264 bitrate), and setting the preset quality to, for example, JM_ENC_PRESET_LOW_LATENCY_HP_GUID (low latency high definition), a set of default encoding parameters is obtained to get the JM_ENC_PRESET_CONFIG object presetConfig.

[0093] Complete the JM_ENC_PRESET_CONFIG object presetConfig and fill in various parameters you want to encode, such as: color space format: JM_ENC_HEVC_PROFILE_MAIN_444_GUID (YUV444 for H265), rate control mode: CBR (constant bit rate mode), maximum bit rate, average bit rate, etc.

[0094] Create a JM_ENC_INITIALIZE_PARAMS object initParams, complete the encoder parameters, set presetConfig to initParams, and set the encoding size, H265 / H264, encoding quality, etc.

[0095] The encoder is initialized by calling the API interface jmEncInitializeEncoder() in pEncFn along with the parameters enc_handle and initParams.

[0096] By calling the API interface jmEncCreateBitstreamBuffer() in pEncFn and the parameter enc_handle to obtain the JM_ENC_CREATE_BITSTREAM_BUFFER object, the pointer enc_bitstream_ptr of type JM_ENC_OUTPUT_PTR for the bitstream output is obtained and recorded.

[0097] Step 7: Write the UnInitializeCapture() interface to destroy the capture session.

[0098] Destroy the capture session by creating a JMFBC_DESTROY_CAPTURE_SESSION_PARAMS object destroyCaptureParams, and calling the system interface memset(), which in turn calls the API interface jmFBCDestroyCaptureSession() in pFbcFn with parameters fbc_handle and destroyCaptureParams to destroy the capture session.

[0099] Destroy the acquisition handle by creating a JMFBC_DESTROY_HANDLE_PARAMS object named destroyHandleParams, and calling the system interface memset(), which in turn calls the API interface jmFBCDestroyHandle() in pFbcFn with parameters fbc_handle and destroyHandleParams to destroy the acquisition handle.

[0100] Step 8: Write the UnInitializeEncoder() interface to destroy the encoder session.

[0101] Release the bitstream buffer by calling the API interface jmEncDestroyBitstreamBuffer() in pEncFn, along with the parameters enc_handle and the pointer enc_bitstream_ptr of type JM_ENC_OUTPUT_PTR recorded in step 6.8.

[0102] Destroy the encoder by calling the API interface jmEncDestroyEncoder() in pEncFn and the parameter enc_handle.

[0103] Destroy the encoder handle by creating a JM_ENC_DESTROY_HANDLE_PARAMS object destroyEncHandleParams, and calling the API interface jmEncDestroyHandle() in pFbcFn with parameters fbc_handle and destroyEncHandleParams to destroy the acquisition handle.

[0104] Step 9: Create a capture and encoding thread (e.g., ...) Figure 5 (As shown).

[0105] The thread first executes steps 2, 3, and 4, then calls steps 5 and 6, and finally creates a loop.

[0106] Within the loop, the initial part creates JMFBC_FRAME_GRAB_INFO acquisition information and JMFBC_GRAB_FRAME_PARAMS acquisition parameter objects, and sets the parameters. The acquisition type is set to JMFBC_TOSYS_GRAB_FLAGS_NOFLAGS (success is only returned when there is a change in the screen, reducing resource consumption in static desktops or low frame rate scenes). The code is as follows: JMFBC_FRAME_GRAB_INFO pciInfo; JMFBC_GRAB_FRAME_PARAMS grabParams; memset(&grabParams, 0, sizeof(grabParams)); memset(&pciInfo, 0, sizeof(pciInfo)); grabParams.version = JMFBC_GRAB_FRAME_PARAMS_VER; grabParams.pFrameGrabInfo = &pciInfo; grabParams.timeoutMs = 30; grabParams.flags = JMFBC_TOSYS_GRAB_FLAGS_NOFLAGS; A desktop image is captured by calling the API interface jmFBCGrabFrame() in pFbcFn along with the parameters fbc_handle and grabParams.

[0107] Determine the return value of the interface jmFBCGrabFrame(), which is of type JMFBCSTATUS, and obtain the specific return value fbcStatus.

[0108] Check the return value fbcStatus. If the return value is JMFBC_ERR_TIMEOUT (timeout) or JMFBC_ERR_INTERRUPT (interrupted), continue and execute the next acquisition action. If the return value is not JMFBC_SUCCESS, it is another error, and the entire acquisition and encoding process is reset directly (call the interfaces of steps 7 and 8, and then call steps 5 and 6 again). If JMFBC_SUCCESS is returned, the image was successfully acquired this time.

[0109] Register the acquired image with the encoder. Create a JM_ENC_REGISTER_RESOURCE object registerParams, set the pciInfo obtained in step 9.2 to registerParams, and set the registered resource type JM_ENC_INPUT_RESOURCE_TYPE_DRM and the acquired image type JM_ENC_BUFFER_FORMAT_ABGR. Then, complete the registration by calling the API interface jmEncRegisterResource() and enc_handle in pEncFn. Part of the code is as follows: registerParams.resourceType = JM_ENC_INPUT_RESOURCE_TYPE_DRM; registerParams.bufferFormat = JM_ENC_BUFFER_FORMAT_ABGR; Encode the registered frame, create an object encParams of the format JM_ENC_PIC_PARAMS for this encoding, set the image size and other information to encParams, call the API interface jmEncEncodePicture() in pEncFn with enc_handle and encParams as parameters, and then get the return value encStatus of type JMENCSTATUS.

[0110] Determine the encoding return value encStatus. If it is not JM_ENC_SUCCESS, reset the entire acquisition and encoding process. If it is JM_ENC_SUCCESS, the encoding was successfully completed.

[0111] After encoding is complete, first lock the buffer, create a JM_ENC_LOCK_BITSTREAM type object lockParams, set the JM_ENC_OUTPUT_PTR type pointer enc_bitstream_ptr recorded in step 6, 8 to lockParams, and then call the API interface jmEncLockBitstream() in pEncFn with enc_handle and lockParams as parameters to lock it, and then get the return value encStatus of type JMENCSTATUS.

[0112] Check the encoding return value encStatus. If it is not JM_ENC_SUCCESS, continue and proceed to the next acquisition loop. If it is JM_ENC_SUCCESS, it means that the lock was successful. Then call the system interface memcpy() to copy the bitstream pointed to by lockParams.pBitstreamBufferPtr from the encoder to memory. The bitstream size is lockParams.bitstreamSizeInBytes.

[0113] The bitstream copied to memory is transmitted, the acquisition and encoding of this frame ends, and the next loop is executed.

[0114] Step 10: End the data collection process.

[0115] 1. Exit the loop, and call steps 7 and 8 to release resources, join the thread, and the process ends.

[0116] The embodiments provided in this disclosure solve the problems of low frame rate and high latency caused by X acquisition in the Jingjia micro graphics card environment under the prior art, compared with the prior art. By making full use of the GPU's capabilities and reducing the CPU load, the acquisition frame rate is improved and the latency is reduced, so as to achieve a better user experience after the image is zoomed out.

[0117] Based on the same inventive concept, this application also provides a desktop image acquisition and encoding device for implementing the desktop image acquisition and encoding method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more desktop image acquisition and encoding device embodiments provided below can be found in the limitations of the desktop image acquisition and encoding method described above, and will not be repeated here.

[0118] In one exemplary embodiment, such as Figure 6 As shown, a desktop image acquisition and encoding device is provided. The device is applied to a cloud desktop system under Linux and includes: Create module 11 to create an independent acquisition and encoding thread; The configuration module 12 is used to complete the pre-configuration of the graphics card for acquisition and encoding within the acquisition and encoding thread. The pre-configuration includes: configuring acquisition and encoding environment variables, obtaining the acquisition-related function interface list of the graphics card, obtaining the encoding-related function interface list of the graphics card, initializing the acquisition session, initializing the encoder session, obtaining the interface for destroying the acquisition session, and obtaining the interface for destroying the encoder session. Encoding module 13 is used to start a loop acquisition and encoding process within the acquisition and encoding thread, and sequentially execute acquisition parameter configuration, call the graphics card acquisition interface to complete desktop image acquisition, acquisition result processing, image resource registration to the graphics card encoder, call the graphics card encoding interface to complete image encoding, encoding result processing, and bit stream extraction and transmission operations. Termination module 14 is used to terminate the cyclic acquisition and encoding process, destroy the acquisition session and encoder session, release the corresponding resources and reclaim the acquisition and encoding thread after the acquisition and encoding task is completed.

[0119] In one embodiment, regarding the configuration of the acquisition encoding environment variables, the configuration module 12 is specifically used for: Set the environment variables for the graphics card device path, VA-API driver name, built-in encoding parameters, and desktop capture display domain.

[0120] In one embodiment, regarding the acquisition of the list of acquisition-related function interfaces of the graphics card, the configuration module 12 is specifically used for: Load the encoding dynamic library corresponding to the graphics card; The pointer to the instance creation function is retrieved from the encoded dynamic library through the dynamic symbol resolution interface. By creating a pointer to the instance function, the acquisition encoding API function list is obtained and initialized to obtain the acquisition-related function interface list of the graphics card.

[0121] In one embodiment, regarding the acquisition of the graphics card's encoding-related function interface list, the configuration module 12 is specifically used for: Load the encoding dynamic library corresponding to the graphics card and establish a mapping association between the current process and the encoding dynamic library; Using a dynamic symbol resolution interface, the encoding interface instance creation function is retrieved from the encoding dynamic library, and a pointer object of the encoding interface instance creation function is obtained; Based on the pointer object, the list of encoding-related functions is constructed and initialized, resulting in a list of directly callable encoding-related function interfaces.

[0122] In one embodiment, regarding the initialization of the acquisition session, the configuration module 12 is specifically configured to: Call the graphics card acquisition function interface to create an acquisition program handle; Obtain the graphics card acquisition status and the acquisition size and starting position information of the monitor; Create a data acquisition session and configure the acquisition range and data transmission type to direct rendering manager mode; Set the format of the image data returned by the acquisition.

[0123] In one embodiment, regarding the initialization of the encoder session, the configuration module 12 is specifically configured to: Call the handle creation interface of the encoding function interface set, configure the encoding frame rate, the encoding resolution matching the acquisition size, and the image source type of the acquisition output to generate the encoder handle; The encoder status query interface, combined with the encoder handle, is used to detect whether the encoder is in normal working condition. Call the encoding session creation interface to create an encoding session based on a valid encoder handle, and set the encoder device type to the video hardware acceleration interface type; Call the encoding preset configuration query interface, select H265 or H264 encoding format, configure low latency high-definition preset image quality, and generate a basic encoding preset configuration object; Configure the color space format, constant bit rate control mode, and bitstream threshold parameters; Instantiate and populate the encoder initialization parameters, associate the configured color space format, constant bit rate control mode and bit stream threshold parameters, and configure the encoding core parameters; Call the encoder initialization interface, pass in the encoder handle and the initialization parameters, and complete the overall encoder initialization; The code stream buffer creation interface is invoked to generate a code stream output pointer object through the encoder handle, and the storage and recording of the code stream output pointer object are completed.

[0124] In one embodiment, in terms of processing the acquisition results, the encoding module 13 is specifically used for: If the collected result is a graphics card frame buffer timeout error or a graphics card frame buffer interruption error, then proceed directly to the next loop. If the acquisition result is an error state indicating that the graphics card frame buffer was not successfully executed, then the acquisition encoding process will be reset and the next loop will begin. If the acquisition result indicates that the graphics card frame buffer has been successfully executed, then the image resource registration process begins.

[0125] In one embodiment, regarding the reset operation of the acquisition encoding process, the encoding module 13 specifically uses: Destroy the encoder session, destroy the acquisition session, reinitialize the acquisition session, and reinitialize the encoder session in sequence.

[0126] In one embodiment, in terms of processing the encoded result, the encoding module 13 is specifically used for: If the encoding result is an error status indicating that the graphics card encoding was not successfully executed, the acquisition encoding process will be reset and the next loop will begin. If the encoding result indicates that the graphics card encoding was successfully executed, then proceed to the bit stream extraction and transmission operation; The bitstream extraction and transmission operation includes: calling the graphics card encoding bitstream locking interface to lock the bitstream buffer, copying the bitstream data to memory through the memory copy interface, and entering the next loop after completing the bitstream data transmission.

[0127] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a desktop image acquisition and encoding method.

[0128] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0129] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0130] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0132] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0133] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A desktop image acquisition and encoding method, characterized in that, The method is applied to a cloud desktop system under a Linux system, and comprises the following steps: An independent acquisition and coding thread is created; A front-end configuration of a graphics card in the acquisition and coding thread is completed, and the front-end configuration comprises the following steps: configuring an acquisition and coding environment variable, obtaining a list of acquisition-related function interfaces of the graphics card, obtaining a list of coding-related function interfaces of the graphics card, initializing an acquisition session, initializing an encoder session, obtaining an interface for destroying the acquisition session, and obtaining an interface for destroying the encoder session; A loop acquisition and coding process is started in the acquisition and coding thread, and the loop acquisition and coding process comprises the following steps in sequence: acquisition parameter configuration, calling a graphics card acquisition interface to complete desktop image acquisition, acquisition result processing, image resource registration to a graphics card encoder, calling a graphics card coding interface to complete image coding, coding result processing, and code stream extraction and transmission operation; After the acquisition and coding task is completed, the loop acquisition and coding process is terminated, the acquisition session and the encoder session are destroyed, corresponding resources are released, and the acquisition and coding thread is recycled.

2. The method of claim 1, wherein, The configuration of the acquisition and coding environment variable comprises the following steps: A graphics card device path environment variable is set, a VA-API driver name environment variable is set, a built-in coding parameter environment variable is set, and a desktop acquisition display domain environment variable is set.

3. The method of claim 1, wherein, The obtaining of the list of acquisition-related function interfaces of the graphics card comprises the following steps: A coding dynamic library corresponding to the graphics card is loaded; A pointer of an instance creation function is retrieved and obtained from the coding dynamic library through a dynamic symbol analysis interface; The list of acquisition and coding API functions is obtained and initialized through the pointer of the instance creation function, so as to obtain the list of acquisition-related function interfaces of the graphics card.

4. The method of claim 1, wherein, The obtaining of the list of coding-related function interfaces of the graphics card comprises the following steps: The coding dynamic library corresponding to the graphics card is loaded, and a mapping association between a current process and the coding dynamic library is established; An encoding interface instance creation function is retrieved from the coding dynamic library through a dynamic symbol analysis interface, and a pointer object of the encoding interface instance creation function is obtained; Based on the pointer object, the list of coding-related functions is constructed and initialized, so as to obtain the list of directly callable coding-related function interfaces.

5. The method of claim 1, wherein, The initialization of the acquisition session comprises the following steps: A graphics card acquisition function interface is called to create an acquisition program handle; Graphics card acquisition status and acquisition size and starting position information of a display are obtained; An acquisition session is created, and an acquisition range and an acquisition data transmission type are configured as a direct rendering manager mode; An image data format returned by acquisition is set.

6. The method of claim 1, wherein, The initialization of the encoder session comprises the following steps: A handle of a coding function interface set is called to create an interface, a coding frame rate, a coding resolution matching the acquisition size, and an image source type matching acquisition output are configured, and an encoder handle is generated; Whether the encoder is in a normal working state is detected through an encoder state query interface in combination with the encoder handle; An encoding session creation interface is called, an encoding session is created based on the valid encoder handle, and an encoder device type is set as a video hardware acceleration interface type; An encoding preset configuration query interface is called, an H265 or H264 coding format is selected, a low-delay high-definition class preset image quality is configured, and a basic encoding preset configuration object is generated; Configure color space format, constant bit rate control mode and bitstream threshold parameters; Instance and fill in the encoder initialization parameters, and associate the configured color space format, constant bit rate control mode and bitstream threshold parameters, and configure the encoding core parameters; Call the encoder initialization interface, pass in the encoder handle and the initialization parameters, and complete the overall initialization of the encoder; Call the bitstream buffer creation interface to generate a bitstream output pointer object through the encoder handle, and complete the storage and recording of the bitstream output pointer object.

7. The method of claim 1, wherein, The acquisition result processing includes: If the acquisition result is a graphics card frame buffer timeout error state or a graphics card frame buffer interruption error state, directly enter the next loop; If the acquisition result is a non-graphics card frame buffer execution success error state, perform an acquisition and encoding process reset operation and then enter the next loop; If the acquisition result is a graphics card frame buffer execution success state, enter the image resource registration link.

8. The method of claim 7, wherein, The acquisition and encoding process reset operation includes: destroying the encoder session, destroying the acquisition session, reinitializing the acquisition session, and reinitializing the encoder session.

9. The method of claim 1, wherein, The encoding result processing includes: If the encoding result is a non-graphics card encoding execution success error state, perform an acquisition and encoding process reset operation and then enter the next loop; If the encoding result is a graphics card encoding execution success state, enter the bitstream extraction and transmission operation; The bitstream extraction and transmission operation includes: calling the graphics card encoding bitstream locking interface to lock the bitstream buffer, copying the bitstream data to the memory through the memory copy interface, and entering the next loop after completing the bitstream data transmission.

10. A tabletop image acquisition encoding apparatus, characterized by The device is applied to a cloud desktop system under a Linux system, and includes: A creation module configured to create an independent acquisition and encoding thread; A configuration module configured to complete the acquisition and encoding front-end configuration of a graphics card in the acquisition and encoding thread, wherein the front-end configuration includes: configuring acquisition and encoding environment variables, obtaining a list of acquisition-related function interfaces of the graphics card, obtaining a list of encoding-related function interfaces of the graphics card, initializing an acquisition session, initializing an encoder session, obtaining a destroy acquisition session interface, and obtaining a destroy encoder session interface; An encoding module configured to start a loop acquisition and encoding process in the acquisition and encoding thread, and sequentially perform acquisition parameter configuration, call a graphics card acquisition interface to complete desktop image acquisition, acquisition result processing, image resource registration to a graphics card encoder, call a graphics card encoding interface to complete image encoding, encoding result processing, and bitstream extraction and transmission operation; A termination module configured to terminate the loop acquisition and encoding process after the acquisition and encoding task is completed, destroy the acquisition session and the encoder session, release the corresponding resources, and recycle the acquisition and encoding thread.

Images