Desktop screenshot data processing method, device and equipment of interactive classroom, and medium

Abstract

The application discloses a kind of interactive classroom desktop screenshot data processing method, device, equipment and medium, and the application belongs to computer technical field.The method comprises: initializing desktop screenshot to obtain the first frame screenshot image of successful screenshot;The first frame screenshot image is mapped from the video memory of GPU to CPU, and is copied to the cache of CPU, and the screenshot data is read from the cache of CPU for initial frame encoding by GPU;Whether the current frame desktop screenshot is successfully screened is identified;If current frame screenshot fails, the screenshot data is read from the cache of CPU for current frame encoding by GPU.The technical solution can avoid the problem of long data processing time and large memory occupation caused by multiple mapping of screenshot data, improve the stability of screenshot frame rate and the efficiency of screenshot data processing.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a method, apparatus, device, and medium for processing desktop screenshot data in an interactive classroom. Background Technology

[0002] With the development of live-streaming classes, more and more education systems are using live streaming for course delivery and classroom interaction. During live-streaming, the teacher's presentation is mainly transmitted to the student's end via desktop screenshots. To improve the student's interactive experience, higher demands are being placed on the processing of desktop screenshot data on the teacher's end.

[0003] In existing technologies, the main method for processing desktop screenshots on the teacher's end involves continuously taking screenshots of the teacher's teaching device desktop at preset time intervals. Whether or not to render the screenshot image depends on whether a secondary screen is present on the teacher's end. The screenshot data is then mapped from the GPU to the CPU for encoding and storage, before being sent to the student's end. However, processing desktop screenshots on the teacher's end using existing technologies suffers from unstable screenshot frame rates, numerous data mapping operations, and low processing efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, and medium for processing desktop screenshot data in an interactive classroom. This method can solve the problems of unstable screenshot frame rate, excessive number of screenshot data mappings, and low screenshot data processing efficiency. By mapping the first frame screenshot image from the GPU's video memory and caching it in the CPU for the GPU to perform initial frame encoding, the method identifies whether the current frame desktop screenshot was successfully captured. If the current frame screenshot fails, the screenshot data is read from the CPU's cache for the GPU to perform current frame encoding. This avoids the problems of long data processing time and excessive memory consumption caused by multiple mappings of screenshot data, thereby improving the stability of the screenshot frame rate and the efficiency of screenshot data processing.

[0005] In a first aspect, embodiments of this application provide a method for processing desktop screenshot data in an interactive classroom, the method comprising:

[0006] Initialize the desktop screenshot function to capture the first frame of the screenshot after a successful capture.

[0007] The first frame screenshot image is mapped from the GPU's video memory to the CPU and copied to the CPU's cache. The screenshot data is read from the CPU's cache for the GPU to perform initial frame encoding.

[0008] Determine whether the current frame of the desktop screenshot was successfully captured;

[0009] If the current frame fails to capture the screen, the screenshot data is read from the CPU cache and used by the GPU to encode the current frame.

[0010] Furthermore, after identifying whether the current frame desktop screenshot was successfully captured, the method further includes:

[0011] If the current frame capture is successful, the mapping of the previous frame's screenshot image from the GPU's video memory to the CPU is closed, and the mapping of the current frame's screenshot image from the GPU's video memory to the CPU is established. The current frame's screenshot image is then copied to the CPU's cache.

[0012] Screenshot data is read from the CPU cache and used by the GPU for current frame encoding.

[0013] Furthermore, after successfully identifying the current frame as a screenshot, the method further includes:

[0014] Identify whether local rendering of the current frame screenshot is required;

[0015] If local rendering is required, the current frame screenshot image will be rendered locally on the GPU using a preset method.

[0016] Furthermore, before establishing the mapping from the GPU's video memory to the CPU for the current frame screenshot image, the method further includes:

[0017] Identify whether scaling of the current frame screenshot is required;

[0018] If scaling is required, the current frame screenshot image is scaled to the target size, and then the operation of establishing the mapping of the current frame screenshot image from the GPU's video memory to the CPU is performed.

[0019] Furthermore, the scaling process is performed in the GPU;

[0020] After scaling, the current frame screenshot is converted into the target data format.

[0021] Furthermore, after reading screenshot data from the CPU cache for the GPU to perform current frame encoding, the method further includes:

[0022] Identify whether to stop screenshotting;

[0023] If so, release the GPU memory and CPU cache occupied by the screenshot data, and end the desktop screenshot data processing flow.

[0024] Furthermore, it identifies whether the current frame of the desktop screenshot was successfully captured, including:

[0025] If the device driver detects changes in input or output data, the screenshot is considered successful.

[0026] If no changes in input or output data are detected by the device driver, the screenshot is considered to have failed.

[0027] Secondly, embodiments of this application provide a desktop screenshot data processing device for an interactive classroom, the device comprising:

[0028] The screenshot initialization module is used to initialize the desktop screenshot in order to obtain the first frame of the screenshot image after a successful screenshot.

[0029] The screenshot image mapping module is used to map the first frame screenshot image from the GPU's video memory to the CPU, copy it to the CPU's cache, and read the screenshot data from the CPU's cache for the GPU to perform initial frame encoding.

[0030] The screenshot recognition module is used to identify whether the current frame of the desktop screenshot was successfully captured;

[0031] The screenshot data encoding module is used to read screenshot data from the CPU cache and provide it to the GPU for encoding the current frame if the current frame fails to be captured.

[0032] Furthermore, the screenshot data encoding module is also used for:

[0033] If the current frame capture is successful, the mapping of the previous frame's screenshot image from the GPU's video memory to the CPU is closed, and the mapping of the current frame's screenshot image from the GPU's video memory to the CPU is established. The current frame's screenshot image is then copied to the CPU's cache.

[0034] Screenshot data is read from the CPU cache and used by the GPU for current frame encoding.

[0035] Furthermore, the screenshot data encoding module also includes:

[0036] The local rendering recognition unit is used to identify whether the current frame screenshot needs to be rendered locally;

[0037] The screenshot image rendering unit is used to perform local rendering of the current frame screenshot image on the GPU using a preset method if local rendering is required.

[0038] Furthermore, the screenshot data encoding module also includes:

[0039] The scaling recognition unit is used to identify whether the current frame screenshot image needs to be scaled.

[0040] The screenshot image scaling unit is used to scale the current frame screenshot image to the target size if scaling processing is required, and then perform the operation of establishing the mapping of the current frame screenshot image from the GPU's video memory to the CPU.

[0041] Furthermore, the scaling process of the screenshot image scaling unit is performed in the GPU;

[0042] The screenshot image scaling unit is also used to convert the current frame screenshot image into a target data format after scaling.

[0043] Furthermore, the screenshot data encoding module is also used for:

[0044] Identify whether to stop screenshotting;

[0045] If so, release the GPU memory and CPU cache occupied by the screenshot data, and end the desktop screenshot data processing flow.

[0046] Furthermore, the screenshot recognition module is specifically used for:

[0047] If the device driver detects changes in input or output data, the screenshot is considered successful.

[0048] If no changes in input or output data are detected by the device driver, the screenshot is considered to have failed.

[0049] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0050] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0051] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0052] In this embodiment, the desktop screenshot is initialized to obtain the first frame screenshot image of a successful screenshot; the first frame screenshot image is mapped from the GPU's video memory to the CPU and copied to the CPU's cache; screenshot data is read from the CPU's cache for the GPU to perform initial frame encoding; it is determined whether the current frame desktop screenshot was successful; if the current frame screenshot failed, screenshot data is read from the CPU's cache for the GPU to perform current frame encoding. This interactive classroom desktop screenshot data processing method solves the problems of unstable screenshot frame rate, numerous screenshot data mappings, and low screenshot data processing efficiency. By mapping the first frame screenshot image from the GPU's video memory and caching it to the CPU for initial frame encoding, and determining whether the current frame desktop screenshot was successful, and if the current frame screenshot failed, reading screenshot data from the CPU's cache for the GPU to perform current frame encoding, the problem of long data processing time and high memory consumption caused by multiple screenshot data mappings can be avoided, thus improving the stability of the screenshot frame rate and the efficiency of screenshot data processing. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the desktop screenshot data processing method for interactive classrooms provided in Embodiment 1 of this application;

[0054] Figure 2 This is a flowchart illustrating the desktop screenshot data processing method for interactive classrooms provided in Embodiment 2 of this application;

[0055] Figure 3 This is a schematic diagram of the desktop screenshot data processing device for interactive classrooms provided in Embodiment 3 of this application;

[0056] Figure 4 This is a schematic diagram of the structure of the electronic device provided in Embodiment 4 of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0058] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0059] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0060] The desktop screenshot data processing method, apparatus, device, and medium for interactive classrooms provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0061] Example 1

[0062] Figure 1 This is a flowchart illustrating the desktop screenshot data processing method for an interactive classroom provided in Embodiment 1 of this application. Figure 1 As shown, the specific steps include the following:

[0063] S101, Initialize desktop screenshot to obtain the first screenshot image after successful screenshot;

[0064] Firstly, this solution can be used in interactive classrooms where desktop screenshot data processing is required. Specifically, it can be used for processing DXGI (DirectX Graphics Infrastructure) screenshot data on the interactive classroom live streaming device. In this scenario, considering the current device environment, the interactive classroom live streaming device needs to have a graphics card driver installed and the operating system to be Windows 8 or later. By taking screenshots of the desktop image on the live streaming device using DXGI, mapping and caching the screenshot data in the CPU (Central Processing Unit), and then having the GPU (Graphics Processing Unit) encode the screenshot data of the current frame, the efficiency of screenshot data processing can be improved.

[0065] Based on the above usage scenarios, it is understood that the executing entity of this application may be software or system platform installed on the interactive classroom live streaming device that can take screenshots of the device desktop image and send and receive screenshot data, etc., without further limitation here.

[0066] In this solution, DXGI can be used for screen capture and recording tasks such as desktop live streaming and desktop recording. Specifically, the main process of using DXGI for screen capture involves creating a D3DDevice; obtaining the path through a series of interfaces to acquire the IDXGIOutputDuplication interface; calling AcquireNextFrame to obtain the current desktop data and storing it in IDXGIResource; mapping the data from the GPU to memory; and copying the required data to its own buffer. Because DXGI provides desktop images through the GPU, it has a very low CPU utilization rate and features high performance and speed. The DXGI described runs on interactive classroom equipment systems running Windows 8 and above.

[0067] In this solution, initializing the desktop screenshot can be achieved by using DXGI to capture the initial desktop image when the interactive classroom device starts running and collecting the screenshot data. Specifically, the initial desktop image can be the first frame image on the teacher's device screen when it starts running. This first frame image can be an image displayed on the teacher's device screen according to pre-set parameters, such as screen layout, course PPT, classroom blackboard writing, and course files, etc., without further limitations. If the desktop screenshot initialization operation is successful, that is, the initial desktop image when the interactive classroom device starts running is captured, the screenshot image is stored, thus obtaining the first frame screenshot image that was successfully captured; if the initialization operation fails, that is, the screenshot operation cannot be completed, the screenshot can be retried or the system screenshot operation can be stopped and the system can report a screenshot error. The screenshot image can be stored and transmitted in the form of screenshot data.

[0068] S102, the first frame screenshot image is mapped from the GPU's video memory to the CPU and copied to the CPU's cache. The screenshot data is read from the CPU's cache for the GPU to perform initial frame encoding.

[0069] In this solution, a GPU is a microprocessor specifically designed for image and graphics-related computations in personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). GPUs reduce the reliance of graphics cards on the CPU and perform some of the tasks originally handled by the CPU, especially in 3D graphics processing. The core technologies employed by GPUs include hardware T&L (geometry transformation and lighting processing), cubic environment mapping and vertex blending, texture compression and bump mapping, and a dual-texture four-pixel 256-bit rendering engine. Hardware T&L technology can be considered a hallmark of GPUs. A raster display system cannot function without a graphics processor (GPU). The GPU is a crucial component of the graphics system architecture, serving as the link between the computer and the display terminal. It not only stores graphics but also performs most graphics functions, thus significantly reducing the CPU's workload and improving display capabilities and speed.

[0070] The graphics processor consists of the following components: the core of the graphics card, commonly known as the GPU, whose main task is to construct and render the video information input by the system; the display buffer memory used to store the graphics information to be displayed and to save the intermediate data of graphics calculation; the size and speed of the display buffer directly affect the performance of the main chip; and the RAM / D / A converter converts binary numbers into analog signals that are compatible with the display.

[0071] In this scheme, the CPU, as the core of the computer system's computation and control, is the final execution unit for information processing and program execution, one of the main devices of an electronic computer, and a core component of the computer. The CPU is the core component of the computer responsible for reading instructions, decoding instructions, and executing instructions. It mainly consists of two parts: the control unit and the arithmetic logic unit (ALU), including high-speed cache memory and the data and control buses that connect them. Its main functions are processing instructions, executing operations, controlling timing, and processing data. In computer architecture, the CPU is the core hardware unit that controls and allocates all hardware resources (such as memory and input / output units) of the computer and performs general-purpose calculations. The CPU is the core of the computer's computation and control. All software layer operations in the computer system are ultimately mapped to CPU operations through the instruction set. The CPU's operating principle can be divided into four stages: Fetch, Decode, Execute, and Writeback. The CPU fetches instructions from memory or cache memory, places them in the instruction register, decodes the instructions, and executes them. Generally speaking, the CPU structure can be roughly divided into an arithmetic logic unit, a register unit, and a control unit. The arithmetic logic unit (ALU) is primarily responsible for performing logical operations, such as shift operations and other logical operations. It can also perform fixed-point or floating-point arithmetic operations, address calculations, and conversions, making it a multi-functional arithmetic unit. The register unit, on the other hand, is used to temporarily store instructions, data, and addresses. The control unit is mainly used to analyze instructions and issue corresponding control signals.

[0072] In this solution, the GPU's video memory, also called the frame buffer, is used to store rendering data processed or about to be retrieved by the graphics card chip. In this solution, it can be used to store the screenshot data of the first frame screenshot image. The CPU cache can be a temporary storage device located between the CPU and main memory. Its capacity is smaller than main memory, but its exchange speed is much faster. The data in the CPU cache is a small part of the data in main memory, but this small part is what the CPU will access in a short time. When the CPU needs to access a large amount of data, it can bypass main memory and directly access the data from the CPU cache, thereby speeding up the reading speed. In this solution, the CPU cache can store the screenshot data that needs to be read. The screenshot data can be the image data of the first frame screenshot image, such as pixel values, grayscale values, and image size, which can be obtained by calling the encoding program through DXGI. Mapping the first frame screenshot image from the GPU's video memory to the CPU can be done by establishing a mapping table that stores the video memory address of the first frame screenshot image in the GPU and the address of the CPU. The CPU can read the first frame screenshot image in the GPU according to the mapping table. The screenshot data of the first frame screenshot image mapped to the CPU address is copied to the CPU cache through the corresponding interface, and the screenshot data is read from the CPU cache through encoding for the GPU to perform initial frame encoding. The initial frame can be the first frame screenshot image.

[0073] S103, identify whether the current frame desktop screenshot was successfully captured;

[0074] In this solution, the identification of whether a screenshot was successful can be achieved by identifying whether a desktop screenshot operation was performed on the current frame's desktop image using DXGI. If the current frame's desktop image is exactly the same as the screenshot image of the previous frame, the screenshot operation is not performed, i.e., the screenshot fails. If the current frame's desktop image is not exactly the same as the screenshot image of the previous frame, the screenshot operation is performed, i.e., the screenshot is successful. Specifically, identifying whether the current frame's desktop image is exactly the same as the screenshot image of the previous frame can be achieved by obtaining the image data of the current frame's desktop image and the screenshot image of the previous frame through image recognition, comparing whether the image data is exactly the same, and determining whether to perform a screenshot of the previous frame's desktop. The current frame's desktop screenshot operation can be performed on the current frame's desktop image after initializing the desktop screenshot and obtaining the first successfully captured screenshot image.

[0075] S104: If the current frame fails to capture the screen, the screenshot data is read from the CPU cache and used by the GPU to encode the current frame.

[0076] In this scheme, if the current frame screenshot fails (i.e., the image data of the desktop image in the current frame is exactly the same as the image data of the previous frame), then the screenshot operation for the current frame's desktop image is no longer performed, and the screenshot data of the previous frame can be directly used as the screenshot data for the current frame. Since the screenshot data of each successfully captured desktop image can be stored in the GPU, and can be mapped from the GPU's video memory and copied to the CPU's cache, the screenshot data of the previous frame can be read from the CPU's cache for the GPU to encode the current frame. Specifically, if the current frame's desktop image screenshot is successful, the screenshot data of the current frame is stored in the GPU's video memory address, and a new mapping relationship is established between the GPU and the CPU. This mapping relationship can be used to provide the CPU with permission to access and read the GPU's video memory address for the current screenshot data; if the current frame's desktop image screenshot fails, no new mapping relationship is established, and the CPU directly reads the screenshot data of the previous frame using the mapping relationship of the previous frame. After the CPU reads the screenshot data of the previous frame from the GPU's video memory, it copies the screenshot data to the cache and provides encoding to read the screenshot data in the cache for the GPU to encode the current frame.

[0077] Based on the above embodiments, optionally, after identifying whether the current frame desktop screenshot was successfully captured, the method further includes:

[0078] If the current frame capture is successful, the mapping of the previous frame's screenshot image from the GPU's video memory to the CPU is closed, and the mapping of the current frame's screenshot image from the GPU's video memory to the CPU is established. The current frame's screenshot image is then copied to the CPU's cache.

[0079] Screenshot data is read from the CPU cache and used by the GPU for current frame encoding.

[0080] In this scheme, if the current frame screenshot is successful (i.e., the image data of the current frame's desktop image is not completely identical to the screenshot data of the previous frame's desktop screenshot), then a screenshot operation is performed on the current frame's desktop image, and the screenshot data is acquired. The mapping between the previous frame's screenshot image and the GPU's video memory (VRAM) is closed, and a new mapping relationship is established between the current frame's screenshot image and the GPU's VRAM and the CPU. This mapping relationship can be used to provide the CPU with permission to access and read the GPU's VRAM address for the current screenshot data. After the CPU reads the current screenshot data, it copies the screenshot data to the CPU's cache, and then encodes the screenshot data in the CPU cache for the GPU to encode the current frame.

[0081] In this solution, if the current frame capture is successful, the mapping of the previous frame's screenshot image from the GPU's video memory to the CPU is closed, and the mapping of the current frame's screenshot image from the GPU's video memory to the CPU is established. The current frame's screenshot image is then copied to the CPU's cache. Screenshot data is read from the CPU's cache for the GPU to encode the current frame. This improves screenshot efficiency and the reliability of the desktop screenshot data mapping.

[0082] Based on the above embodiments, optionally, after successfully identifying the current frame as a screenshot, the method further includes:

[0083] Identify whether local rendering of the current frame screenshot is required;

[0084] If local rendering is required, the current frame screenshot image will be rendered locally on the GPU using a preset method.

[0085] In this solution, after successfully capturing the current frame, it determines whether local rendering of the captured image is required. Specifically, this can be determined by checking if a secondary screen is installed on the teacher's device in the interactive classroom equipment system. If a secondary screen is installed, local rendering of the current frame is required; otherwise, it is not. Local rendering can be a process of generating an image from a model, specifically, a process of displaying the current frame's screenshot on the teacher's secondary screen. If local rendering is required, a preset method is used to locally render the current frame's screenshot on the GPU. This preset method can be D3D11 texture rendering.

[0086] In this solution, it is determined whether local rendering of the current frame screenshot image is required. If local rendering is required, the current frame screenshot image is rendered locally on the GPU using a preset method. Local rendering can be performed according to needs, avoiding the problem of delay and lag in receiving screenshot images on the student's end due to excessive processing time of screenshot data, thus improving the efficiency of desktop screenshot data processing.

[0087] Based on the above embodiments, optionally, before establishing the mapping of the current frame screenshot image from the GPU's video memory to the CPU, the method further includes:

[0088] Identify whether scaling of the current frame screenshot is required;

[0089] If scaling is required, the current frame screenshot image is scaled to the target size, and then the operation of establishing the mapping of the current frame screenshot image from the GPU's video memory to the CPU is performed.

[0090] In this solution, determining whether scaling of the current frame screenshot image is necessary can involve identifying whether the screenshot data resolution of the current frame screenshot image meets a preset resolution condition. This preset resolution condition can be that the screenshot data resolution is no greater than 1920*1080. If the condition is met, the mapping relationship between the current frame screenshot image and the GPU's video memory can be directly established. If the condition is not met, the screenshot data resolution of the current frame screenshot image is scaled, and then the operation of establishing the mapping relationship between the current frame screenshot image and the GPU's video memory is performed. Specifically, the scaling process can be reducing the screenshot data resolution of the current frame screenshot image to a target size, where the target size can be a resolution of 1920*1080.

[0091] In this solution, it is determined whether the current frame screenshot image needs to be scaled. If scaling is required, the current frame screenshot image is scaled to the target size, and then the operation of establishing the mapping of the current frame screenshot image from the GPU's video memory to the CPU is performed. This can avoid the problem of unstable frame rate caused by the low resolution and poor performance of the interactive classroom system, reduce the encoding consumption of the overall desktop screenshot to rendering, and improve the processing efficiency of desktop screenshot data.

[0092] Based on the above embodiments, optionally, the scaling process is performed in the GPU;

[0093] After scaling, the current frame screenshot is converted into the target data format.

[0094] In this solution, the scaling process is performed on the GPU. After scaling, the current frame screenshot is converted into the target data format. Here, the target data format can be a data format that occupies less data space than ordinary image data formats (such as RGB format) when storing the same image. For example, the target data format can be NV12 data format. NV12 data format is a YUV data storage method. YUV divides the image representation into two parts: luminance (brightness) and chroma. Chroma is further divided into two sub-dimensions: hue and saturation. Generally, Y represents luminance, U represents hue, and V represents saturation, each stored in one byte. Therefore, YUV, like RGB, uses three bytes to represent color, converted through a fixed linear mapping. To reduce image size, YUV maintains the number of Y values ​​while reducing the number of U and V values. For example, several pixels may share one U or V value. Different sharing methods result in different encoding methods, such as YUV 4:4:4; YUV 4:2:2; YUV 4:2:0, etc. The YUV data storage methods are broadly divided into two categories: one is to store the three YUV components alternately, and the other is to store YUV components per image. The former can be simply understood as storing information by pixel; the latter can be understood as storing by component, for example, storing the Y values ​​of all pixels together. The NV12 data format can be two planes, storing the Y and UV components respectively. The UV components share a plane and are arranged alternately in the order U, V, U, V, with every four Y components sharing a set of U and V components.

[0095] In this solution, the scaling process is performed in the GPU, which can improve the efficiency of scaling data calculation and thus improve the efficiency of DXGI screenshot data processing. After scaling, the current frame screenshot image is converted into the target data format, which can reduce the amount of DXGI screenshot data and improve the processing efficiency of screenshot data in the GPU.

[0096] The technical solution provided in this embodiment initializes the desktop screenshot to obtain the first frame screenshot image of a successful screenshot; maps the first frame screenshot image from the GPU's video memory to the CPU and copies it to the CPU's cache; reads the screenshot data from the CPU's cache for the GPU to perform initial frame encoding; identifies whether the current frame desktop screenshot was successful; if the current frame screenshot failed, reads the screenshot data from the CPU's cache for the GPU to perform current frame encoding. Through the above-described desktop screenshot data processing method for interactive classrooms, the problems of unstable screenshot frame rate, numerous screenshot data mappings, and low screenshot data processing efficiency can be solved. By mapping the first frame screenshot image from the GPU's video memory and caching it to the CPU for the GPU to perform initial frame encoding, identifying whether the current frame desktop screenshot was successful, and reading the screenshot data from the CPU's cache for the GPU to perform current frame encoding if the current frame screenshot failed, the problem of long data processing time and high memory consumption caused by multiple screenshot data mappings can be avoided, thus improving the stability of the screenshot frame rate and the efficiency of screenshot data processing.

[0097] Example 2

[0098] Figure 2 This is a flowchart illustrating the desktop screenshot data processing method for an interactive classroom provided in Embodiment 2 of this application. Figure 2 As shown, the specific steps include the following:

[0099] S201, Initialize desktop screenshot to obtain the first frame of the screenshot image after successful screenshot;

[0100] S202, the first frame screenshot image is mapped from the GPU's video memory to the CPU and copied to the CPU's cache. The screenshot data is read from the CPU's cache for the GPU to perform initial frame encoding.

[0101] S203, identify whether the current frame desktop screenshot was successfully captured;

[0102] Based on the above embodiments, optionally, identifying whether the current frame desktop screenshot was successfully captured includes:

[0103] If the device driver detects changes in input or output data, the screenshot is considered successful.

[0104] If no changes in input or output data are detected by the device driver, the screenshot is considered to have failed.

[0105] In this solution, the device driver can be software code running on the interactive classroom live streaming device. Its main function is to complete the data transmission between the computer system and the hardware device. For example, the device driver can complete the data transmission between the computer system data on the interactive classroom live streaming device and hardware such as the mouse, keyboard, and monitor. The device driver acts as a medium between the operating system and the hardware on the interactive classroom live streaming device, enabling bidirectional communication of information between the operating system and the hardware. That is, it conveys the functions of the hardware device itself to the operating system, and at the same time, it conveys the standard instructions of the operating system to the hardware device, thereby achieving a seamless connection between the two. Since the device driver contains information about the hardware device, it can tell the computer system the functions of the hardware, and it can also convey the instructions received by the system device to the hardware. Therefore, the device driver on the interactive classroom live streaming device can be used to detect and identify whether the current frame of the desktop screenshot was successfully captured. Specifically, the system data of the operating system and the input or device data of the hardware device on the interactive classroom live streaming device can be compared with the data stored in the CPU through the device driver. If the data is inconsistent with the data stored in the CPU, it means that there is a change in the input or output data, that is, the screenshot of the current frame of the desktop is successful. If they are consistent, it means that there is no change in the input or output data, that is, the screenshot of the current frame of the desktop fails.

[0106] In this solution, the success of a screenshot is determined by whether the input or output data of the device driver detection system has changed. If the device driver detects a change in the data, the screenshot is considered successful; otherwise, the screenshot is considered unsuccessful. This avoids excessive mapping processing of the screenshot data and long processing time, thus improving the processing efficiency of desktop screenshot data.

[0107] S204 If the current frame fails to capture the screen, the capture data is read from the CPU cache for the GPU to encode the current frame.

[0108] Based on the above embodiments, optionally, after reading screenshot data from the CPU cache for the GPU to perform current frame encoding, the method further includes:

[0109] Identify whether to stop screenshotting;

[0110] If so, release the GPU memory and CPU cache occupied by the screenshot data, and end the desktop screenshot data processing flow.

[0111] In this solution, identifying whether to stop screenshotting can be done by determining whether to stop screen recording on the current teacher's device. This identification can be achieved through image recognition to determine if the teacher's device has closed course files, whiteboard notes, or auxiliary screens, or by identifying whether the teacher's device issues a stop or pause recording command. No further limitations are specified here. If the identification result indicates that screenshotting has stopped, the screenshot data stored in the GPU's video memory and the CPU's cache is cleared through encoding, releasing the GPU's video memory and the CPU's cache, and ending the desktop screenshot data processing flow.

[0112] In this solution, after reading the screenshot data from the CPU cache for the GPU to encode the current frame, it is determined whether to stop the screenshot. If so, the GPU's video memory and the CPU's cache occupied by the screenshot data are released, and the desktop screenshot data processing flow is terminated, which can release system device memory and system resources in a timely manner.

[0113] The technical solution provided in this application identifies whether the image content of the current frame DXGI screenshot is the same as the image content of the previous frame DXGI screenshot; if they are the same, the screenshot fails; if they are different, the screenshot succeeds; it also identifies whether to stop the screenshot; if so, it releases the GPU's video memory and the CPU's cache occupied by the screenshot data and ends the desktop screenshot data processing flow. This can avoid excessive mapping processing of screenshot data and long processing time, and promptly release system device memory and system resources, thereby improving the processing efficiency of desktop screenshot data.

[0114] Example 3

[0115] Figure 3 This is a schematic diagram of the desktop screenshot data processing device for an interactive classroom provided in Embodiment 3 of this application. Figure 3 As shown, it specifically includes the following:

[0116] The screenshot initialization module 301 is used to initialize the desktop screenshot in order to obtain the first frame of the screenshot image after a successful screenshot.

[0117] The screenshot image mapping module 302 is used to map the first frame screenshot image from the GPU's video memory to the CPU, copy it to the CPU's cache, and read the screenshot data from the CPU's cache for the GPU to perform initial frame encoding.

[0118] The screenshot recognition module 303 is used to identify whether the current frame of the desktop screenshot was successfully captured.

[0119] The screenshot data encoding module 304 is used to read screenshot data from the CPU cache and provide it to the GPU for encoding the current frame if the current frame fails to be captured.

[0120] Furthermore, the screenshot data encoding module is also used for:

[0121] If the current frame capture is successful, the mapping of the previous frame's screenshot image from the GPU's video memory to the CPU is closed, and the mapping of the current frame's screenshot image from the GPU's video memory to the CPU is established. The current frame's screenshot image is then copied to the CPU's cache.

[0122] Screenshot data is read from the CPU cache and used by the GPU for current frame encoding.

[0123] Furthermore, the screenshot data encoding module also includes:

[0124] The local rendering recognition unit is used to identify whether the current frame screenshot needs to be rendered locally;

[0125] The screenshot image rendering unit is used to perform local rendering of the current frame screenshot image on the GPU using a preset method if local rendering is required.

[0126] Furthermore, the screenshot data encoding module also includes:

[0127] The scaling recognition unit is used to identify whether the current frame screenshot image needs to be scaled.

[0128] The screenshot image scaling unit is used to scale the current frame screenshot image to the target size if scaling processing is required, and then perform the operation of establishing the mapping of the current frame screenshot image from the GPU's video memory to the CPU.

[0129] Furthermore, the scaling process of the screenshot image scaling unit is performed in the GPU;

[0130] The screenshot image scaling unit is also used to convert the current frame screenshot image into a target data format after scaling.

[0131] Furthermore, the screenshot data encoding module is also used for:

[0132] Identify whether to stop screenshotting;

[0133] If so, release the GPU memory and CPU cache occupied by the screenshot data, and end the desktop screenshot data processing flow.

[0134] Furthermore, the screenshot recognition module is specifically used for:

[0135] If the device driver detects changes in input or output data, the screenshot is considered successful.

[0136] If no changes in input or output data are detected by the device driver, the screenshot is considered to have failed.

[0137] The technical solution provided in this embodiment includes a screenshot initialization module for initializing desktop screenshots to obtain the first frame screenshot image of a successful screenshot; a screenshot image mapping module for mapping the first frame screenshot image from the GPU's video memory to the CPU and copying it to the CPU's cache, and reading screenshot data from the CPU's cache for the GPU to perform initial frame encoding; a screenshot recognition module for recognizing whether the current frame desktop screenshot was successfully captured; and a screenshot data encoding module for reading screenshot data from the CPU's cache for the GPU to perform current frame encoding if the current frame screenshot fails. The desktop screenshot data processing device in the interactive classroom described above can solve the problems of unstable screenshot frame rate, excessive number of screenshot data mappings, and low screenshot data processing efficiency. By mapping the first frame screenshot image from the GPU's video memory and caching it in the CPU, the GPU can perform initial frame encoding and identify whether the current frame desktop screenshot was successful. If the current frame screenshot fails, the screenshot data is read from the CPU's cache for the GPU to perform current frame encoding. This avoids the problems of long data processing time and excessive memory consumption caused by multiple mappings of screenshot data, thus improving the stability of the screenshot frame rate and the efficiency of screenshot data processing.

[0138] The desktop screenshot data processing device in the interactive classroom of this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0139] The desktop screenshot data processing device for the interactive classroom in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0140] The desktop screenshot data processing device for interactive classrooms provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0141] Example 4

[0142] like Figure 4 As shown, this application embodiment also provides an electronic device 400, including a processor 401, a memory 402, and a program or instructions stored in the memory 402 and executable on the processor 401. When the program or instructions are executed by the processor 401, they implement the various processes of the above-described interactive classroom desktop screenshot data processing method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0143] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0144] Example 5

[0145] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described interactive classroom desktop screenshot data processing method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0146] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0147] Example 6

[0148] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described interactive classroom desktop screenshot data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0149] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0152] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0153] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A method for processing desktop screenshot data in an interactive classroom, characterized in that, The method includes: Initialize the desktop screenshot function to capture the first frame of the screenshot after a successful capture. The first frame screenshot image is mapped from the GPU's video memory to the CPU and copied to the CPU's cache. The screenshot data is read from the CPU's cache for the GPU to perform initial frame encoding. Determine whether the current frame of the desktop screenshot was successfully captured; If the current frame fails to capture the screen, the screenshot data is read from the CPU cache and used by the GPU to encode the current frame. The step of reading screenshot data from the CPU cache for the GPU to encode the current frame if the current frame fails to capture the screenshot includes: If the image data of the current frame is the same as the image data of the previous frame, it is determined that the current frame screenshot failed. Then, the screenshot data of the previous frame is read from the CPU cache and used as the screenshot data of the current frame for the GPU to encode the current frame.

2. The method according to claim 1, characterized in that, After identifying whether the current frame desktop screenshot was successfully captured, the method further includes: If the current frame capture is successful, the mapping of the previous frame's screenshot image from the GPU's video memory to the CPU is closed, and the mapping of the current frame's screenshot image from the GPU's video memory to the CPU is established. The current frame's screenshot image is then copied to the CPU's cache. Screenshot data is read from the CPU cache and used by the GPU for current frame encoding.

3. The method according to claim 2, characterized in that, After successfully identifying the current frame as a screenshot, the method further includes: Identify whether local rendering of the current frame screenshot is required; If local rendering is required, the current frame screenshot image will be rendered locally on the GPU using a preset method.

4. The method according to claim 2, characterized in that, Before establishing the mapping from the GPU's video memory to the CPU for the current frame screenshot image, the method further includes: Identify whether scaling of the current frame screenshot is required; If scaling is required, the current frame screenshot image is scaled to the target size, and then the operation of establishing the mapping of the current frame screenshot image from the GPU's video memory to the CPU is performed.

5. The method according to claim 4, characterized in that, The scaling process is performed in the GPU; After scaling, the current frame screenshot is converted into the target data format.

6. The method according to claim 1 or 2, characterized in that, After reading screenshot data from the CPU cache for the GPU to encode the current frame, the method further includes: Identify whether to stop screenshotting; If so, release the GPU memory and CPU cache occupied by the screenshot data, and end the desktop screenshot data processing flow.

7. The method according to claim 1, characterized in that, To determine whether the current frame of the desktop screenshot was successfully captured, including: If the device driver detects changes in input or output data, the screenshot is considered successful. If no changes in input or output data are detected by the device driver, the screenshot is considered to have failed.

8. A desktop screenshot data processing device for an interactive classroom, characterized in that, The device includes: The screenshot initialization module is used to initialize the desktop screenshot in order to obtain the first frame of the screenshot image after a successful screenshot. The screenshot image mapping module is used to map the first frame screenshot image from the GPU's video memory to the CPU, copy it to the CPU's cache, and read the screenshot data from the CPU's cache for the GPU to perform initial frame encoding. The screenshot recognition module is used to identify whether the current frame of the desktop screenshot was successfully captured; The screenshot data encoding module is used to read screenshot data from the CPU cache and provide it to the GPU for encoding the current frame if the current frame fails to be captured. The step of reading screenshot data from the CPU cache for the GPU to encode the current frame if the current frame fails to capture the screenshot includes: If the image data of the current frame is the same as the image data of the previous frame, it is determined that the current frame screenshot failed. Then, the screenshot data of the previous frame is read from the CPU cache and used as the screenshot data of the current frame for the GPU to encode the current frame.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the desktop screenshot data processing method for an interactive classroom as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, they implement the steps of the desktop screenshot data processing method for interactive classrooms as described in any one of claims 1-7.

Images