Desktop screen recording method and device and storage medium

Abstract

The invention provides a desktop screen recording method and device and a storage medium, the desktop screen recording method and device are applied to a computer device, the computer device comprises a core processor and a graphics processor, and the method comprises the steps that a current desktop is displayed, and the current desktop comprises a desktop picture and a mouse cursor; obtaining the desktop picture to obtain a desktop image; the position and the image of the mouse cursor in the current desktop are obtained, mouse cursor data are obtained, and the mouse cursor data comprise a mouse cursor image; synthesizing a mouse cursor image in the mouse cursor data into the desktop image through the core processor to obtain a desktop synthesized image; and encoding the desktop composite image through the graphics processor to obtain a current screen recording picture. According to the technical scheme, the graphics processor does not need to have the capability of processing multiple paths of data, the requirement for the processing capability of the graphics processor can be lowered, the method can be suitable for the graphics processor with low processing capability, and the applicability is higher.

Description

Technical Field

[0001] This application relates to the field of screen recording, and more particularly to desktop screen recording methods, devices, and storage media. Background Technology

[0002] Desktop recording is a technique that records activity on a computer screen. It can be used to create instructional videos, presentations, software tutorials, and more.

[0003] In screen recording scenarios, in addition to recording the computer desktop, it is also necessary to record the image and trajectory of the mouse cursor on the desktop, so that the recorded screen contains both the desktop and the mouse cursor. In a computer system, the desktop and mouse cursor data exist on different planes and need to be acquired and processed separately. How to acquire and process the desktop and mouse cursor data, which are on different planes, to form a complete screen recording has become a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a desktop recording method, device, and storage medium to acquire and process desktop screen and mouse cursor data on different planes, and obtain a screen recording containing the desktop screen and mouse cursor.

[0005] Firstly, a desktop screen recording method is provided, applied to a computer device, the computer device including a core processor and a graphics processor, the method comprising:

[0006] Display the current desktop, which includes the desktop screen and the mouse cursor;

[0007] The desktop screen is captured to obtain a desktop image;

[0008] The position and image of the mouse cursor in the current desktop are obtained to obtain mouse cursor data, which includes the mouse cursor image.

[0009] The core processor synthesizes the mouse cursor image from the mouse cursor data into the desktop image to obtain a composite desktop image.

[0010] The graphics processor encodes the composite desktop image to obtain the current screen recording.

[0011] In this technical solution, the current desktop, including the desktop image and mouse cursor, is displayed. The desktop image is acquired to obtain the desktop picture, and the position and image of the mouse cursor on the current desktop are acquired to obtain mouse cursor data, including the mouse cursor image. The core processor synthesizes the mouse cursor image from the mouse cursor data into the desktop image to obtain a composite desktop image. Then, the graphics processor encodes the composite desktop image to obtain the current screen recording. By using the core processor to synthesize the mouse cursor image into the desktop image and the graphics processor to encode the synthesized desktop image, the acquisition and processing of desktop images and mouse cursor data on different planes are realized, resulting in a screen recording containing both the desktop image and the mouse cursor. Since the processing power of the core processor is usually higher than that of the graphics processor, the processing power of the core processor is used to synthesize the mouse cursor image into the desktop image to obtain the composite desktop image. The graphics processor only needs to encode the composite desktop image, that is, the graphics processor only needs to process one channel of data, and does not need to have the ability to process multiple channels of data. This reduces the processing power requirements of the graphics processor and is applicable to graphics processors with lower processing power, making it more versatile.

[0012] In conjunction with the first aspect, in one possible implementation, the desktop image is stored in the storage area corresponding to the graphics card driver of the computer device; before the core processor synthesizes the mouse cursor image in the mouse cursor data into the desktop image to obtain the composite desktop image, the method further includes: mapping the desktop image to a first memory address, wherein the first memory address is a memory address that the core processor can access.

[0013] Before the core processor composites the mouse cursor image into the desktop image to obtain the composite desktop image, the desktop image is mapped to a memory address accessible to the core processor. This facilitates the core processor's access to and processing of the desktop image. Furthermore, because the core processor accesses and processes the desktop image based on memory mapping, the desktop image is always stored in the storage area corresponding to the graphics card driver. This eliminates the need to copy the desktop image to the core processor's memory, reducing the processing overhead caused by copying, lowering processing latency, and thus improving processing speed.

[0014] In conjunction with the first aspect, in one possible implementation, after acquiring the desktop screen and obtaining the desktop image, the method further includes: mapping the desktop image to a second memory address, wherein the second memory address is a memory address accessible by the graphics processor.

[0015] After acquiring the desktop image, mapping it to a memory address accessible to the graphics processor (GPU) facilitates the GPU's access to and processing of the composite desktop image after the core processor synthesizes it. Because the GPU accesses and processes the composite desktop image based on memory mapping, the desktop image is always stored in the storage area corresponding to the graphics card driver. This eliminates the need for copying and duplication of the image between the storage area corresponding to the graphics card driver, the core processor's memory, and the GPU's memory, thereby improving processing speed and increasing the recording frame rate.

[0016] In conjunction with the first aspect, in one possible implementation, the desktop image is stored in the storage area corresponding to the graphics card driver of the computer device; before the core processor synthesizes the mouse cursor image from the mouse cursor data into the desktop image to obtain the composite desktop image, the method further includes: moving and storing the desktop image to the memory corresponding to the core processor; before the graphics processor encodes the composite desktop image to obtain the current screen recording, the method further includes: moving and storing the composite desktop image to the memory corresponding to the graphics processor.

[0017] Before the core processor composites the mouse cursor image into the desktop image to obtain the composite desktop image, the desktop image is moved and stored in the memory corresponding to the core processor, which makes it easier for the core processor to process the desktop image; before the graphics processor encodes the composite desktop image, the composite desktop image is moved and stored in the memory corresponding to the graphics processor, which makes it easier for the graphics processor to process the composite desktop image.

[0018] In conjunction with the first aspect, in one possible implementation, before the core processor synthesizes the mouse cursor image from the mouse cursor data into the desktop image to obtain the composite desktop image, the method further includes: converting the image format of the desktop image into a first image format, wherein the image data volume corresponding to the first image format is less than the image data volume corresponding to the image format of the desktop image before conversion; and converting the image format of the mouse cursor image into a second image format that matches the first image format, wherein the image data volume corresponding to the second image format is less than the image data volume corresponding to the image format of the mouse cursor image before conversion.

[0019] Before compositing the mouse cursor image into the desktop image through the core processor to obtain the composite desktop image, converting the image formats of the desktop image and the mouse cursor image into image formats with smaller image data sizes can reduce the amount of computation generated by image compositing, reduce the computational overhead of the core processor, improve processing speed, and thus increase the recording frame rate.

[0020] In conjunction with the first aspect, in one possible implementation, before encoding the desktop composite image by the graphics processor to obtain the current screen recording, the method further includes: converting the image format of the desktop composite image into a first image format, wherein the image data volume corresponding to the first image format is less than the image data volume corresponding to the image format of the desktop composite image before conversion.

[0021] Before encoding the desktop composite image using a graphics processor to obtain the current screen recording, converting the image format of the desktop composite image to a smaller image format can reduce computational overhead and lower the encoding computation load of the graphics processor, thus making it suitable for graphics processors with poor processing performance.

[0022] In conjunction with the first aspect, in one possible implementation, the size of the mouse cursor image is smaller than the size of the desktop image, and the mouse cursor data further includes target position coordinates, which reflect the position of the mouse cursor image on the current desktop; the step of compositing the mouse cursor image in the mouse cursor data into the desktop image through the core processor to obtain a composite desktop image includes: drawing the mouse cursor image at the image position corresponding to the target position coordinates in the desktop image through the core processor to obtain the composite desktop image.

[0023] When the size of the mouse cursor image is smaller than the size of the desktop image, by drawing the mouse cursor image on the desktop image, it is possible to composite the mouse cursor image and the desktop image, so that the composite desktop image is consistent with the desktop screen seen by the naked eye.

[0024] In conjunction with the first aspect, in one possible implementation, the size of the mouse cursor image is the same as the size of the desktop image; the step of compositing the mouse cursor image from the mouse cursor data into the desktop image through the core processor to obtain a composite desktop image includes: performing a superposition operation on the image values ​​of the same image positions in the mouse cursor image and the desktop image through the core processor to obtain the composite desktop image.

[0025] When the size of the mouse cursor image is the same as the size of the desktop image, by superimposing the image values ​​of the same image position in the mouse cursor image and the desktop image, it is possible to synthesize the mouse cursor image and the desktop image, so that the synthesized desktop image is consistent with the desktop screen seen by the naked eye.

[0026] In a second aspect, a desktop screen recording device is provided for use in a computer device, the computer device including a core processor and a graphics processor, the device comprising:

[0027] The display module is used to display the current desktop, which includes the desktop screen and the mouse cursor.

[0028] The desktop image acquisition module is used to acquire the desktop screen and obtain a desktop image;

[0029] A mouse cursor data acquisition module is used to acquire the position and image of the mouse cursor on the current desktop, and obtain mouse cursor data, wherein the mouse cursor data includes a mouse cursor image;

[0030] The compositing module is used to composite the mouse cursor image from the mouse cursor data into the desktop image through the core processor to obtain a composite desktop image;

[0031] The encoding module is used to encode the desktop composite image through the graphics processor to obtain the current screen recording screen.

[0032] Thirdly, a computer device is provided, including a memory and a processor, the memory being connected to the processor, the processor including a core processor and a graphics processor, the core processor and the graphics processor being configured to cooperate in executing one or more computer programs stored in the memory, the core processor and the graphics processor, when executing one or more computer programs, enabling the computer device to implement the desktop recording method of the first aspect described above.

[0033] Fourthly, a computer-readable storage medium is provided, which stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the desktop recording method of the first aspect.

[0034] This application achieves the following technical effects: by using a core processor to composite the mouse cursor image into a desktop image, and then using a graphics processor to encode the composite desktop image, it realizes the acquisition and processing of desktop screen and mouse cursor data on different planes, resulting in a screen recording containing both the desktop screen and the mouse cursor. Since the processing power of a core processor is usually higher than that of a graphics processor, by using the processing power of the core processor to composite the mouse cursor image into the desktop image to obtain a composite desktop image, the graphics processor only needs to encode the composite desktop image. That is, the graphics processor only needs to process one channel of data, and does not need to have the ability to process multiple channels of data. This reduces the processing power requirements of the graphics processor, making it suitable for graphics processors with lower processing power and thus more versatile. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating a process for recording the desktop of a computer device.

[0037] Figure 2 A flowchart illustrating a desktop screen recording method provided in an embodiment of this application;

[0038] Figure 3 A schematic diagram of the current desktop, desktop image, and mouse cursor image provided in the embodiments of this application;

[0039] Figure 4 A flowchart illustrating another desktop screen recording method provided in an embodiment of this application;

[0040] Figure 5 A flowchart illustrating yet another desktop screen recording method provided in this application embodiment;

[0041] Figure 6 This is a schematic diagram of the structure of a desktop screen recording device provided in an embodiment of this application;

[0042] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0044] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0045] The technical solution of this application is applicable to desktop screen recording scenarios for computer devices.

[0046] In computer equipment, the core processor and the graphics processing unit are two important components.

[0047] A core processor, also known as a general-purpose processor or central processing unit (CPU), is essentially the brain of a computer. It performs various general-purpose computing tasks, such as running the operating system, processing text, and executing applications. A core processor consists of an arithmetic logic unit (ALU), a control unit, and registers. Core processors typically have a limited number of cores, each with powerful processing capabilities and complex control logic. This allows for higher clock frequencies and lower latency with a limited number of cores, enabling the processing of complex logic and control tasks.

[0048] A graphics processing unit (GPU) is a component of a computer's graphics card. A graphics card is dedicated hardware for image processing and rendering, consisting of a GPU, corresponding memory, and a display interface. The GPU is essentially the brain of the graphics card, possessing powerful computing capabilities in image processing and parallel computing. It can quickly process graphics data and is primarily used for graphics-related applications. GPUs typically have a large number of simple cores, each with a low clock speed, suitable for handling highly parallel graphics computing tasks.

[0049] In computer devices, the core processor and graphics processing unit (GPU) work together to provide computing power and perform various image processing and display tasks. For example, in a game, the core processor handles game logic and control, while the GPU renders the graphics. Communication and collaboration between the core processor and GPU are coordinated by the graphics card driver. The GPU driver translates the graphics instructions sent by the core processor into an instruction format that the GPU can understand. The core processor can then allocate graphics tasks to the GPU for processing through the GPU driver. When the computer starts up, the operating system running on the core processor loads the GPU driver and communicates with the GPU. Based on information provided by the operating system, the GPU renders the desktop environment to be displayed, including the desktop background, icons, windows, and other elements. The operating system also communicates with the computer's mouse control hardware (such as mouse peripherals and touchscreens) to obtain sensor signals. Based on these signals, it determines the mouse's position and state and sends this information to the GPU. The GPU then renders the image of the mouse cursor and sends the processed desktop environment and mouse cursor to the monitor through the display interface, thus completing the desktop display. Typically, the desktop environment and mouse cursor are on different planes. When recording the desktop of a computer device, it is necessary to record both the desktop screen and the mouse cursor. Since the desktop screen and the mouse cursor are on different planes, it is necessary to acquire the desktop screen and mouse cursor data separately and combine the desktop screen and mouse cursor data together to form the screen recording.

[0050] In some technical solutions, it is through Figure 1 The illustrated process involves recording the computer screen by capturing the desktop image and the mouse cursor image. These two images are then shared as two separate data streams with the GPU. The GPU then composites and encodes these images to create a screen recording that includes both the desktop and mouse cursor data. This technique requires a GPU with multi-channel data compositing capabilities and is suitable for GPUs with high processing power. It is not suitable for GPUs with lower processing power, such as those lacking compositing capabilities, thus limiting the applicability of this technique.

[0051] In view of this, this application proposes a novel desktop screen recording scheme. After acquiring the desktop screen and obtaining the desktop image and the mouse cursor image, the CPU of the computer device is used to synthesize the desktop image and the mouse cursor image to obtain a composite desktop image. Then, the graphics processing unit (GPU) of the computer device is used to encode the composite desktop image synthesized by the CPU, resulting in a screen recording containing both the desktop image and the mouse cursor. Essentially, the processing power of the CPU is used to replace the GPU in synthesizing the desktop image and mouse cursor image, resulting in only one data stream of the composite desktop image. The GPU only needs to encode the composite desktop image synthesized by the CPU, meaning it only needs to process one data stream, eliminating the need for multi-stream processing capabilities. This reduces the processing power requirements for the GPU and makes it suitable for GPUs with lower processing power, thus increasing its applicability. Furthermore, since the GPU is used to encode the composite desktop image, compared to encoding it through the CPU, the GPU's image processing capabilities are fully utilized, while avoiding excessive consumption of the CPU's processing power, which helps improve processing speed and thus increases the recording frame rate.

[0052] The technical solution of this application is described in detail below. The technical solution of this application can be applied to computer equipment, including but not limited to desktop computers, laptops, etc.

[0053] See Figure 2 , Figure 2 This application provides a flowchart illustrating a desktop screen recording method. The method is applied to a computer device, which includes a core processor and a graphics processor, such as... Figure 2 As shown, the method includes the following steps:

[0054] S101 displays the current desktop, which includes the desktop screen and the mouse cursor.

[0055] Here, the current desktop refers to the system desktop that the computer device is currently displaying. For example, the current desktop displayed on the computer device can be as follows: Figure 3 As shown in P1.

[0056] S102, Obtain the desktop screen and get the desktop image.

[0057] For example, the current desktop displayed on a computer device, such as Figure 3 As shown in P1, the obtained desktop image can be as follows: Figure 3 As shown in P2.

[0058] Since the desktop screen is rendered by the graphics processing unit (GPU) in a computer device, the desktop image can be obtained by capturing the desktop display on the computer device through the GPU driver. This desktop image is then stored in the storage area corresponding to the GPU driver. The storage area corresponding to the GPU driver is used to store various data generated by the GPU driver during operation. The desktop image format can be a bitmap format such as RGB or RGBA.

[0059] In one specific implementation, the desktop image can be obtained by using the graphics card driver and the ffmpeg tool running on the computer device to obtain the current desktop screen displayed on the computer device through the graphics processor in the computer device.

[0060] Optionally, after acquiring the desktop image, it can be moved and stored in the memory corresponding to the computer device's core processor. Moving and storing the desktop image in the memory corresponding to the computer device's core processor means moving the desktop image from the storage area corresponding to the graphics card driver to the memory area corresponding to the computer device's core processor for storage. The memory corresponding to the computer device's core processor refers to the memory area that the core processor can directly access. Before the core processor composites the mouse cursor image into the desktop image to obtain the composite desktop image, moving and storing the desktop image in the memory corresponding to the core processor facilitates the core processor's processing of the desktop image.

[0061] S103, obtain the position and image of the mouse cursor on the current desktop, and get the mouse cursor data.

[0062] Here, mouse cursor data is divided into mouse cursor position data and mouse cursor image data. Mouse cursor position data reflects the position of the mouse cursor on the current desktop, while mouse cursor image data reflects the image style of the mouse cursor on the current desktop.

[0063] The mouse cursor data includes a mouse cursor image, which can be in RGBA format. The background of the mouse cursor image is transparent; that is, all pixels in the mouse cursor image except the cursor itself are transparent.

[0064] The size of the mouse cursor image can be the same as the size of the desktop image. When the mouse cursor image and desktop image are the same size, the number of pixels in the mouse cursor image is the same as the number of pixels in the desktop image, and the mouse cursor image contains mouse cursor position data. For example, the current desktop and mouse cursor displayed on a computer device are shown below. Figure 3 As shown in P1, the desktop image obtained through step S102 is as follows: Figure 3As shown in P2, the mouse cursor image can be as follows: Figure 3 As shown in P3, P3 is the same size as P2. The position of the mouse cursor in P3 reflects the position of the mouse cursor on the current desktop.

[0065] The size of the mouse cursor image can also differ from the size of the desktop image; the mouse cursor image is smaller than the desktop image. When the mouse cursor image size differs from the desktop image size, the number of pixels in the mouse cursor image is less than the number of pixels in the desktop image. The mouse cursor data also includes target position coordinates, which reflect the position of the mouse cursor image on the current desktop. There can be one or more target position coordinates, which can include the position coordinates of one or more image feature points of the mouse cursor image on the current desktop. For example, the target position coordinates can include the position coordinates of the center point of the mouse cursor image on the current desktop; or, the target position coordinates can include the position coordinates of the four vertices of the mouse cursor image (top, bottom, left, right) on the current desktop, etc., and this application does not impose limitations. For example, the current desktop and mouse cursor displayed on a computer device are as follows: Figure 3 As shown in P1, the desktop image obtained through step S101 is as follows: Figure 3 As shown in P2, the mouse cursor image can also be as follows: Figure 2 As shown in P4, the size of P4 is smaller than that of P2. The mouse cursor data also includes the target position coordinates (x1, y1). The target position coordinates (x1, y1) are the position coordinates of the center point of P4 in P1, which reflects the position of P4 in P1.

[0066] Specifically, the position and image of the mouse cursor on the current desktop can be obtained through the core processor in the computer device, thus obtaining mouse cursor data.

[0067] In one specific implementation, the position and image of the mouse cursor on the current desktop can be obtained through a desktop display service running on the computing device, thus obtaining mouse cursor data. For example, if the computer device runs an X11 display service, the position and image of the mouse cursor on the current desktop can be obtained through the Xprotocol C-language binding (XCB) protocol, thus obtaining mouse cursor data.

[0068] S104 uses the core processor of the computer device to composite the mouse cursor image from the mouse cursor data into the desktop image, thus obtaining a composite desktop image.

[0069] Here, the computer device's core processor composites the mouse cursor image into the desktop image to obtain a composite desktop image. This means that the computer device's core processor combines the mouse cursor image and the desktop image into a single image, so that the composite desktop image contains both the current desktop screen and the current desktop mouse cursor.

[0070] If the size of the mouse cursor image is smaller than the size of the desktop image, the mouse cursor data also includes the target position coordinates. The computer device's core processor can draw the mouse cursor image at the image position corresponding to the target position in the desktop image to obtain a composite desktop image.

[0071] For example, the mouse cursor image is as follows Figure 3 As shown in P4, the desktop image is as follows Figure 3 As shown in P2, the target location coordinates are Figure 3 The position coordinates in the image are (x1, y1), which can be determined by the core processor of the computer device. Figure 3 At the position of the mouse cursor in the image with coordinates (x1, y1) in P2, the composite image of the desktop is obtained.

[0072] When the size of the mouse cursor image is smaller than the size of the desktop image, drawing the mouse cursor image within the desktop image allows the resulting composite desktop image to match the desktop screen as seen by the naked eye. Because the mouse cursor image is smaller, the computational load required for core processor synthesis is reduced, thus decreasing the processing overhead of the core processor.

[0073] If the size of the mouse cursor image is the same as the size of the desktop image, the computer's core processor can perform superposition operations on the image values ​​of the same image positions in the mouse cursor image and the desktop image to obtain a composite desktop image.

[0074] In one specific implementation, the image values ​​of the same image position in the mouse cursor image and the desktop image can be superimposed using the following calculation formula to obtain a composite desktop image: z3=z1+z2*(1-A), where z3 represents the image value obtained by superposition, z1 represents the image value at the target image position of the mouse cursor image, z2 represents the image value at the target image position in the desktop image, the target image position is any image position in the image, and A represents the transparency of the mouse cursor image.

[0075] For example, both the desktop image and the mouse cursor image contain m pixels. The desktop image is in RGB format, and the mouse cursor image is in RGBA format. The m pixels in the desktop image are represented by the RGB data set J1 = {(R1i, G1i, B1i), i = 1, 2, ..., m}, and the m pixels in the mouse cursor image are represented by the RGBA data set J2 = {(R2i, G1i, B1i, Ai), i = 1, 2, ..., m}. For the i-th pixel, R1i, R2i, and Ai can be substituted into the above calculation formula to calculate the R value of the i-th pixel. G1i, G2i, and Ai can be substituted into the above calculation formula to calculate the G value of the i-th pixel. B1i, B2i, and Ai can be substituted into the above calculation formula to calculate the B value of the i-th pixel. For m pixels in the desktop image and mouse cursor image, the calculation method of the i-th pixel is used to superimpose the values ​​to obtain the R, G, and B values ​​of each pixel in the composite desktop image, thus obtaining the RGB data corresponding to each pixel in the composite desktop image.

[0076] When the size of the mouse cursor image is the same as the size of the desktop image, the mouse cursor image and the desktop image can be composited by superimposing the image values ​​of the same image position in the mouse cursor image and the desktop image.

[0077] It should be understood that the above calculation formula is only an example of one implementation method for superimposing the image values ​​of the same image position in the mouse cursor image and the desktop image to obtain a composite desktop image, and does not limit this application. There may be other superimposition operation methods, which are not limited in this application. The method of compositing the mouse cursor image into the desktop image to obtain a composite desktop image is not limited to the above implementation method.

[0078] After acquiring the desktop image, it is moved and stored in the memory corresponding to the core processor of the computer device. After compositing the desktop composite image, it can be moved and stored in the memory corresponding to the graphics processor of the computer device. Moving and storing the desktop composite image in the memory corresponding to the graphics processor means moving the desktop composite image from the memory corresponding to the core processor of the computer device to the memory corresponding to the graphics processor for storage. Moving and storing the desktop composite image in the memory corresponding to the graphics processor before encoding the desktop composite image by the graphics processor makes it easier for the graphics processor to process the desktop composite image.

[0079] S105 uses the computer device's graphics processor to encode the desktop composite image to obtain the current screen recording screen.

[0080] The computer's graphics processor can encode the composite image on the desktop using any video encoding format to obtain the current screen recording. For example, the computer's graphics processor can use the MPEG4 format to encode the composite image on the desktop to obtain the current screen recording.

[0081] In the above Figure 2 In the corresponding technical solution, the current desktop is displayed, which includes the desktop image and the mouse cursor. The desktop image is acquired to obtain the desktop image, and the position and image of the mouse cursor on the current desktop are acquired to obtain mouse cursor data, which includes the mouse cursor image. The core processor synthesizes the mouse cursor image from the mouse cursor data into the desktop image to obtain a composite desktop image. Then, the graphics processor encodes the composite desktop image to obtain the current screen recording screen. By synthesizing the mouse cursor image into the desktop image through the core processor and encoding the composite desktop image through the graphics processor, the acquisition and processing of desktop images and mouse cursor data on different planes are realized, resulting in a screen recording screen containing both the desktop image and the mouse cursor. Since the processing power of the core processor is usually higher than that of the graphics processor, the processing power of the core processor is used to synthesize the mouse cursor image into the desktop image to obtain the composite desktop image. The graphics processor only needs to encode the composite desktop image, that is, the graphics processor only needs to process one channel of data, and does not need to have the ability to process multiple channels of data. This reduces the processing power requirements of the graphics processor and is applicable to graphics processors with lower processing power, making it more versatile.

[0082] See Figure 4 , Figure 4 This is a flowchart illustrating another desktop screen recording method provided in an embodiment of this application. The method is applied to a computer device, which includes a core processor and a graphics processor, such as... Figure 4 As shown, the method includes the following steps:

[0083] S201 displays the current desktop, which includes the desktop screen and the mouse cursor.

[0084] S202, Obtain the desktop screen and get the desktop image.

[0085] S203, obtain the position and image of the mouse cursor on the current desktop, and get the mouse cursor data.

[0086] S204 uses the core processor of the computer device to composite the mouse cursor image from the mouse cursor data into the desktop image, thus obtaining a composite desktop image.

[0087] The specific implementation methods of steps S201 to S204 are described here, and will not be repeated here.

[0088] S205 converts the image format of the desktop composite image to the first image format.

[0089] Here, the image data size corresponding to the first image format is smaller than the image data size corresponding to the original desktop composite image format. The image data size corresponding to an image format refers to the amount of information contained in an image of a certain image format. The size of the image data size depends on factors such as the color depth and compression method corresponding to the image format.

[0090] Taking the original desktop composite image in RGB format as an example, each pixel in an RGB image includes the values ​​of three color channels: red, green, and blue. Each color is typically represented using 8 bits (1 byte), and one pixel requires 3 bytes of storage. The image data size of an RGB desktop composite image is 3*m, where m is the number of pixels in the desktop composite image. The image format of the desktop composite image can be converted from RGB to NV12 format. NV12 format, also known as YUV420P format, stores luminance information (Y) and chrominance information (UV) separately. The Y channel data is stored on one plane (Y plane), while the UV channel data is interleaved and stored on another plane (UV plane). Each chrominance information corresponds to 4 luminance information values, and one pixel requires 1.5 bytes of storage. The image data size of an NV12 desktop composite image is 1.5*m, meaning the image data size of NV12 format is smaller than that of RGB format.

[0091] S206: The computer device's graphics processor encodes the desktop composite image in the first image format to obtain the current screen recording screen.

[0092] The specific implementation of step S206 can be found in the description of step S105 above, and will not be repeated here.

[0093] In the above Figure 4In the corresponding technical solution, after acquiring the desktop image and mouse cursor data, the core processor of the computer device composites the mouse cursor image from the mouse cursor data into the desktop image to obtain the composite desktop image. Then, the image format of the composite desktop image is converted into a first image format, and the graphics processor of the computer device encodes the composite desktop image to obtain the current screen recording. Since the image data volume corresponding to the first image format is smaller than the data volume of the image format of the composite desktop image before conversion, converting the image format of the composite desktop image to an image format with smaller image data volume can reduce the computational overhead and reduce the encoding operation of the graphics processor, thus making it suitable for graphics processors with poor processing performance.

[0094] See Figure 5 , Figure 5 This is a flowchart illustrating another desktop screen recording method provided in an embodiment of this application. The method is applied to a computer device, which includes a core processor and a graphics processor, such as... Figure 5 As shown, the method includes the following steps:

[0095] S301 displays the current desktop, which includes the desktop screen and the mouse cursor.

[0096] S302, acquire the desktop screen and obtain the desktop image.

[0097] The specific implementation methods of steps S301 to S302 are described here, and will not be repeated here.

[0098] S303 maps the desktop image to a second memory address.

[0099] Here, the second memory address is the memory address that the computer device's graphics processor can access.

[0100] Mapping a desktop image to a second memory address refers to sharing the desktop image with the virtual address space accessible by the computer's graphics processor based on memory mapping technology. This makes the desktop image appear as part of the graphics processor's memory, allowing the graphics processor to process the desktop image while the desktop image itself remains stored in the memory area corresponding to the graphics card driver.

[0101] In one specific implementation scenario, the graphics processor of the computer device uses the video acceleration application programming interface (VAAPI) for hardware decoding. After obtaining the desktop image through step S301, the desktop image can be mapped to a second memory address through the interface provided by VAAPI, so that the desktop image can be accessed by the VAAPI hardware acceleration library.

[0102] After acquiring the desktop image, mapping it to a memory address accessible to the graphics processor (GPU) facilitates the GPU's access to and processing of the composite desktop image after the core processor synthesizes it. Because this memory-mapping method allows the GPU to access and process the composite desktop image, the synthesized desktop image is always stored in the storage area corresponding to the graphics card driver. This eliminates the need for copying and duplication of the image between the storage area corresponding to the graphics card driver, the core processor's memory, and the GPU's memory, thereby improving processing speed and increasing the recording frame rate.

[0103] S304 converts the image format of the desktop image to the first image format.

[0104] Here, the image data size corresponding to the first image format is smaller than the image data size of the original desktop image format.

[0105] Taking the RGB format of the desktop image as an example, the image format of the desktop image can be converted from RGB format to NV12 format. For relevant descriptions of RGB and NV12 formats, please refer to the description of step S205 above, which will not be repeated here.

[0106] In one specific implementation, the image format of the desktop image can be converted into a first image format based on the video processing capabilities of VAAPI mentioned in step S303 above.

[0107] S305, map the desktop image of the first image format to the first memory address.

[0108] Here, the first memory address is the memory address that the core processor of the computer device can access.

[0109] Mapping a desktop image in a first image format to a first memory address means sharing the desktop image in the first image format to a virtual address space accessible to the core processor of a computer device based on memory mapping technology. This makes the desktop image in the first image format regarded as part of the memory of the core processor of the computer device, thereby enabling the core processor of the computer device to process the desktop image in the first image format, while the desktop image in the first image format itself is still stored in the storage area corresponding to the graphics card driver.

[0110] In one specific implementation, the desktop image of the first image format can be mapped to the first memory address through the VAAPI cache mapping interface mentioned in step S303 above, so that the desktop image of the first image format can be accessed and processed by the core processor of the computer device.

[0111] Before the core processor composites the mouse cursor image into the desktop image to obtain the composite desktop image, the desktop image is mapped to a memory address accessible to the core processor. This facilitates the core processor's access to and processing of the desktop image. Furthermore, because the core processor accesses and processes the desktop image based on memory mapping, the desktop image is always stored in the storage area corresponding to the graphics card driver. This eliminates the need to copy the desktop image to the core processor's memory, reducing the processing overhead caused by copying, lowering processing latency, and thus improving processing speed.

[0112] S306 retrieves the position and image of the mouse cursor on the current desktop, thus obtaining the mouse cursor data.

[0113] For details on the specific implementation of step S306, please refer to the description of step S103 above, which will not be repeated here.

[0114] S307, convert the image format of the mouse cursor image in the mouse cursor data to a second image format that matches the first image format.

[0115] Here, the second image format that matches the first image format means that the channel representing the pixel value corresponding to the second image format is the same as the channel representing the pixel value corresponding to the first image format, and the second image format has one more transparency channel than the first image format.

[0116] For example, if the first image format is RGB, and the corresponding channels for representing image values ​​are the R, G, and B channels, then the second image format is RGBA. Similarly, if the first image format is NV12 (i.e., YUV420P), and the corresponding channels for representing image values ​​are the Y, U, and V channels, then the second image format is YUVA420P.

[0117] The image data size corresponding to the second image format is smaller than the image data size corresponding to the original mouse cursor image format.

[0118] For example, if the mouse cursor image is in RGBA format and the first image is in YUV420P format, then the second image is in YUVA420P format.

[0119] By converting the image format of the mouse cursor image to a format that matches the desktop image, it becomes easier to composite the desktop image and the mouse cursor image.

[0120] S308 uses the core processor of the computer device to composite the mouse cursor image in the second image format into the desktop image in the first image format, thus obtaining a composite desktop image.

[0121] S309 uses the computer device's graphics processor to encode the desktop composite image to obtain the current screen recording screen.

[0122] The specific implementation methods of steps S308 to S309 are described here, and will not be repeated here.

[0123] In the above Figure 5 In the corresponding technical solution, after acquiring the desktop image, the desktop image is mapped to a second memory address. Then, the image format of the desktop image is converted to a first image format, and the desktop image in the first image format is mapped to the first memory address. Next, the image format of the mouse cursor image in the acquired mouse cursor data is converted to a second image format that matches the first image format. Then, the mouse cursor image in the second image format is composited into the desktop image in the first image format to obtain a composite desktop image. Finally, the core processor of the computer device encodes the composite desktop image to obtain the current screen recording. This realizes the acquisition and processing of desktop images and mouse cursor data in different planes, resulting in a screen recording containing both the desktop image and the mouse cursor. Before the core processor composites the mouse cursor image into the desktop image to obtain the composite desktop image, converting the image formats of the desktop image and the mouse cursor image into image formats with smaller image data sizes reduces the computational load generated by image compositing, reduces the computational overhead of the core processor, improves processing speed, and thus increases the recording frame rate.

[0124] The method of this application has been described above; the apparatus of this application will be described below.

[0125] See Figure 6 , Figure 6 This is a schematic diagram of a desktop screen recording device provided in an embodiment of this application. The device is applied to a computer device, which includes a core processor and a graphics processor, such as...Figure 6 As shown, the desktop recording device 40 includes:

[0126] Display module 401 is used to display the current desktop, which includes a desktop screen and a mouse cursor;

[0127] Desktop image acquisition module 402 is used to acquire the desktop screen and obtain a desktop image;

[0128] The mouse cursor data acquisition module 403 is used to acquire the position and image of the mouse cursor in the current desktop and obtain mouse cursor data, wherein the mouse cursor data includes the mouse cursor image.

[0129] The compositing module 404 is used to composite the mouse cursor image in the mouse cursor data into the desktop image through the core processor to obtain a composite desktop image;

[0130] The encoding module 405 is used to encode the desktop composite image through the graphics processor to obtain the current screen recording screen.

[0131] In one possible design, the desktop image is stored in the storage area corresponding to the graphics card driver of the computer device; the desktop image acquisition module 402 is further configured to map the desktop image to a first memory address, wherein the first memory address is a memory address that the core processor can access.

[0132] In one possible design, the desktop image acquisition module 402 is further configured to map the desktop image to a second memory address, wherein the second memory address is a memory address accessible by the graphics processor.

[0133] In one possible design, the desktop image is stored in the storage area corresponding to the graphics card driver of the computer device; the desktop image acquisition module 402 is further configured to move and store the desktop image to the memory corresponding to the core processor; the synthesis module 404 is further configured to move and store the synthesized desktop image to the memory corresponding to the graphics processor.

[0134] In one possible design, the desktop image acquisition module 404 is further configured to convert the image format of the desktop image into a first image format, wherein the image data volume corresponding to the first image format is less than the image data volume corresponding to the image format of the desktop image before conversion; the mouse cursor data acquisition module 402 is further configured to convert the image format of the mouse cursor image into a second image format that matches the first image format, wherein the image data volume corresponding to the second image format is less than the image data volume corresponding to the image format of the mouse cursor image before conversion.

[0135] In one possible design, the above-mentioned compositing module 404 is further configured to convert the image format of the desktop composite image into a first image format, wherein the image data volume corresponding to the first image format is less than the image data volume corresponding to the image format of the desktop composite image before conversion.

[0136] In one possible design, the size of the mouse cursor image is smaller than the size of the desktop image, and the mouse cursor data also includes the target position coordinates, which are used to reflect the position of the mouse cursor image on the current desktop; the above-mentioned synthesis module 404 is specifically used to: draw the mouse cursor image at the image position corresponding to the target position coordinates in the desktop image through the core processor, so as to obtain the desktop composite image.

[0137] In one possible design, the size of the mouse cursor image is the same as the size of the desktop image; the aforementioned compositing module 404 is specifically used to: perform superposition operations on the image values ​​of the same image positions in the mouse cursor image and the desktop image using the core processor to obtain the composite desktop image.

[0138] It should be noted that, Figure 6 For any content not mentioned in the corresponding embodiments, please refer to the description of the foregoing method embodiments, which will not be repeated here.

[0139] The aforementioned device displays the current desktop, which includes the desktop image and the mouse cursor. It acquires the desktop image to obtain a desktop image and acquires the position and image of the mouse cursor on the current desktop to obtain mouse cursor data, which includes the mouse cursor image. The core processor synthesizes the mouse cursor image from the mouse cursor data into the desktop image to obtain a composite desktop image. Then, the graphics processor encodes the composite desktop image to obtain the current screen recording. By using the core processor to synthesize the mouse cursor image into the desktop image and the graphics processor to encode the synthesized desktop image, the device acquires and processes desktop images and mouse cursor data located on different planes to obtain a screen recording containing both the desktop image and the mouse cursor. Since the processing power of the core processor is usually higher than that of the graphics processor, the core processor's processing power is used to synthesize the mouse cursor image into the desktop image to obtain the composite desktop image. The graphics processor only needs to encode the composite desktop image, meaning it only needs to process one data stream and does not need to have the ability to process multiple data streams. This reduces the processing power requirements of the graphics processor and makes it suitable for graphics processors with lower processing power, thus increasing its applicability.

[0140] See Figure 7 , Figure 7This is a schematic diagram of the structure of a computer device 50 provided in an embodiment of this application. The computer device 50 includes a processor 501 and a memory 502. The memory 502 is connected to the processor 501, for example, via a bus.

[0141] Processor 501 is configured to support the computer device 50 in performing the corresponding functions in the methods described in the above method embodiments. Processor 501 includes a core processor and a graphics processor. The core processor may be a central processing unit (CPU); optionally, processor 501 may also include a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0142] Memory 502 is used to store program code, etc. Memory 502 may include volatile memory (VM), such as random access memory (RAM); memory 502 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 502 may also include combinations of the above types of memory.

[0143] The memory 502 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the desktop screen recording method in the embodiments of this application. The core processing and graphics processor work together to execute the various functional applications and data processing of the desktop screen recording method by running the non-volatile software programs, instructions, and modules stored in the memory, thereby realizing the functions of the desktop screen recording method provided in the above method embodiments.

[0144] The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the desktop recording device, etc. In some embodiments, the memory may include memory remotely configured relative to the processor, which can be connected to the charging diagnostic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0145] The one or more modules are stored in the memory. When executed by the one or more processors, they perform the desktop screen recording method in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0146] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in the foregoing embodiments.

[0147] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0148] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A desktop screen recording method, characterized in that, Applied to a computer device, the computer device including a core processor and a graphics processor, the method includes: Display the current desktop, which includes the desktop screen and the mouse cursor; The desktop screen is captured to obtain the desktop image; The position and image of the mouse cursor in the current desktop are obtained to obtain mouse cursor data, which includes the mouse cursor image. The core processor synthesizes the mouse cursor image from the mouse cursor data into the desktop image to obtain a composite desktop image. The graphics processor encodes the composite desktop image to obtain the current screen recording.

2. The method according to claim 1, characterized in that, The desktop image is stored in the storage area corresponding to the graphics card driver of the computer device; Before the process of compositing the mouse cursor image from the mouse cursor data into the desktop image via the core processor to obtain the composite desktop image, the method further includes: The desktop image is mapped to a first memory address, which is a memory address accessible to the core processor.

3. The method according to claim 2, characterized in that, After obtaining the desktop image, the process further includes: The desktop image is mapped to a second memory address, which is a memory address accessible by the graphics processor.

4. The method according to claim 1, characterized in that, The desktop image is stored in the storage area corresponding to the graphics card driver of the computer device; Before the process of compositing the mouse cursor image from the mouse cursor data into the desktop image via the core processor to obtain the composite desktop image, the method further includes: The desktop image is moved and stored in the memory corresponding to the core processor; Before encoding the desktop composite image using the graphics processor to obtain the current screen recording, the method further includes: The desktop composite image is moved and stored in the memory corresponding to the graphics processor.

5. The method according to any one of claims 1-4, characterized in that, Before the process of compositing the mouse cursor image from the mouse cursor data into the desktop image via the core processor to obtain the composite desktop image, the method further includes: The image format of the desktop image is converted to a first image format, where the image data size corresponding to the first image format is smaller than the image data size corresponding to the original image format of the desktop image. The image format of the mouse cursor image is converted to a second image format that matches the first image format, wherein the image data size of the second image format is smaller than the image data size of the original mouse cursor image format.

6. The method according to any one of claims 1-4, characterized in that, Before encoding the desktop composite image using the graphics processor to obtain the current screen recording, the method further includes: The image format of the desktop composite image is converted to a first image format, where the image data size of the first image format is smaller than the image data size of the original desktop composite image format.

7. The method according to any one of claims 1-4, characterized in that, The size of the mouse cursor image is smaller than the size of the desktop image, and the mouse cursor data also includes target position coordinates, which are used to reflect the position of the mouse cursor image on the current desktop. The step of merging the mouse cursor image from the mouse cursor data into the desktop image through the core processor to obtain a composite desktop image includes: The core processor draws the mouse cursor image at the image position corresponding to the target position coordinates in the desktop image to obtain the composite desktop image.

8. The method according to any one of claims 1-4, characterized in that, The size of the mouse cursor image is the same as the size of the desktop image; The step of merging the mouse cursor image from the mouse cursor data into the desktop image through the core processor to obtain a composite desktop image includes: The core processor performs superposition operations on the image values ​​of the same image position in the mouse cursor image and the desktop image to obtain the composite desktop image.

9. A computer device, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor including a core processor and a graphics processor, the core processor and the graphics processor cooperating to execute one or more computer programs stored in the memory, the core processor and the graphics processor, when executing the one or more computer programs, causing the computer device to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-8.

Images