Screen recording method and related apparatus

Abstract

The present application discloses a screen recording method and a related apparatus, applied to a recording application process corresponding to an application, the recording application process comprising a primary rendering thread, a secondary rendering thread and an encoding thread, and the secondary rendering thread and the encoding thread being enabled in response to a screen recording operation triggered by the application. The method comprises: by means of the primary rendering thread, copying rendering data stored in a back buffer in the primary rendering thread to a bridge buffer, a shared texture identifier being bound to the bridge buffer (501); by means of the secondary rendering thread and on the basis of the shared texture identifier, acquiring the rendering data from the bridge buffer, drawing the rendering data onto an input rendering surface provided by the encoding thread, to obtain an output rendering surface, and transmitting the output rendering surface to the encoding thread (502); and encoding the output rendering surface by means of the encoding thread, to obtain recorded video frame data (503). The method can meet the actual screen recording requirements of users, and improve the screen recording efficiency.

Description

Screen recording method and related device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 20, 2023, application number 202310740938.6, and application name “A screen recording method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to screen recording technology. Background Art

[0003] Screen recording is the ability for a terminal device to record the content displayed on its screen and generate a corresponding video. With the growing diversity of internet services, screen recording has been widely used in various scenarios, such as live game streaming and online meetings.

[0004] In related technologies, applications on terminal devices that have screen recording requirements (such as game applications, conference applications, etc.) need to use the screen recording capabilities provided by the terminal device's system to record the display content. Specifically, the display content is recorded by calling the screen rendering process in the system.

[0005] However, the above recording method usually has the following problems: calling the screen rendering process in the system to record the display content will record all the display content on the screen; in this way, the recorded content not only includes the display content of the application with screen recording requirements required by the user, but also includes the display content or notification content of other applications on the terminal device that the user does not need, which does not meet the user's actual screen recording needs; and, since all the display content on the screen is recorded, there is a need to obtain the display content of other applications. Therefore, the application with screen recording requirements also needs to apply for system permissions before screen recording, and then perform screen recording after obtaining authorization, which is cumbersome.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a screen recording method and related devices, which can effectively meet the user's actual screen recording needs, simplify the operation process of screen recording, and improve the efficiency of implementing the screen recording function.

[0008] In view of this, a first aspect of the present application provides a screen recording method, which is executed by a computer device through a recording application process corresponding to an application, wherein the recording application process includes a main rendering thread, a secondary rendering thread, and an encoding thread, and the secondary rendering thread and the encoding thread are started in response to a screen recording operation triggered by the application; the method includes:

[0009] Copying the rendering data stored in the back buffer of the main rendering thread to the bridge buffer through the main rendering thread; the bridge buffer is bound to a shared texture identifier;

[0010] Obtaining the rendering data from the bridge buffer based on the shared texture identifier through the secondary rendering thread; drawing the rendering data onto an input rendering surface provided by the encoding thread to obtain an output rendering surface, and transmitting the output rendering surface to the encoding thread;

[0011] The encoding thread performs encoding processing based on the output rendering surface to obtain recorded video frame data.

[0012] A second aspect of the present application provides a screen recording apparatus, which is deployed on a computer device and supports the running of a recording application process corresponding to an application. The recording application process includes a main rendering thread, a secondary rendering thread, and an encoding thread. The secondary rendering thread and the encoding thread are started in response to a screen recording operation triggered by the application. The apparatus includes:

[0013] A data copy module, configured to copy the rendering data stored in the back buffer of the main rendering thread to a bridge buffer via the main rendering thread; the bridge buffer is bound to a shared texture identifier;

[0014] a rendering module, configured to obtain, through the secondary rendering thread, the rendering data from the bridge buffer based on the shared texture identifier; draw the rendering data onto an input rendering surface provided by the encoding thread to obtain an output rendering surface; and transmit the output rendering surface to the encoding thread;

[0015] The encoding module is used to perform encoding processing based on the output rendering surface through the encoding thread to obtain the recorded video frame data.

[0016] A third aspect of the present application provides a computer device, the device comprising a processor and a memory:

[0017] The memory is used to store computer programs;

[0018] The processor is used to execute the steps of the screen recording method described in the first aspect according to the computer program.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the steps of the screen recording method described in the first aspect.

[0020] In a fifth aspect, the present application provides a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the steps of the screen recording method described in the first aspect.

[0021] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0022] An embodiment of the present application provides a screen recording method, in which each application is assigned an independent recording application process, so that the display screen of the application itself is recorded through the recording application process. Therefore, the screen recording method is applied to the recording application process corresponding to the application, and the recording application process includes a main rendering thread, a secondary rendering thread and an encoding thread, wherein the secondary rendering thread and the encoding thread are started in response to a screen recording operation triggered by the application. In the screen recording method, the rendering data stored in the back buffer of the main rendering thread is copied to the bridge buffer through the main rendering thread, and the bridge buffer is bound to a shared texture identifier. The secondary rendering thread obtains the rendering data from the bridge buffer based on the shared texture identifier, draws the rendering data onto the input rendering surface provided by the encoding thread, obtains the output rendering surface, and transmits the output rendering surface to the encoding thread; the output rendering surface is encoded by the encoding thread to obtain the recorded video frame data. In the above-mentioned screen recording method, since the corresponding recording application is pre-assigned to the application to record its own display screen, the recording of the application's own display screen is achieved through the main rendering thread, secondary rendering thread and encoding thread in the recording application process, and does not involve the use of the screen rendering process in the system. Therefore, the display content or notification content of other applications not related to the application will not be recorded, which effectively meets the user's actual screen recording needs. Moreover, since the above-mentioned screen recording method is implemented through the recording application process corresponding to the application, the rendering data can be directly obtained and used in the recording application process. Without involving the acquisition of data in other applications, there is no need to apply for system permissions before screen recording, which simplifies the operation process of screen recording and improves the implementation efficiency of the screen recording function. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a schematic diagram of a screen recording method provided by the related art;

[0024] FIG2 is a specific flow chart of obtaining video frame data through a system API provided by the related art;

[0025] FIG3 is a schematic diagram of an application scenario of the screen recording method provided in an embodiment of the present application;

[0026] FIG4 is a schematic diagram of the rendering logic of an application on a terminal device provided by an embodiment of the present application;

[0027] FIG5 is a flow chart of a screen recording method provided by an embodiment of the present application;

[0028] FIG6 is a schematic diagram of the interaction between various threads in a recording application process provided by an embodiment of the present application;

[0029] FIG7 is a schematic structural diagram of a screen recording device provided in an embodiment of the present application;

[0030] FIG8 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0031] FIG9 is a schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0033] The terms "first," "second," "third," "fourth," etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0034] In the related art, when an application on a terminal device that has a screen recording requirement, such as a game application, a conference application, etc., performs screen recording, the display content is recorded by calling a screen rendering process in the system.

[0035] Refer to Figure 1, which is a schematic diagram of the screen recording method in the above-mentioned related technology. When the user triggers the screen recording process through the screen recording application in the terminal device (which can be any application with screen recording function), the terminal device will record all the content displayed on the screen, involving the display content of multiple applications in the terminal device. Therefore, before starting screen recording, the screen recording application needs to apply for screen recording permission from the system. When the screen recording application receives the result of the permission application that the system agrees to perform screen recording, the screen recording application can call the system's application programming interface (API) to start recording the content displayed on the screen of the terminal device. After starting screen recording, the screen recording application obtains each video frame obtained by recording through the system API, and transmits the video frame to the encoder. The encoder encodes the video frame to obtain the video frame data obtained by screen recording. According to the actual scene requirements, the video frame data is transmitted to the server or stored in the local video file of the terminal device.

[0036] The specific process for obtaining video frame data through the system API is shown in Figure 2. In Figure 2, the screen recording application process, as the recording requester, completes the screen recording permission application and initiates screen recording through the system-provided screen recording API. It also creates an encoder, which provides a rendering surface for carrying the screen recording content and transmits this rendering surface to the system screen rendering and management process through the screen recording API. The system screen rendering and management process collects the drawing instructions and content rendering surfaces from all application processes on the terminal device screen. Based on the rendering content corresponding to the drawing instructions and content rendering surfaces in each application process, it synthesizes the final display image and displays it on the terminal device screen. After initiating screen recording, the system screen rendering and management process writes the collected rendering content into the rendering surface provided by the encoder and transmits the rendered content back to the encoder in the screen recording application. The encoder then encodes the rendered content, ultimately generating the video frame data required for screen recording. Based on the actual recording requirements, this video frame data is written to a local video file or pushed to a server.

[0037] It can be seen that in the screen recording method described in the relevant technology, each video frame in the screen recording is obtained through the screen rendering process in the system - the system screen rendering and management process. By calling the screen rendering process in the system to record the display content, all the display content on the screen will be recorded. In this way, the recorded content not only includes the display content of the application with screen recording requirements (i.e., the screen recording application) required by the user, but also includes the display content or notification content of other applications on the terminal device that the user does not need, which does not meet the user's actual screen recording needs and causes unnecessary waste of data processing resources; and, since all the display content on the screen is recorded, there is a need to obtain the display content of other applications. Therefore, the application with screen recording requirements also needs to apply for system permissions before screen recording, and then perform screen recording after obtaining authorization, which is cumbersome.

[0038] At the same time, when calling the screen rendering process in the system to record the display content, cross-process data interaction is required, that is, the rendering surface is interactively rendered between the screen recording application process and the screen rendering process in the system. When a large number of video frames need to be transmitted across processes, certain performance issues will arise.

[0039] In order to solve the above problems, an embodiment of the present application provides a screen recording method, which is applied to a recording application process corresponding to an application, including a main rendering thread, a secondary rendering thread and an encoding thread, wherein the secondary rendering thread and the encoding thread are started in response to a screen recording operation triggered by an application. In this screen recording method, the main rendering thread copies the rendering data stored in the back buffer of the main rendering thread to a bridge buffer, which is bound to a shared texture identifier. The secondary rendering thread obtains the rendering data from the bridge buffer based on the shared texture identifier, draws the rendering data onto the input rendering surface provided by the encoding thread, obtains the output rendering surface, and transmits the output rendering surface to the encoding thread; the encoding thread encodes the output rendering surface to obtain the recorded video frame data.

[0040] In the above-mentioned screen recording method, the recording of the application's own display screen is achieved by recording the main rendering thread, secondary rendering thread and encoding thread within the application process, and does not involve the use of the screen rendering process in the system. Therefore, the display content or notification content of other applications unrelated to the application will not be recorded, effectively meeting the user's actual screen recording needs. Moreover, since the above-mentioned screen recording method is implemented through the recording application process corresponding to the application, the rendering data can be directly obtained and used within the recording application process. Without involving the acquisition of data in other applications, there is no need to apply for system permissions before screen recording, which simplifies the operation process of screen recording and improves the implementation efficiency of the screen recording function.

[0041] At the same time, the screen recording method provided in the embodiment of the present application only performs data interaction within the recording application process, that is, data is transmitted between the main rendering thread, secondary rendering thread and encoding thread within the recording application process. Compared with the related technology of cross-process data transmission between the system and the screen recording application, the screen recording method provided in the embodiment of the present application avoids related performance issues.

[0042] The screen recording method provided in the embodiments of this application can be implemented by a computer device, which can be a terminal device. Terminal devices include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, and vehicle-mounted terminals. The terminal device can be directly or indirectly connected to the server via wired or wireless communication, and this application does not impose any restrictions thereon.

[0043] To facilitate understanding of the screen recording method provided in the embodiments of the present application, the following uses the screen recording method executed by a terminal device as an example to exemplify the application scenario of the screen recording method.

[0044] See Figure 3, which is a schematic diagram of an application scenario of the screen recording method provided in an embodiment of the present application. In the application scenario shown in Figure 3, a terminal device 300 is used as an example of a computer device for executing the screen recording method provided in an embodiment of the present application.

[0045] As shown in Figure 3, in the embodiment of the present application, the application involved can be any application with a screen recording function, exemplarily a game application, or other types of applications. After the screen recording process is triggered in the application, it is necessary to obtain the display content of the application on the screen of the terminal device 300 and realize the screen recording of the application by saving the display content in real time.

[0046] If you need to record the display content on the screen when the application is running, a process specifically responsible for screen recording will be started. This process can be called a recording application process. This recording application process will capture the images and sounds on the screen (if audio recording is set) and encode them into video frame data. In one possible implementation method, the operating system and various applications may implement security measures and permission management. Therefore, in the embodiment of the present application, by assigning an independent recording application process to each application, it is ensured that each recording application process is subject to appropriate permissions and access control, thereby protecting the security of user data and avoiding recording the display content or notification content of irrelevant applications. In other words, a correspondence between each application and the recording application process is set, so that in the screen recording process of each application, the recording application process corresponding to the application can be selected based on the correspondence to execute the screen recording process of the application.

[0047] The recording application process of the application may include a main rendering thread, a secondary rendering thread and an encoding thread, and the main rendering thread, the secondary rendering thread and the encoding thread jointly complete the screen recording function of the recording application process. The data transmission between the threads in the recording application process is carried out within the recording application process. Among them, the recording application process can be the process used for screen recording in the application. When the screen recording is not performed, the rendering logic of the application on the terminal device 300 is shown in Figure 4. The main rendering thread is used to render the content that needs to be displayed in the application to the screen of the terminal device 300 and provide it to the user for viewing. When the main rendering thread performs the rendering operation, the drawing engine or other screen rendering framework produces drawing instructions based on the model data (such as models of people, vehicles, buildings, etc.) and texture data in the application, and generates a specific rendering function based on the drawing instructions. The rendering function is executed by the main rendering thread to render the model data and texture data into the picture, and finally the rendering data is displayed on the screen.

[0048] After screen recording begins, the video frame currently being displayed on the screen of terminal device 300 is the current video frame. The back buffer of the primary rendering thread stores rendering data corresponding to the next video frame following the current one. As the video frame plays on the screen of terminal device 300, real-time access to the content displayed in the application can be achieved by acquiring the rendering data in the back buffer. Therefore, upon detecting the initiation of screen recording, the primary rendering thread creates a shared texture identifier for sharing rendering data and a bridge buffer for storing rendering data, and binds the shared texture identifier to the bridge buffer. The rendering data in the back buffer is copied by the primary rendering thread to the bridge buffer. After the secondary rendering thread acquires the shared texture identifier, it can then acquire the rendering data in the bridge buffer bound to it based on the shared texture identifier. This means that the secondary rendering thread can acquire the rendering data in the back buffer. Furthermore, since the content corresponding to the shared texture identifier is the rendering data in the bridge buffer, the content corresponding to the shared texture identifier is also updated as the rendering data is copied from the back buffer to the bridge buffer. This enables the secondary rendering thread to acquire the rendering data in the back buffer in real time, and screen recording is achieved through this real-time acquired rendering data.

[0049] In the recording application process, the encoding thread is used to encode the video screen obtained by the secondary rendering thread to obtain the video frame data of the screen recording. During screen recording, the encoding thread will create an input rendering surface and provide the input rendering surface to the secondary rendering thread to carry the rendering content generated by the secondary rendering thread. In the secondary rendering thread, after obtaining the rendering data, the rendering data is drawn to the input rendering surface provided by the encoding thread. After the drawing is completed, the output rendering surface corresponding to the input rendering surface is obtained. The output rendering surface is a video frame displayed on the screen of the terminal device 300 during the screen recording process. Since the video frame is generated based on the rendering data in the application, the video frame only includes the display content in the application, which meets the user's screen recording needs on the application. The output rendering surface drawn in the secondary rendering thread is transmitted to the encoding thread. When the encoding thread receives the output rendering surface, it encodes the output rendering surface. After the encoding process, each output rendering surface forms the video frame data of this screen recording process. In the video frame data, each video frame is arranged in chronological order. The video frame data is stored frame by frame, which is the video generated by the screen recording.

[0050] That is to say, when screen recording is performed in an application, rendering data can be obtained and used within the application by recording the application process, avoiding cross-process transmission of video frames; the rendering data of the back buffer is obtained through the main rendering thread and copied to the bridge buffer. The secondary rendering thread can obtain the rendering data in the bridge buffer bound to the shared texture identifier created by the main rendering thread based on the shared texture identifier, and draw the rendering data on the input rendering surface provided by the encoding thread. The output rendering surface obtained after the drawing is completed is transferred to the encoding thread for encoding, and finally the video frame data of this screen recording is obtained. Only a simple rendering data copy operation is performed in the main rendering thread, and the secondary rendering of the rendering data in the application is performed by the secondary rendering thread, which reduces the impact of screen recording on the main rendering thread; at the same time, the recording of the application's own display screen is achieved by recording the application process, which does not involve the rendering content and data in other applications. There is no need to apply for system permissions before screen recording, which simplifies the operation process of screen recording and effectively meets the user's needs for screen recording within the application.

[0051] Referring to FIG5 , FIG5 is a flowchart of a screen recording method provided in an embodiment of the present application. The method can be executed by a terminal device as the aforementioned computer device. The method is applied to a recording application process corresponding to an application. The recording application process includes a main rendering thread, a secondary rendering thread, and an encoding thread. The secondary rendering thread and the encoding thread are started in response to a screen recording operation triggered by the application. As shown in FIG5 , the method specifically includes the following steps:

[0052] Step 501: The main rendering thread copies rendering data stored in the back buffer of the main rendering thread to a bridge buffer; the bridge buffer is bound to a shared texture identifier.

[0053] Rendering can refer to the process by which software on various electronic devices, such as computers, mobile devices, and game consoles, generates an image from a model. A model is a description of a three-dimensional object or virtual scene that is strictly defined in a language or data structure, including information such as geometry, viewpoint, texture, lighting, and shadows. Textures are usually mapped onto the surface of a three-dimensional model to make the surface of the model appear more realistic and detailed. It should be noted that this application does not limit the type of terminal device that executes the screen recording method. The terminal device can be a mobile phone, computer, etc.

[0054] In the embodiment of the present application, the process of generating a specific scene screen based on abstract model data and texture data is rendering. For example, the process of generating different display screens corresponding to each control in the game application based on the operating parameters such as the change mode, change angle, and change speed of each control saved in the game application. In the terminal device, the main rendering thread can use the open graphics programming interface (Open Graphics Library, OpenGL). OpenGL is a cross-language, cross-platform application program interface for rendering two-dimensional or three-dimensional vector graphics. It is widely used in various image rendering scenarios, including but not limited to game rendering, computer-aided design and drawing, etc., to realize the rendering of various graphics rendering scenarios in terminal device applications.

[0055] The back buffer and the bridge buffer are both frame buffers. The frame buffer can be used as a memory buffer to store pixel data of rendering results, including color, depth, template and other information.

[0056] Among them, the main rendering thread is the thread that renders the display content in the application during the application running process, and is used to render the display content in the application to obtain rendering data. The obtained rendering data is saved in the back buffer of the main rendering thread. The rendering data is a kind of pixel data. By drawing this pixel data, the display screen of the application on the terminal device screen can be obtained. It should be noted that this application does not impose any restrictions on the type of application. For example, the application can be a game application, a conference application, a live broadcast application, etc.

[0057] The bridge buffer is a buffer created by the main rendering thread for sharing rendering data. In an embodiment of the present application, in addition to being stored in the back buffer, the rendering data generated in the main rendering thread is also copied to the bridge buffer for storage. In an embodiment of the present application, the main rendering thread can copy the rendering data stored in the back buffer to the bridge buffer. Since the bridge buffer is bound to a shared texture identifier, the shared texture identifier is a unique identifier for identifying a texture object, which can be used to reference the texture object during rendering. Whether the bridge buffer is bound to the shared texture identifier reflects whether the texture object in the bridge buffer (such as the rendering data in the embodiment of the present application) can be referenced. Therefore, when the bridge buffer is bound to a shared texture identifier, the shared texture identifier can be used to reference the texture object in the bridge buffer and obtain the rendering data copied from the back buffer by the bridge buffer. Among them, the shared texture identifier can be various types of identifiers, such as symbols, numbers, letters, etc.

[0058] When the application is running normally, the main rendering thread is mainly used to render the display content in the application to generate a display screen. When performing screen recording, the embodiment of the present application only performs the copy operation of the rendering data in the main rendering thread, and the operation of copying the rendering data from the back buffer to the bridge buffer in the main rendering thread is run on the cache of the graphics processing unit (GPU). The amount of processed data is small and the execution speed is fast, which reduces the impact on the frame rate and rendering efficiency when the main rendering thread is running normally.

[0059] In a possible implementation, the screen recording method of the present application may further include the following steps:

[0060] A secondary graphics program context corresponding to the secondary rendering thread is created through the secondary rendering thread according to the primary graphics program context corresponding to the primary rendering thread, and the primary graphics program context is determined as the parent graphics program context of the secondary graphics program context.

[0061] The shared texture identifier created in the main rendering thread corresponding to the parent graphics program context supports use by the secondary rendering thread corresponding to the secondary graphics program context.

[0062] In the OpenGL application programming interface, the graphics program context (OpenGL Context) represents the current state and environment of OpenGL, and contains all OpenGL state information, such as the current rendering mode, buffer objects, texture objects, shader programs, etc. In OpenGL, all rendering operations are performed in the current OpenGL context.

[0063] In an embodiment of the present application, when the primary rendering thread is running, the corresponding primary graphics program context contains the program running status record and running data of the primary rendering thread. The primary rendering thread provides its own primary graphics program context to the secondary rendering thread, and the secondary rendering thread can create a secondary graphics program context corresponding to itself based on the primary graphics program context. The secondary graphics program context includes the program running status record and running data of the secondary rendering thread.

[0064] When the secondary rendering thread creates a secondary graphics program context based on the primary graphics program context, it can also specify a corresponding parent graphics program context for the secondary graphics program context. After specifying the parent graphics program context, the shared texture identifier in the thread corresponding to the parent graphics program context can be made available to the thread corresponding to the child graphics program context. In this embodiment of the present application, the primary graphics program context corresponding to the primary rendering thread is determined to be the parent graphics program context of the secondary graphics program context, and the secondary graphics program context is then the child graphics program context of the primary graphics program context. In this way, the shared texture identifier created in the primary rendering thread corresponding to the parent graphics program context can be used by the secondary rendering thread corresponding to the child graphics program context, forming the basis for sharing rendering data between the primary rendering thread and the secondary rendering thread. Based on the shared texture identifier, the secondary rendering thread can access the bridge buffer bound to the shared texture identifier and obtain rendering data from the bridge buffer.

[0065] A secondary graphics program context corresponding to the secondary rendering thread is created through the main graphics program context, and the main graphics program context is determined as the parent graphics program context of the secondary graphics program context, so that the secondary rendering thread can use the shared texture identifier in the main rendering thread to realize the sharing of rendering data between the main rendering thread and the secondary rendering thread, thereby improving the efficiency of the secondary rendering thread in obtaining rendering data.

[0066] In a possible implementation, the screen recording method of the present application may further include the following steps (not shown in the accompanying drawings):

[0067] Step 11: If the main rendering thread detects that the shared texture data is not initialized, the main rendering thread creates a shared texture identifier and a bridge buffer, and binds the shared texture identifier and the bridge buffer; the main rendering thread executes to copy the rendering data stored in the back buffer in the main rendering thread to the bridge buffer.

[0068] In the main rendering thread, the main rendering thread can detect whether the shared texture data has been initialized. In one possible implementation, the shared texture data in the main rendering thread can be determined to be initialized by detecting whether a shared texture identifier and a bridge buffer bound to the shared texture identifier exist. If a shared texture identifier and a bridge buffer bound to the shared texture identifier exist in the main rendering thread, the shared texture data in the main rendering thread is considered to have been initialized. Otherwise, the shared texture data in the main rendering thread is considered to have not been initialized.

[0069] In an embodiment of the present application, the purpose of detecting whether the shared texture data has been initialized is to determine whether there is a corresponding buffer to save the copied rendering data after the copy operation is performed on the rendering data in the back buffer of the main rendering thread; if so, the copy operation can be performed directly on the data in the back buffer; if not, it is necessary to create a corresponding shared texture identifier and a bridge buffer to save the rendering data copied from the back buffer.

[0070] If the main rendering thread detects that the shared texture data is not initialized, the main rendering thread creates a shared texture identifier and a bridge buffer, and binds the shared texture identifier and the bridge buffer; the main rendering thread executes to copy the rendering data stored in the back buffer in the main rendering thread to the bridge buffer.

[0071] When the shared texture data in the main rendering thread is not initialized, the main rendering thread needs to create a shared texture identifier and a bridge buffer and bind the two. The bridge buffer is used to store the rendering data copied from the back buffer. Based on the aforementioned copy operation of the rendering data and the binding relationship between the bridge buffer and the shared texture identifier, the shared texture identifier is used to map the rendering data in the bridge buffer. Based on this mapping, the secondary rendering thread can obtain the same rendering data as in the back buffer of the main rendering thread through the shared texture identifier; and when new rendering data is copied and the rendering data in the bridge buffer is updated, the mapping between the shared texture identifier and the rendering data will also be updated accordingly.

[0072] Binding a shared texture identifier to a bridge buffer can refer to associating the shared texture identifier with the bridge buffer to enable efficient sharing and transmission of rendering data during the rendering process. Through this binding relationship, the texture data (i.e., rendering data) associated with the shared texture identifier can be read directly from the bridge buffer without reloading or copying the data. This can significantly reduce data transmission overhead and improve rendering efficiency and performance.

[0073] Step 12: If it is detected through the main rendering thread that the shared texture data has been initialized, the main rendering thread is executed to copy the rendering data stored in the back buffer of the main rendering thread to the bridge buffer.

[0074] When the shared texture data in the main rendering thread has been initialized, a copy operation is performed on the rendering data stored in the back buffer of the main rendering thread based on the shared texture identifier created by the main rendering thread when the shared texture data is initialized and the bridge buffer bound to the shared texture identifier, and the aforementioned rendering data is copied from the back buffer to the bridge buffer. Since the bridge buffer is bound to the shared texture identifier, the sharing of the rendering data in the bridge buffer can be achieved based on the shared texture identifier, thereby achieving the sharing of the rendering data in the back buffer of the main rendering thread.

[0075] The main rendering thread initializes shared texture data, creates a shared texture identifier and a bridge buffer bound to the shared texture identifier, saves the rendering data copied from the back buffer in the bridge buffer, and implements sharing of rendering data based on the shared texture identifier bound to the bridge buffer. The texture data (i.e., rendering data) associated with the shared texture identifier can be read directly from the bridge buffer without reloading or copying the data. This can greatly reduce data transmission overhead and improve rendering efficiency and performance.

[0076] In one possible implementation, step 501 is implemented as follows (not shown in the drawings):

[0077] Step 21: Through the main rendering thread, set the bridge buffer as the current write buffer in the main rendering thread, and set the back buffer as the current read buffer in the main rendering thread.

[0078] During the operation of the main rendering thread, a current read buffer for reading data and a current write buffer for writing data are set. The data read from the current read buffer will be written to the current write buffer.

[0079] In an embodiment of the present application, when it is necessary to copy the rendering data in the back buffer of the main rendering thread, the back buffer will be set as the current read buffer so that the rendering data stored therein can be read from the current read buffer. At the same time, the bridge buffer will be used as the current write buffer in the main rendering thread for writing the aforementioned rendering data.

[0080] Step 22: Execute the buffer copy function through the main rendering thread to copy the rendering data from the current read buffer to the current write buffer.

[0081] A buffer copy function is a function that copies the contents of one buffer to another, such as the glBlitFramebuffer function in OpenGL. This function is used when copying rendered data from the back buffer. The main rendering thread executes the buffer copy function, which copies the rendered data from the current read buffer to the current write buffer, that is, copies the rendered data from the back buffer to the bridge buffer.

[0082] By setting the current write buffer and the current read buffer in the main rendering thread and executing the buffer copy function, rendering data sharing between the back buffer and the bridge buffer is achieved.

[0083] In one possible implementation, step 22 is implemented as follows:

[0084] The buffer copy function is executed by the main rendering thread to copy the color data stored in the color buffer in the current read buffer to the current write buffer.

[0085] In the main rendering thread, the rendering data stored in the current read buffer is a type of pixel data, which includes information such as color, depth, and template. Among them, the buffer that stores color is the color buffer. In OpenGL, the buffer that reads pixel data is usually the color buffer. The pixel data is the pixel value corresponding to each pixel on the terminal device display screen. The color of the corresponding pixel can be determined by the pixel value. When reading rendering data from the current read buffer, it can be set to read data from the color buffer of the current read buffer, read the color data stored in the color buffer, and copy the color data to the current write buffer, realizing the sharing of color data between the back buffer and the bridge buffer.

[0086] Color data is the content directly presented to the user on the screen. The ultimate goal of screen recording is to capture the content displayed on the screen, so color data is the most direct and critical information. The embodiment of the present application copies the color data to the current write buffer so that the color data can be directly read from the current write buffer for data sharing, facilitating the rapid rendering of the most direct and critical information of the content displayed on the screen, thereby better achieving screen recording.

[0087] As an embodiment, based on step 21 and step 22, after the screen recording is completed, the following steps may also be included:

[0088] In response to the screen recording stop operation triggered by the application, the read frame buffer target and the write frame buffer target in the main rendering thread are restored through the main rendering thread. The restoration process is used to restore the read frame buffer target and the write frame buffer target to the values ​​before responding to the screen recording operation.

[0089] The read frame buffer target is used to indicate the current read buffer, and the write frame buffer target is used to indicate the current write buffer.

[0090] During the screen recording process, the current read buffer pointed to by the read frame buffer target in the main rendering thread is the back buffer, and the current write buffer pointed to by the write frame buffer target is the bridge buffer. This setting is a specific setting in the screen recording scenario. After the screen recording is completed, this specific setting may affect the normal operation of the main rendering thread. Therefore, in response to the screen recording stop operation triggered by the application, when the screen recording ends, the main rendering thread needs to restore the read frame buffer target and write frame buffer target in the main rendering thread to the values ​​before the screen recording starts, so that the main rendering thread can be restored to the working state before the screen recording starts and continue to perform the rendering tasks before the screen recording starts.

[0091] After screen recording stops, the read frame buffer target and write frame buffer target in the main rendering thread are restored to their values ​​before responding to the screen recording operation, reducing the impact of screen recording on the main rendering thread in the application.

[0092] Step 502: Obtain the rendering data from the bridge buffer based on the shared texture identifier through the secondary rendering thread; draw the rendering data onto the input rendering surface provided by the encoding thread to obtain an output rendering surface, and transmit the output rendering surface to the encoding thread.

[0093] The secondary rendering thread can be a thread that runs independently of the main rendering thread in the recording application process. It is used to perform secondary rendering on the rendering data obtained by the main rendering thread, thereby restoring the rendering content drawn by the main rendering thread according to the rendering data, and performing screen recording based on the rendering content.

[0094] In an embodiment of the present application, the secondary rendering thread can access the bridge buffer bound to the shared texture identifier based on the shared texture identifier, and then reference the corresponding texture object from the bridge buffer. That is, based on the shared texture identifier, the rendering data copied from the back buffer can be obtained in the bridge buffer. The sharing of the rendering data in the back buffer is achieved through the shared texture identifier and the bridge buffer, so that the secondary rendering thread can draw the same rendering content as the main rendering thread. It should be noted that in an embodiment of the present application, the rendering data is the pixel data collected by the main rendering thread, which records the pixel values ​​corresponding to each pixel point in the display screen, and the rendering content is the specific content drawn based on the rendering data.

[0095] After obtaining these rendering data, the secondary rendering thread needs to perform secondary rendering on the rendering data in the application, and draw the rendering content obtained by the secondary rendering into the input rendering surface provided by the encoding thread. Among them, the input rendering surface is a blank surface created by the encoding thread for carrying rendering content. When drawing, you can use drawing commands to draw rendering data on the input rendering surface. In an embodiment of the present application, the input rendering surface can be an abstract interface for displaying the picture, and drawing the rendering data to the input rendering surface can be writing the rendering data into the memory buffer of the input rendering surface for subsequent display or processing. When the secondary rendering thread completes drawing the input rendering surface, it obtains the corresponding output rendering surface, which is a video frame obtained by screen recording. The secondary rendering thread then transmits the output rendering surface to the encoding thread for subsequent processing in the encoding thread.

[0096] In one possible implementation, the secondary rendering thread draws the input rendering surface through the following steps (not shown in the figure):

[0097] Step 31: Build the shader through the secondary rendering thread;

[0098] Step 32: Draw the rendering result through the shader based on the mapping relationship between the vertex coordinates of the input rendering surface and the texture coordinates of the rendering data;

[0099] Step 33: Output the rendering result to the input rendering surface to obtain the output rendering surface.

[0100] A shader is a drawing tool that draws graphics based on rendering data. When creating a shader in the secondary rendering thread, a vertex shader and a fragment shader can be created through the rendering function in OpenGL. The vertex shader is used to draw output vertices, and the fragment shader is used to draw output colors. By linking the vertex shader and the fragment shader, the program object corresponding to the shader can be obtained.

[0101] After the secondary rendering thread obtains the rendering data and the input rendering surface, it needs to match the texture coordinates of the rendering data with the vertex coordinates of the input rendering surface before it can draw the content corresponding to the rendering data into the input rendering surface and restore the rendering content drawn by the main rendering thread based on the rendering data.

[0102] When rendering using a shader, the rendering result corresponding to the rendering data is drawn based on the mapping relationship between the vertex coordinates of the input rendering surface and the texture coordinates of the rendering data. Specifically, during the rendering process, the input rendering surface can be divided into two parts along the diagonal of the rectangle corresponding to the input rendering surface, each part containing triangles. The triangles in the two parts are drawn separately according to the aforementioned mapping relationship to complete the rendering of the input rendering surface.

[0103] The rendering result obtained based on the rendering data is drawn on the input rendering surface provided by the encoding thread. For example, the swapbuffer function in OpenGL can be used to draw the rendering result onto the input rendering surface. In the secondary rendering thread, the input rendering surface is the output object of the secondary rendering thread's rendering result.

[0104] The embodiment of the present application uses shaders for secondary rendering, which can generate more realistic and vivid images. In addition, the shaders run on the GPU, which can fully utilize the GPU's parallel processing capabilities. Using shaders for secondary rendering can greatly speed up the rendering speed and improve rendering efficiency.

[0105] Step 503: Encoding the output rendering surface through the encoding thread to obtain recorded video frame data.

[0106] The encoding thread is a thread used to encode the output rendering surface to obtain the recorded video frame data. After being processed by the main rendering thread and the secondary rendering thread, the output rendering surface corresponding to each frame displayed by the application during the screen recording process is output to the encoding thread for encoding. The encoding process is the process of encoding and sorting each output rendering surface to obtain a video frame sequence. After the encoding thread obtains the output rendering surface, it encodes the video frame images corresponding to each output rendering surface in a preset order to generate video frame data, wherein the preset order can be the time order corresponding to the recording; the video frame data can be a video frame combination sequence arranged in a preset order. By playing the video frame combination sequence, all video frames displayed by the application during the screen recording process can be viewed, and the video corresponding to this screen recording can be obtained.

[0107] In an embodiment of the present application, the video frame data finally generated by the encoding thread can be written into the local video file of the terminal device for viewing, or uploaded to the server for backup, meeting the user's different screen recording needs for applications.

[0108] It should be noted that the video frame data in the embodiments of the present application can be output in a variety of different forms, and the present application does not make any specific limitations on this.

[0109] In one possible implementation, the encoding thread encodes the output rendering surface through the following steps (not shown in the drawings):

[0110] Step 41: Create a media encoder through the encoding thread;

[0111] Step 42: Create an input rendering surface through the media encoder and transfer the input rendering surface to the secondary rendering thread;

[0112] At this time, step 503 can be implemented as follows:

[0113] The media encoder performs encoding processing based on the output rendering surface to obtain video frame data.

[0114] In OpenGL, the media encoder is used to encode the original video frame image data according to international standards so that it can be efficiently transmitted and stored on the Internet. In the embodiment of the present application, the media encoder is used to encode the output rendering surface output by the secondary rendering thread.

[0115] When creating a media encoder through an encoding thread, the media encoder is provided by the system platform of the terminal device and can be created by calling a common system API. It should be noted that this application does not specifically limit the system platform type of the terminal device.

[0116] After the media encoder is created, the media encoder can create a blank input rendering surface and transmit the blank input rendering surface to the secondary rendering thread. After being processed by the secondary rendering thread, the input rendering surface can be used as input data for the media encoder in the encoding thread.

[0117] The media encoder's data input comes from the secondary rendering thread. The corresponding input data is the output rendering surfaces drawn by the secondary rendering thread based on the rendering data. The media encoder encodes these output rendering surfaces and arranges them in a preset order to generate the video frame data for the screen recording. This video frame data can be transmitted to a server or stored in a local video file on the terminal device, depending on the user's needs. The video frame data includes all images displayed on the terminal device screen during the screen recording.

[0118] The encoding thread creates a media encoder and uses the media encoder to create an input rendering surface, which can bring benefits such as hardware acceleration, efficient resource management, flexibility and scalability, reduced latency, and improved video quality. These benefits make media encoders widely applicable in video processing, real-time rendering, and streaming media transmission.

[0119] An embodiment of the present application provides a screen recording method, which is applied to a recording application process corresponding to an application, and the recording application process includes a main rendering thread, a secondary rendering thread, and an encoding thread, wherein the secondary rendering thread and the encoding thread are started in response to a screen recording operation triggered by the application. In this screen recording method, the main rendering thread copies the rendering data stored in the back buffer of the main rendering thread to a bridge buffer, and the bridge buffer is bound to a shared texture identifier. The secondary rendering thread obtains the rendering data from the bridge buffer based on the shared texture identifier, draws the rendering data onto the input rendering surface provided by the encoding thread, obtains the output rendering surface, and transmits the output rendering surface to the encoding thread; the encoding thread encodes the output rendering surface to obtain the recorded video frame data. In the above-mentioned screen recording method, the recording of the application's own display screen is achieved by recording the main rendering thread, secondary rendering thread and encoding thread within the application process, and does not involve the use of the screen rendering process in the system. Therefore, the display content or notification content of other applications not related to the application will not be recorded, effectively meeting the user's actual screen recording needs. Moreover, since the above-mentioned screen recording method is implemented through the recording application process corresponding to the application, the rendering data can be directly obtained and used within the recording application process. Without involving the acquisition of data in other applications, there is no need to apply for system permissions before screen recording, which simplifies the operation process of screen recording and improves the implementation efficiency of the screen recording function.

[0120] Furthermore, during the screen recording process, the size of the video obtained by screen recording can be set according to the actual needs of the user, which is specifically achieved through the following steps (not shown in the accompanying drawings):

[0121] Step 51: Obtain the recording video size set by the application; the recording video size is used to indicate the size of the video frame obtained by screen recording;

[0122] Step 52: Create an input rendering surface corresponding to the size of the recorded video through the encoding thread, and provide the input rendering surface to the secondary rendering thread to obtain video frame data corresponding to the size of the recorded video based on the input rendering surface.

[0123] When performing screen recording in the application, users can set the recording video size. When the media encoder in the encoding thread creates the input rendering surface, it will create an input rendering surface that conforms to the recording video size set by the user. The secondary rendering thread draws the rendering content on the input rendering surface, and the obtained output rendering surface also conforms to the recording video size. Then, in the video frame data output by the final encoding thread, each video frame also conforms to the recording video size, thereby realizing the determination of the size of the output video frame of this screen recording by the recording video size, meeting the different needs of users for the screen recording video size.

[0124] Furthermore, during the screen recording process, you can also set the frame rate of the video obtained by screen recording according to the actual needs of the user. This can be achieved by:

[0125] Get the recording video frame rate set by the application; the recording video frame rate is used to indicate the frame rate of the video obtained by screen recording;

[0126] At this point, the secondary rendering thread draws the input rendering surface through the following steps:

[0127] Through the secondary rendering thread, according to the drawing cycle corresponding to the recorded video frame rate, the input rendering surface is drawn based on the rendering data to obtain the output rendering surface.

[0128] During screen recording, the recording video frame rate can be set. When the secondary rendering thread draws the input rendering surface, it draws the input rendering surface based on the drawing cycle corresponding to the recording video frame rate. For example, when the recording video frame rate is 30 frames and the frame rate of the main rendering thread itself is 60 frames, the secondary rendering thread draws the input rendering surface once every video frame. After the output rendering surface is encoded by the encoding thread, the frame rate corresponding to the video frame data conforms to the recording video frame rate.

[0129] When the recorded video frame rate is higher than the frame rate of the application's main rendering thread, the secondary rendering thread will draw duplicate frames when drawing the input rendering surface. For example, if the recorded video frame rate is 60 frames and the main rendering thread's frame rate is 30 frames, each video frame corresponding to the rendering data will be drawn twice. By increasing the number of duplicate frames, the frame rate of the video obtained by screen recording can be increased.

[0130] By setting the recording video frame rate, screen recording video frame data corresponding to different frame rates can be obtained, effectively meeting users' different requirements for screen recording video frame rates.

[0131] Refer to Figure 6, which is a schematic diagram of the interaction of various threads in the recording application process provided by an embodiment of the present application, wherein the recording application process includes a main rendering thread, a secondary rendering thread and an encoding thread.

[0132] In the main rendering thread, by receiving the drawing instruction, executing the specific rendering function corresponding to the drawing instruction, and judging whether the shared texture data has been initialized, if so, directly copying the rendering data in the back buffer to the bridge buffer bound to the shared texture identifier; if not, creating a shared texture identifier and a bridge buffer, and binding the shared texture identifier to the bridge buffer, and then copying the rendering data in the back buffer to the bridge buffer bound to the shared texture identifier; when the main rendering thread runs normally, the rendering data in the back buffer will be rendered and displayed on the screen for the user to watch.

[0133] In the secondary rendering thread, after the secondary rendering thread is started, a secondary graphics program context corresponding to the secondary rendering thread is created based on the main graphics program context provided by the main rendering thread, and the main graphics program context is confirmed as the parent graphics program context of the secondary graphics program context, and the secondary graphics program context is determined to be the child graphics program context of the main graphics program context; the secondary rendering thread obtains the input rendering surface created by the encoding thread, and based on the shared texture identifier, obtains the rendering data in the bridge buffer, performs secondary rendering based on the rendering data, draws the rendering result onto the input rendering surface, obtains the output rendering surface after the drawing is completed, and provides the output rendering surface to the encoding thread for encoding processing.

[0134] After the encoding thread is started, it will first create a media encoder, and then the media encoder will create an input rendering surface, which will be transferred to the secondary rendering thread for drawing. When the secondary rendering thread completes the drawing, the encoding thread receives the output rendering surface carrying the rendering results from the secondary rendering thread, and encodes the output rendering surface to obtain the video frame data of this screen recording.

[0135] Referring to FIG. 7 , FIG. 7 is a schematic structural diagram of a screen recording device 700 provided in an embodiment of the present application. The device supports the operation of a recording application process corresponding to an application. The recording application process includes a main rendering thread, a secondary rendering thread, and an encoding thread. The secondary rendering thread and the encoding thread are started in response to a screen recording operation triggered by the application. As shown in FIG. 7 , the screen recording device 700 includes:

[0136] The data copy module 701 is used to copy the rendering data stored in the back buffer of the main rendering thread to the bridge buffer through the main rendering thread; the bridge buffer is bound to a shared texture identifier;

[0137] The rendering module 702 is configured to obtain the rendering data from the bridge buffer based on the shared texture identifier through the secondary rendering thread; draw the rendering data onto the input rendering surface provided by the encoding thread to obtain an output rendering surface; and transmit the output rendering surface to the encoding thread;

[0138] The encoding module 703 is configured to perform encoding processing on the output rendering surface through the encoding thread to obtain recorded video frame data.

[0139] Optionally, the apparatus further includes a graphics program context creation module, configured to:

[0140] Creating, by the secondary rendering thread, a secondary graphics program context corresponding to the secondary rendering thread based on the primary graphics program context corresponding to the primary rendering thread, and determining the primary graphics program context as a parent graphics program context of the secondary graphics program context;

[0141] The shared texture identifier created in the primary rendering thread corresponding to the parent graphics program context supports use by the secondary rendering thread corresponding to the secondary graphics program context.

[0142] Optionally, the rendering module 702 is further configured to:

[0143] If the main rendering thread detects that the shared texture data is not initialized, the main rendering thread creates the shared texture identifier and the bridge buffer, and binds the shared texture identifier and the bridge buffer; and executes the main rendering thread to copy the rendering data stored in the back buffer of the main rendering thread to the bridge buffer;

[0144] If it is detected through the main rendering thread that the shared texture data has been initialized, the main rendering thread is executed to copy the rendering data stored in the back buffer of the main rendering thread to the bridge buffer.

[0145] Optionally, the data replication module 701 is specifically configured to:

[0146] By means of the main rendering thread, setting the bridge buffer as the current write buffer in the main rendering thread, and setting the back buffer as the current read buffer in the main rendering thread;

[0147] The main rendering thread executes a buffer copy function to copy the rendering data from the current read buffer to the current write buffer.

[0148] Optionally, the data replication module 701 is specifically configured to:

[0149] The buffer copy function is executed by the main rendering thread to copy the color data stored in the color buffer in the current read buffer to the current write buffer.

[0150] Optionally, the device further includes a parameter restoration module, configured to:

[0151] In response to a screen recording stop operation triggered by the application, performing, by the main rendering thread, a restoration process on a read frame buffer target and a write frame buffer target in the main rendering thread, wherein the restoration process is used to restore the read frame buffer target and the write frame buffer target to values ​​before responding to the screen recording operation;

[0152] The read frame buffer target is used to indicate the current read buffer area, and the write frame buffer target is used to indicate the current write buffer area.

[0153] Optionally, the rendering module 702 is specifically configured to:

[0154] Building a shader through the secondary rendering thread;

[0155] Drawing the rendering result based on the mapping relationship between the vertex coordinates of the input rendering surface and the texture coordinates of the rendering data by the shader;

[0156] The rendering result is output to the input rendering surface to obtain the output rendering surface.

[0157] Optionally, the encoding module 703 is further configured to:

[0158] Creating a media encoder through the encoding thread;

[0159] Creating the input rendering surface through the media encoder, and transmitting the input rendering surface to the secondary rendering thread;

[0160] The encoding module 703 is specifically used for:

[0161] The output rendering surface is encoded by the media encoder to obtain the video frame data.

[0162] Optionally, the device further comprises a size setting module, configured to:

[0163] Obtaining a recording video size set by the application; the recording video size is used to indicate the size of a video frame obtained by screen recording;

[0164] The input rendering surface corresponding to the recorded video size is created through the encoding thread, and the input rendering surface is provided to the secondary rendering thread to obtain the video frame data corresponding to the recorded video size based on the input rendering surface.

[0165] Optionally, the device further includes a frame rate setting module, configured to:

[0166] Obtaining a recording video frame rate set by the application; the recording video frame rate is used to indicate the frame rate of the video obtained by screen recording;

[0167] The rendering module 702 is specifically used for:

[0168] The input rendering surface is rendered based on the rendering data by the secondary rendering thread in accordance with a rendering cycle corresponding to the recorded video frame rate to obtain the output rendering surface.

[0169] The screen recording device provided in the embodiment of the present application records the display screen of the application itself by recording the main rendering thread, secondary rendering thread and encoding thread within the application process. It does not involve the use of the screen rendering process in the system. Therefore, it will not record the display content or notification content of other applications unrelated to the application, effectively meeting the user's actual screen recording needs. Moreover, since the above-mentioned screen recording device is implemented through the recording application process corresponding to the application, the rendering data can be directly obtained and used within the recording application process. Without involving the acquisition of data in other applications, there is no need to apply for system permissions before screen recording, which simplifies the operation process of screen recording and improves the implementation efficiency of the screen recording function.

[0170] The embodiment of the present application also provides a computer device, which may specifically be a terminal device or a server. The terminal device and server provided in the embodiment of the present application will be introduced below from the perspective of hardware entity.

[0171] See Figure 8, which is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. As shown in Figure 8, for ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of the present application. The terminal device can be any terminal device including a mobile phone, tablet computer, personal digital assistant (PDA), point of sales (POS), car computer, etc., taking the mobile phone as an example:

[0172] FIG8 shows a block diagram of a partial structure related to a terminal device provided in an embodiment of the present application. Referring to FIG8 , the computer includes: a radio frequency (RF) circuit 810, a memory 820, an input unit 830 (including a touch panel 831 and other input devices 832), a display unit 840 (including a display panel 841), a sensor 850, an audio circuit 860 (which is connected to a speaker 861 and a microphone 862), a wireless fidelity (WiFi) module 870, a processor 880, and a power supply 890 and other components. Those skilled in the art will understand that the computer structure shown in FIG8 does not constitute a limitation on the computer, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0173] The memory 820 can be used to store software programs and modules. The processor 880 executes the various computer functions and data processing by running the software programs and modules stored in the memory 820. The memory 820 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the computer (such as audio data, a phone book, etc.). In addition, the memory 820 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0174] The processor 880 is the computer's control center, connecting all computer components using various interfaces and circuits. It executes software programs and / or modules stored in the memory 820 and accesses data stored in the memory 820 to perform various computer functions and process data. Optionally, the processor 880 may include one or more processing units. Preferably, the processor 880 integrates an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 880.

[0175] In the embodiment of the present application, the processor 880 included in the terminal device also has the following functions:

[0176] Copying the rendering data stored in the back buffer of the main rendering thread to the bridge buffer through the main rendering thread; the bridge buffer is bound to a shared texture identifier;

[0177] Obtaining the rendering data from the bridge buffer based on the shared texture identifier through the secondary rendering thread; drawing the rendering data onto an input rendering surface provided by the encoding thread to obtain an output rendering surface, and transmitting the output rendering surface to the encoding thread;

[0178] The output rendering surface is encoded by the encoding thread to obtain recorded video frame data.

[0179] Optionally, the processor 880 is further configured to execute steps of any implementation of the screen recording method provided in the embodiments of the present application.

[0180] Referring to Figure 9, Figure 9 is a schematic diagram of the structure of a server 900 provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 922 (for example, one or more processors) and a memory 932, and one or more storage media 930 (for example, one or more mass storage devices) for storing application programs 942 or data 944. Among them, the memory 932 and the storage medium 930 can be temporary storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 can be configured to communicate with the storage medium 930 to execute a series of instruction operations in the storage medium 930 on the server 900.

[0181] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems, such as Windows Server 2003. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM etc.

[0182] The steps executed by the server in the above embodiment may be based on the server structure shown in FIG9 .

[0183] The CPU 922 is configured to execute the following steps:

[0184] Copying the rendering data stored in the back buffer of the main rendering thread to the bridge buffer through the main rendering thread; the bridge buffer is bound to a shared texture identifier;

[0185] Obtaining the rendering data from the bridge buffer based on the shared texture identifier through the secondary rendering thread; drawing the rendering data onto an input rendering surface provided by the encoding thread to obtain an output rendering surface, and transmitting the output rendering surface to the encoding thread;

[0186] The output rendering surface is encoded by the encoding thread to obtain recorded video frame data.

[0187] Optionally, the CPU 922 can also be used to execute the steps of any implementation method of the screen recording method provided in the embodiments of the present application.

[0188] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program, which is used to execute any implementation of a screen recording method described in the aforementioned embodiments.

[0189] The present application also provides a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform any one of the screen recording methods described in the aforementioned embodiments.

[0190] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0192] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0193] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0194] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0195] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0196] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A screen recording method, the method being executed by a computer device through a recording application process corresponding to an application, the recording application process comprising a main rendering thread, a secondary rendering thread and an encoding thread, the secondary rendering thread and the encoding thread being started in response to a screen recording operation triggered by the application; the method comprising: Copying the rendering data stored in the back buffer of the main rendering thread to the bridge buffer through the main rendering thread; The bridge buffer is bound with a shared texture identifier; Obtaining the rendering data from the bridge buffer based on the shared texture identifier through the secondary rendering thread; Drawing the rendering data onto an input rendering surface provided by the encoding thread to obtain an output rendering surface, and transmitting the output rendering surface to the encoding thread; The output rendering surface is encoded by the encoding thread to obtain recorded video frame data.

2. The method according to claim 1, further comprising: Creating, by the secondary rendering thread, a secondary graphics program context corresponding to the secondary rendering thread according to the primary graphics program context corresponding to the primary rendering thread, and determining the primary graphics program context as a parent graphics program context of the secondary graphics program context; The shared texture identifier created in the main rendering thread corresponding to the parent graphics program context supports use by the secondary rendering thread corresponding to the secondary graphics program context.

3. The method according to claim 1 or 2, further comprising: If it is detected through the main rendering thread that the shared texture data is not initialized, the main rendering thread creates the shared texture identifier and the bridge buffer, and binds the shared texture identifier and the bridge buffer; Executing the process of copying the rendering data stored in the back buffer of the main rendering thread to the bridge buffer through the main rendering thread; If it is detected through the main rendering thread that the shared texture data has been initialized, the main rendering thread is executed to copy the rendering data stored in the back buffer in the main rendering thread to the bridge buffer.

4. The method according to any one of claims 1 to 3, wherein the step of copying the rendering data stored in the back buffer of the main rendering thread to the bridge buffer through the main rendering thread comprises: By means of the main rendering thread, setting the bridge buffer as a current write buffer in the main rendering thread, and setting the back buffer as a current read buffer in the main rendering thread; The main rendering thread executes a buffer copy function to copy the rendering data from the current read buffer to the current write buffer.

5. The method according to claim 4, wherein the executing a buffer copy function by the main rendering thread to copy the rendering data from the current read buffer to the current write buffer comprises: The buffer copy function is executed by the main rendering thread to copy the color data stored in the color buffer in the current read buffer to the current write buffer.

6. The method according to claim 4 or 5, further comprising: In response to a screen recording stop operation triggered by the application, performing a restoration process on a read frame buffer target and a write frame buffer target in the main rendering thread through the main rendering thread, wherein the restoration process is used to restore the read frame buffer target and the write frame buffer target to values ​​before responding to the screen recording operation; The read frame buffer target is used to indicate the current read buffer, and the write frame buffer target is used to indicate the current write buffer.

7. The method according to any one of claims 1 to 6, wherein drawing the rendering data onto an input rendering surface provided by the encoding thread to obtain an output rendering surface comprises: Building a shader through the secondary rendering thread; Drawing the rendering result based on the mapping relationship between the vertex coordinates of the input rendering surface and the texture coordinates of the rendering data by the shader; The rendering result is output to the input rendering surface to obtain the output rendering surface.

8. The method according to any one of claims 1 to 7, further comprising: Creating a media encoder through the encoding thread; Creating the input rendering surface through the media encoder, and transmitting the input rendering surface to the secondary rendering thread; The encoding process of the output rendering surface by the encoding thread to obtain the recorded video frame data includes: The output rendering surface is encoded by the media encoder to obtain the video frame data.

9. The method according to any one of claims 1 to 8, further comprising: Get the recorded video size set by the application; The recorded video size is used to indicate the size of the video frame obtained through screen recording; The input rendering surface corresponding to the size of the recorded video is created through the encoding thread, and the input rendering surface is provided to the secondary rendering thread, so as to obtain the video frame data corresponding to the size of the recorded video based on the input rendering surface recording.

10. The method according to claim 1, further comprising: Get the frame rate of the recorded video set by the application; The recorded video frame rate is used to indicate the frame rate of the video obtained by screen recording; Drawing the rendering data onto an input rendering surface provided by the encoding thread to obtain an output rendering surface includes: The rendering data is drawn onto the input rendering surface through the secondary rendering thread according to a drawing cycle corresponding to the recorded video frame rate to obtain the output rendering surface.

11. A screen recording device, the device being deployed on a computer device, the device supporting the running of a recording application process corresponding to an application, the recording application process comprising a main rendering thread, a secondary rendering thread and an encoding thread, the secondary rendering thread and the encoding thread being started in response to a screen recording operation triggered by the application; The device comprises: A data copy module, used for copying the rendering data stored in the back buffer of the main rendering thread to the bridge buffer through the main rendering thread; the bridge buffer is bound with a shared texture identifier; A rendering module, configured to obtain the rendering data from the bridge buffer through the secondary rendering thread based on the shared texture identifier; draw the rendering data onto an input rendering surface provided by the encoding thread to obtain an output rendering surface, and transmit the output rendering surface to the encoding thread; The encoding module is used to encode the output rendering surface through the encoding thread to obtain the recorded video frame data.

12. A computer device, the device comprising a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the screen recording method according to any one of claims 1 to 10 according to the computer program.

13. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program, when executed by an electronic device, implements the screen recording method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the screen recording method according to any one of claims 1 to 10 is implemented.

Images