Screen recording method and apparatus

Abstract

The present disclosure relates to a screen recording method and device, applied to the field of multimedia technology, the method comprising: in response to a received screen recording instruction, obtaining an i-th frame of to-be-combined screen data of an OSD layer; obtaining target whiteboard data of a VIDEO layer corresponding to the i-th frame of to-be-combined screen data, the target acquisition time of the target whiteboard data being closest to the acquisition time of the i-th frame of to-be-combined screen data; and combining the i-th frame of to-be-combined screen data and the target whiteboard data to obtain an i-th frame of combined data, so as to record video data comprising OSD layer data and VIDEO layer data; wherein i is 0, 1, 2, 3, …, and the problem that the recorded video only has the result of writing and lacks the writing process when recording the video of whiteboard writing can be solved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to multimedia technology. More specifically, the present application relates to a screen recording method and device. BACKGROUND

[0002] With the popularity of Android smart display devices (such as smart TVs), whiteboard and other smart software are increasingly used in education, conferences, home entertainment and other scenarios. Users can write, draw and save and share the results on the whiteboard.

[0003] When a user needs to record a video, for the content displayed on the On Screen Display (OSD) layer, the original Android screen capture interface (which can only record the image displayed on the OSD layer) can be used to achieve screen recording.

[0004] However, the writing process is displayed on the VIDEO layer and the writing result is displayed on the OSD layer. Therefore, the video recorded using the original Android screen capture interface only has the writing result but not the writing process. In addition, since the VIDEO layer display area cannot be recorded, the recorded video has a black screen in this area, which seriously affects the user experience. SUMMARY

[0005] To solve the above technical problems or at least partially solve the above technical problems, the present application provides a screen recording method and device, which can solve the problem that the recorded video only has the writing result but not the writing process when recording a video including whiteboard writing.

[0006] In a first aspect, the embodiments of the present application provide a screen recording method, which comprises: in response to a received screen recording instruction, obtaining the i-th frame of to-be-combined screen data of the OSD layer; obtaining target whiteboard data of the VIDEO layer corresponding to the i-th frame of to-be-combined screen data, the target acquisition time of the target whiteboard data being closest to the acquisition time of the i-th frame of to-be-combined screen data; and combining the i-th frame of to-be-combined screen data and the target whiteboard data to obtain the i-th frame of combined data, so as to record video data including OSD layer data and VIDEO layer data; wherein i is 0, 1, 2, 3, ….

[0007] In some embodiments of the present application, before obtaining the i-th frame of screen data to be synthesized of the OSD layer, the method further comprises: creating a custom GLSurfaceView and a custom OpenGL renderer corresponding to the custom GLSurfaceView, the custom OpenGL renderer comprising an external screen recording texture pipeline and a texture superimposition pipeline; and synthesizing the i-th frame of screen data to be synthesized and the target whiteboard data to obtain the i-th frame of synthesized data, comprising: processing the i-th frame of screen data to be synthesized through the external screen recording texture pipeline to obtain the i-th frame of texture; and processing the i-th frame of texture and the target whiteboard data through the texture superimposition pipeline to obtain the i-th frame of synthesized data.

[0008] In some embodiments of the present application, obtaining each frame of screen data to be synthesized of the OSD layer comprises: listening to a data available event of a data buffer area of the OSD layer; and obtaining data of the OSD layer from the data buffer area after listening to a preset number of data available events.

[0009] In some embodiments of the present application, after recording the video data comprising the OSD layer data and the VIDEO layer data, the method further comprises: displaying the i-th frame of synthesized data to play the video data; or encoding the i-th frame of synthesized data to generate encoded video data.

[0010] In some embodiments of the present application, before encoding the i-th frame of synthesized data, the method further comprises: determining whether a first condition is met, the first condition being that a difference between a current time and a target collection time is less than or equal to a time threshold, and the current time being in a video recording state; and encoding the i-th frame of synthesized data, comprising: encoding the i-th frame of synthesized data when the first condition is met; and the method further comprises: discarding the i-th frame of synthesized data when the first condition is not met.

[0011] In some embodiments of the present application, the i-th frame of encoded synthesis data includes a timestamp, which is used to indicate the difference between the time of encoding the i-th frame of synthesis data and the target time; in the case of satisfying the first condition, the encoding of the i-th frame of synthesis data includes: in the case of satisfying the first condition, detecting whether the current is in the data synthesis state; in the case of the current being in the data synthesis state, the i-th frame of synthesis data is encoded, and the sum of the first difference and the system time of receiving the screen recording instruction is determined as the target time, the first difference is 0, or the first difference is used to indicate the difference between the time of encoding the i-th frame of synthesis data and the time of switching from the data synthesis state to the data pause synthesis state last time; in the case of the current being in the data pause state, the i-th frame of synthesis data is encoded, and the data pause state is switched to the data synthesis state, and the sum of the second difference and the system time of receiving the screen recording instruction is determined as the target time, the second difference is used to indicate the difference between the time of encoding the i-th frame of synthesis data and the time of switching from the data synthesis state to the data pause synthesis state last time.

[0012] In some embodiments of the present application, in the case of not satisfying the first condition, the i-th frame of synthesis data is discarded, including: in the case of not satisfying the first condition, detecting whether the current is in the data synthesis state; in the case of the current being in the data synthesis state, setting the data synthesis state to the data pause synthesis state, recording the time of switching the data synthesis state to the data pause synthesis state, and discarding the i-th frame of synthesis data; in the case of the current being in the data pause synthesis state, the i-th frame of synthesis data is discarded.

[0013] In some embodiments of the present application, after the i-th frame of synthesis data is encoded to generate encoded video data, when the encoded video data is played at a target speed, the product of the timestamp and the reciprocal of the target speed is used as the time interval between playing the i-th frame of synthesis data and playing the next frame of synthesis data.

[0014] In a second aspect, the present application provides a screen recording device, which comprises an acquisition module and a synthesis module; the acquisition module is used to acquire the i-th frame of to-be-synthesized screen data of the OSD layer in response to the received screen recording instruction; the acquisition module is also used to acquire the target whiteboard data of the VIDEO layer corresponding to the i-th frame of to-be-synthesized screen data, and the target acquisition time of the target whiteboard data is closest to the acquisition time of the i-th frame of to-be-synthesized screen data; the synthesis module is used to synthesize the i-th frame of to-be-synthesized screen data and the target whiteboard data to obtain the i-th frame of synthesis data, so as to record the video data including the OSD layer data and the VIDEO layer data; wherein i is 0, 1, 2, 3, ….

[0015] In a third aspect, the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the screen recording method according to the first aspect.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program stored on the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the screen recording method according to the first aspect.

[0017] In a fifth aspect, the present application provides a computer program product, comprising: when the computer program product is executed on a computer, the computer program product causes the computer to implement the screen recording method according to the first aspect.

[0018] The technical scheme provided by the embodiments of the present application has the following advantages compared with the prior art: in the embodiments of the present application, in response to the received screen recording instruction, the i-th frame of screen data to be synthesized of the OSD layer is obtained: the target whiteboard data of the VIDEO layer corresponding to the i-th frame of screen data to be synthesized is obtained, the target acquisition time of the target whiteboard data is closest to the acquisition time of the i-th frame of screen data to be synthesized; the i-th frame of screen data to be synthesized and the target whiteboard data are synthesized to obtain the i-th frame of synthesized data, so as to record the video data comprising the OSD layer data and the VIDEO layer data; wherein i is 0, 1, 2, 3, …; one frame of screen data of the OSD layer and one frame of whiteboard data of the corresponding VIDEO layer are synthesized to obtain one frame of synthesized data, and based on the multiple frames of synthesized data, the recorded video data can be obtained, each frame of the video data comprises the OSD layer data and the VIDEO layer data, and the pain point that the Android native video recording can only record the OSD layer data is solved. In this way, when it is necessary to record the writing, drawing and the like of the whiteboard, the video recorded by the present scheme not only has the writing result, but also has the writing process, the entire operation process on the whiteboard is completely copied, and it is convenient for the user to watch subsequently. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the implementation manners in the related art, the drawings needed to be used in the embodiments or related art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0020] Figure 1 A schematic diagram of an operating scenario between a display device and a control device according to some embodiments is shown;

[0021] Figure 2 A hardware configuration block diagram of the control device 100 according to some embodiments is shown.

[0022] Figure 3 A hardware configuration block diagram of the display device 200 is shown according to some embodiments.

[0023] Figure 4 An Android native screen recording structure diagram is shown according to some embodiments.

[0024] Figure 5 A relationship diagram of functional modules of the screen recording is shown according to some embodiments.

[0025] Figure 6 A flow diagram of a screen recording method is shown according to some embodiments.

[0026] Figure 7 A flow diagram of a screen recording method is shown according to some embodiments.

[0027] Figure 8 A flow diagram of a screen recording method is shown according to some embodiments.

[0028] Figure 9 A flow diagram of a screen recording method is shown according to some embodiments.

[0029] Figure 10 A flow diagram of a screen recording method is shown according to some embodiments.

[0030] Figure 11 A flow diagram of a screen recording method is shown according to some embodiments.

[0031] Figure 12 A structure diagram of a screen recording device is shown according to some embodiments.

[0032] Figure 13 A hardware structure diagram of an electronic device is shown according to some embodiments. DETAILED DESCRIPTION

[0033] In order to make the purpose and implementation of the present application more clear, the present application will be described clearly and completely in conjunction with the drawings of the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application.

[0034] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0035] The terms "first", "second", "third", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar or like objects or entities, and do not necessarily imply a specific order or sequence unless otherwise noted. It should be understood that the terms used in this way can be interchanged as appropriate.

[0036] The terms "comprise" and "have" and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a product or device that comprises a list of components does not necessarily have to have all the components explicitly listed, but can include other components not expressly listed or inherent to such products or devices.

[0037] The screen recording method provided by the embodiments of the present application can be used for a display device, which can have various implementation forms, for example, can be a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, etc.

[0038] Figure 1 The operation scenarios between the display device and the control device according to the embodiments are shown in the schematic diagram, wherein the control device includes a smart device or a control apparatus. As shown in Figure 1 The user can operate the display device 200 through the smart device 300 or the control apparatus 100.

[0039] In some embodiments, the control apparatus 100 can be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-distance communication modes, to control the display device 200 through wireless or wired mode. The user can input user instructions through the keys on the remote controller, voice input, control panel input, etc., to control the display device 200.

[0040] In some embodiments, the smart device 300 (such as a mobile terminal, a tablet computer, a computer, a notebook computer, etc.) can also be used to control the display device 200. For example, an application running on the smart device is used to control the display device 200.

[0041] In some embodiments, the display device can not receive instructions using the above-mentioned smart device or control device, but can receive user control through touch or gesture, etc.

[0042] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0043] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0044] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, an external memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0045] like Figure 3 The display device 200 includes at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a user interface 280, an external memory, and a power supply.

[0046] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output.

[0047] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0048] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0049] The communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator can include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the external control apparatus 100 or the server 400 through the communicator 220.

[0050] The user interface 280 can be used to receive control signals of the control apparatus 100 (e.g., an infrared remote controller, etc.). It can also be used to directly receive input operation instructions of a user and convert the operation instructions into instructions that the display device 200 can recognize and respond to. In this case, it can be referred to as a user input interface.

[0051] The detector 230 is used to collect signals of an external environment or interaction with the outside. For example, the detector 230 includes a light receiver for collecting ambient light intensity, or an image collector such as a camera for collecting an external environment scene, user attributes, or user interaction gestures, or a sound collector such as a microphone for receiving external sound.

[0052] The external device interface 240 can include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It can also be a composite input / output interface formed by a plurality of the above interfaces.

[0053] The tuner and demodulator 210 receives broadcast television signals through wired or wireless reception and demodulates audio and video signals and EPG data signals from a plurality of wireless or wired broadcast television signals.

[0054] In some embodiments, the controller 250 and the tuner and demodulator 210 can be located in different split devices, i.e., the tuner and demodulator 210 can also be in an external device of the main body device where the controller 250 is located, such as an external set-top box, etc.

[0055] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored on the storage (internal storage or external storage). The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object displayed on the display 260, the controller 250 can perform an operation related to the object selected by the user command.

[0056] In some embodiments, the controller includes at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), and a Random Access Memory (RAM), a Read-Only Memory (ROM), a first interface to an n-th interface for input / output, a communication bus, and the like.

[0057] The RAM is also called main memory, which is an internal memory for directly exchanging data with the controller. It can be read and written at any time (except when refreshing), and the speed is very fast, and it is usually used as a temporary data storage medium for operating systems or other programs running. The biggest difference between it and ROM is the volatility of data, that is, once the power is off, the stored data will be lost. RAM is used in computers and digital systems to temporarily store programs, data, and intermediate results. ROM works in a non-destructive readout mode and can only read out information. Once the information is written, it is fixed and will not be lost even if the power is cut off, so it is also called fixed memory.

[0058] The user can input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0059] The "user interface" is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the conversion between the internal form of information and the form that the user can accept. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an icon, window, control, etc. interface element displayed in the display screen of the display device, wherein the control can include an icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, and other visible interface elements.

[0060] In order to facilitate the understanding of the scheme of the embodiments of the present application, the following explains the principle of screen recording of Android native and related terms:

[0061] The native screen recording provided by Android: Android provides a MediaProjection service for screen recording. The recording related modules are as follows: Figure 4MediaProjectionManager is provided by Android system, MediaProjection is used to intercept the image in the screen of display device, then project the intercepted image to VirtualDisplay, VirtualDisplay renders the image on Surface, Surface acts as the role of screen data buffer, Surface can be generated based on MediaRecorder for MediaRecorder to consume, to realize video recording, or generated based on MediaCodec for MediaCodec to consume, to re-encode the image in Surface to generate new video.

[0062] Specifically, MediaProjection and MediaProjectionManager are system screen recording interfaces provided by Android, and do not need root permission, but a screen recording permission application box will pop up when used, and the user can start screen recording after agreeing, MediaProjectionManager is a system-level service for managing MediaProjection. The following describes the screen recording process of consumers and producers. The screen data source (producer) generates data in the buffer, and the screen data consumer extracts data from the buffer for use. Different consumers can achieve different functions, such as screen recording saving and screen recording live streaming (screen sharing). VirtualDisplay is a virtual screen on Android, used to capture the content displayed on the screen, and is a screen data producer. Surface corresponds to a screen data buffer, and the screen data producer can produce data on Surface, and the consumer extracts data from Surface for use.

[0063] MediaCodec and MediaRecorder, MediaCodec is more inclined to native, and MediaRecorder is inclined to upper encapsulation. MediaCodec can access video stream data when processing video, and can process native video data, such as video overlay technology, adding subtitles, etc. MediaRecorder class is simpler than MediaCodec, because it has been encapsulated, and the video recording can be realized by calling the interface, such as setting the video encoding format, the video saving path, the video source, etc., and the recorded video can be obtained, but the video stream data cannot be accessed, and the native video data cannot be processed.

[0064] With the popularity of Android smart commercial televisions, whiteboard and other intelligent software are increasingly applied in education, conferences, home entertainment and other scenarios. Users can choose to write, draw, board and other operations, and can save and share the results. Since the writing, erasing and other operations of the whiteboard have high requirements for latency, the original screen refresh mechanism of Android is generally not used, and a customized refresh mechanism is used to directly call the Native framebuffer interface for local refresh drawing of real-time writing graphics, so that the writing latency can be optimized from 120ms to within 50ms. However, since the writing does not use OSD layer drawing, the screen capture interface MediaProjection of Android cannot record the writing process, so the user cannot review the writing process. To solve the above problems, the present application provides a scheme for real-time recording and synthesizing OSD layer and VIDEO layer screen data, avoiding the problem that the recorded video can only correctly display OSD layer content.

[0065] As Figure 5 shown is a relationship diagram of various functional modules of screen recording provided by an embodiment of the present application, including: a whiteboard application 501, a media projection management service 502, a media projection 503, a virtual screen 504, a Surface 505, a bitmap 506, a custom view writing 507, a Native layer 508, a custom GLSurfaceView 509, a custom OpenGL renderer 5091, an external screen recording texture pipeline 50911, a texture superimposition pipeline 50912, a cache area 50913, a screen, a custom EGL thread 510, a virtual screen GL pipeline 5101, and an encoder 5102.

[0066] Specifically, the whiteboard application 501 can be any application that needs to use whiteboard writing and drawing. After starting the whiteboard application 501, based on the received screen recording instruction (i.e., obtaining the recording permission), the whiteboard application 501 obtains the screen recording permission (starting the original screen recording of Android will pop up a prompt box for obtaining the recording permission); the functions of the media projection management service 502, the media projection 503 and the virtual screen 504 are described in Figure 4 , which will not be described here.

[0067] The Surface 505 is different from the Surface in Figure 4 , and is generated based on SurfaceTexture. Among them, SurfaceTexture can capture real-time screen recording data, but does not directly display it on the screen, but converts it into an external texture of OpenGL.

[0068] The whiteboard application 501 creates a bitmap 506 in the process of initializing the interface, and updates the data in the bitmap 506 in real time when receiving writing data, and the bitmap 506 is transmitted to the native layer 508 to realize whiteboard writing, and the bitmap 506 is also transmitted to the custom view writing 507 (i.e. writing data of the VIDEO layer).

[0069] The custom GLSurfaceView 509 obtains screen data of the OSD layer from the Surface 505 and writing data of the VIDEO layer 508 from the custom view writing 507, and specifically, the screen data of the OSD layer obtained from the Surface 505 is input into an external screen recording texture pipeline 50911 of a custom OpenGL renderer 5091 to obtain a texture, the texture and the writing data of the VIDEO layer obtained from the custom view writing 507 are input into a texture superimposition pipeline 50912 of the custom OpenGL renderer 5091 to obtain a superimposed texture, and the superimposed texture is stored in a buffer area 50913.

[0070] The custom GLSurfaceView 509, which inherits from the GLSurfaceView provided by Android, can perform OpenGL graphics rendering through a separate thread (GLThread), and provides an EGL (Embedded Grahpic Library) environment to render graphics onto the screen. The module provides a custom OpenGL renderer, which is drawn in the GLThread thread separated from the UI thread. The specific synthesis function is realized by a custom OpenGL programmable pipeline, including an external screen recording texture pipeline and a texture superimposition pipeline, both of which are output to the framebuffer instead of directly output to the screen. After synthesis is completed, on the one hand, the synthesized video can be output to the screen (in a transparent manner, which is not perceived by the user), and on the other hand, the synthesized video can be output to the encoder through a custom EGL. The OpenGL-related operations of the custom EGL environment are all run in a newly created child thread, so as to avoid affecting the original drawing speed.

[0071] The superimposed texture of the buffer area 50913 can be output to the screen through the EGL, or can be input to a virtual screen GL pipeline 5101 in a custom EGL thread 510 as input, and the output of the virtual screen GL pipeline 5101 is input to an encoder 5102 through the custom EGL for encoding processing.

[0072] GLSurfaceView is a special View that manages an OpenGL Surface, which can render the OpenGL Surface to the Android View, and encapsulates a lot of configurations needed to create the OpenGL environment, making it more convenient to use OpenGL. GLSurfaceView mainly includes the following capabilities: providing an OpenGL rendering thread to prevent rendering from blocking the main thread; providing continuous rendering or on-demand rendering capabilities; encapsulating EGL-related resources and creation and release, greatly simplifying the use of OpenGL and window system interfaces. Custom GLSurfaceView inherits from GLSurfaceView and can call any function provided by GLSurfaceView.

[0073] Open Graphics Library (OpenGL), also known as Open Graphics Library, is a cross-language and cross-platform application programming interface for rendering 2D and 3D vector graphics. OpenGL is an interface for operating a graphics processing unit (GPU). It sends relevant instructions to the GPU through a driver to control the running state of the graphics rendering pipeline state machine. However, when it comes to interacting with the local window system, an intermediate layer is needed, and the intermediate layer is preferably platform-independent. This intermediate control layer is EGL, which implements rendering graphics to the screen.

[0074] SurfaceTexture is used to capture video frames in a video stream, but the processing of the video frames is not directly displayed, but is converted into an external texture of OpenGL, so it can be used for secondary processing of video stream data (such as camera filters, desktop effects, etc.).

[0075] In the embodiments of the present application, a screen recording method is provided, and the screen recording method provided by the embodiments of the present application will be described exemplarily below in combination with Figure 6 Taking the application of the method to a display device as an example, the screen recording method provided by the embodiments of the present application is described exemplarily. The method can include the following steps 601 to 603.

[0076] 601, in response to the received screen recording instruction, obtaining the i-th frame of to-be-combined screen data of the OSD layer.

[0077] It can be understood that the display device starts the whiteboard application, receives the screen recording instruction, obtains the permission to record the screen, and calls the media projection service provided by the Android system to record the data of the OSD layer.

[0078] It can be understood that the i-th frame of to-be-combined screen data of the OSD layer is acquired, and specifically, the frequency of acquiring the screen data of the OSD layer can be the same as or less than the frequency of acquiring the screen data of the OSD layer, and is specifically determined according to actual needs, and embodiments of the present application are not limited.

[0079] It can be understood that the i-th frame of to-be-combined screen data is any frame of screen data of the OSD layer acquired at a certain frequency after starting screen recording, that is, any frame of image of the OSD layer acquired.

[0080] 602, acquire target whiteboard data of the VIDEO layer corresponding to the i-th frame of to-be-combined screen data.

[0081] The target acquisition time of the target whiteboard data is closest to the acquisition time of the i-th frame of to-be-combined screen data.

[0082] It can be understood that when there is a writing operation, the buffer area storing the whiteboard data will be updated, and the target acquisition time of the target whiteboard data is the update time of the bitmap. When the i-th frame of to-be-combined screen data is acquired, the latest frame of whiteboard data corresponding to the buffer area is acquired, that is, the target whiteboard data.

[0083] It can be understood that the target whiteboard data is a frame of image of the VIDEO layer acquired.

[0084] 603, combine the i-th frame of to-be-combined screen data and the target whiteboard data to obtain the i-th frame of combined data.

[0085] The i-th frame of combined data is obtained by combining the i-th frame of to-be-combined screen data and the target whiteboard data, to record video data including OSD layer data and VIDEO layer data, and i is 0, 1, 2, 3, ….

[0086] It can be understood that each frame of data after the combination processing is video stream data, and the recorded video data can be obtained based on multiple frames of combined data.

[0087] In the embodiment of the present application, in response to the received screen recording instruction, the i-th frame of screen data to be synthesized of the OSD layer is obtained: target whiteboard data of the VIDEO layer corresponding to the i-th frame of screen data to be synthesized is obtained, the target acquisition time of the target whiteboard data is closest to the acquisition time of the i-th frame of screen data to be synthesized; the i-th frame of screen data to be synthesized and the target whiteboard data are synthesized to obtain the i-th frame of synthesized data, so as to record the video data including the OSD layer data and the VIDEO layer data; wherein i is 0, 1, 2, 3…; one frame of screen data of the OSD layer and one frame of whiteboard data of the corresponding VIDEO layer are synthesized to obtain one frame of synthesized data, and based on the multiple frames of synthesized data, the recorded video data can be obtained, each frame of the video data includes OSD layer data and VIDEO layer data, and the pain point that the Android native video recording can only record the OSD layer data is solved. In this way, when it is necessary to record the writing, drawing and the like of the whiteboard, the video recorded by the present solution not only has the writing result, but also has the writing process, the entire operation process on the whiteboard is completely copied, and it is convenient for the user to watch subsequently.

[0088] The synthesis of the screen data of the OSD layer and the whiteboard data of the VIDEO layer in the embodiment of the present application can be implemented through a custom GLSurfaceView, and can also be implemented through other synthesis technologies, which is determined according to actual needs, and the embodiment of the present application is not limited. The following is described by taking the implementation of the data synthesis through the custom GLSurfaceView as an example.

[0089] In some embodiments of the present application, in combination with Figure 6 As shown in Figure 7 Before the above step 601, the screen recording return provided by the embodiment of the present application includes the following step 604, and the above step 603 can be implemented through the following steps 603a and 603b.

[0090] 604, create a custom GLSurfaceView and a custom OpenGL renderer corresponding to the custom GLSurfaceView, the custom OpenGL renderer including an external screen recording texture pipeline and a texture superposition pipeline.

[0091] It can be understood that the custom GLSurfaceView can be set to transparent display, and the transparency is to not affect the original display of the application, so as to perform background recording in a user-perception-free manner, and transparently display the synthesized video of the OSD layer and the VIDEO layer.

[0092] 603a, process the i-th frame of screen data to be synthesized through the external screen recording texture pipeline to obtain the i-th frame of texture.

[0093] 603b. Process the texture and target whiteboard data of the i-th frame through the texture overlay pipeline to obtain the composite data of the i-th frame.

[0094] As can be understood, a texture is a structured storage form of quantized color information that can be read and written by a shader. For example, an image might have parameters xyz, where x and y represent the coordinates of the corresponding pixel, and z represents the RGBA value (or other desired information) stored at that pixel. Besides containing necessary information, a texture also includes some texture sampling settings. The process of processing a texture is called the texture pipeline. For a detailed description of textures and the texture pipeline, please refer to existing technologies; details will not be elaborated here.

[0095] It is understandable that after the texture overlay pipeline processes the i-th frame texture and the target whiteboard data, the i-th frame composite data is obtained, which is also a frame texture.

[0096] In some embodiments of this application, before obtaining the i-th frame of screen data to be composited from the OSD layer, a custom GLSurfaceView and a corresponding custom OpenGL renderer are created. The custom OpenGL renderer includes an external screen recording texture pipeline and a texture overlay pipeline. The i-th frame of screen data to be composited is processed through the external screen recording texture pipeline to obtain the i-th frame texture. The i-th frame texture and the target whiteboard data are processed through the texture overlay pipeline to obtain the i-th frame composited data. In this way, the composited screen data of the OSD layer and the whiteboard data of the VIDEO layer are realized through the custom GLSurfaceView, thereby realizing the recording of the content of the OSD layer and the VIDEO layer, so that the recorded whiteboard writing video contains the writing process and the writing result.

[0097] In some embodiments of this application, combined with Figure 6 ,like Figure 8 As shown, step 601 can be implemented through steps 601a and 601b.

[0098] 601a. ​​Listen for data availability events in the OSD layer's data buffer.

[0099] 601b. After listening to the data availability event a preset number of times, retrieve the data from the OSD layer in the data cache.

[0100] The preset number of times can be an integer greater than or equal to 1. The specific preset number of times is determined according to actual needs, and this application embodiment does not limit it.

[0101] It can be understood that the screen recording adopts GLSurfaceView external texture input, and the refresh speed is very high (about 16 ms refresh once). On the other hand, the frequency of writing is controlled by the user, and the frequency is not fixed, which can be refreshed every 10 ms, or not refreshed for a long time. This leads to the input and output of the custom GLSurfaceView being too frequent, and the frame rate of the recorded synthesized video is not high, such as 25 fps. Therefore, when receiving the screen recording data available event (the refresh frame rate is 60 fps), the rendering frequency can be reduced, and the custom OpenGL renderer is requested to render on demand, that is, receiving multiple data available events, and obtaining screen data input into the custom GLSurfaceView only once. For example, the rendering frequency is reduced to half of the original frequency, that is, receiving two data available events, and performing rendering only once. Specifically, a screen recording rendering counter (that is, counting the reported data available events) is set, and when the counter is even, the data is obtained for subsequent rendering processing, and if the counter is odd, the data is discarded and not rendered.

[0102] It can be understood that the data available of the screen data buffer area is monitored, and when new data is stored, a data available event is reported. In this scheme, one data available event can be received, that is, one screen data is obtained, or multiple data available events can be received, and one screen data is obtained. Specifically, the data of the OSD layer is obtained from the data buffer area, which can be any frame of screen data stored in the buffer area (the buffer area is generally small, and at least one frame of continuous screen data is stored, and the difference between adjacent frames of screen data is small, and the human eye cannot distinguish), or one frame of screen data in a fixed position of the buffer area (such as the buffer area can store 3 frames of data, and each frame of data is marked with 0, 1 and 2 in the order of storage, and each time the frame of data marked with 1 is obtained), or the latest one frame of screen data. The actual situation is determined, and the embodiments of the application are not limited.

[0103] In some embodiments of the application, the data available event of the data buffer area of the OSD layer is monitored, and when a preset number of data available events are monitored, the data of the OSD layer is obtained from the data buffer area. In this way, the input frequency of the custom GLSurfaceView can be reduced according to the demand, and the amount of data that needs to be processed by the custom GLSurfaceView is reduced, so that the resources consumed are reduced, and the processing capacity of the custom GLSurfaceView is improved.

[0104] In some embodiments of the application, in combination with Figure 6 For example Figure 9As shown, after step 603 above, the screen recording method provided in this application embodiment further includes step 605 or step 606.

[0105] 605. Display the composite data of the i-th frame to play the video data.

[0106] 606. Encode the composite data of the i-th frame to generate encoded video data.

[0107] It is understandable that displaying the composite data of the i-th frame to play video data means that the composite data of the i-th frame can be displayed on the screen, which can be displayed transparently so that the user is unaware of it; displaying the composite data of the i-th frame also includes calling the MediaRecorder interface to directly generate video data (input composite data, output video data).

[0108] It's understandable; it could also involve encoding the composite data of the i-th frame, which could change the timestamp.

[0109] In some embodiments of this application, after combining the i-th frame of screen data to be synthesized with the target whiteboard data to obtain the i-th frame of synthesized data, and recording video data including OSD layer data and VIDEO layer data, the method further includes: displaying the i-th frame of synthesized data to play the video data; or, encoding the i-th frame of synthesized data to generate encoded video data. Thus, the processing of the synthesized data can be flexibly selected according to actual needs.

[0110] In some embodiments of this application, combined with Figure 9 ,like Figure 10 As shown, before step 606 above, the screen recording method provided in this application embodiment further includes steps 607 and 608 below. Step 606 above can be specifically implemented through step 606a below.

[0111] 607. Determine whether the first condition is met.

[0112] The first condition is that the difference between the current time and the target acquisition time is less than or equal to the duration threshold, and the current time is in video recording mode.

[0113] If yes, proceed to step 606a; otherwise, proceed to step 608.

[0114] 606a. Encode the composite data of the i-th frame.

[0115] 608. Discard the synthesized data of the i-th frame.

[0116] It is understandable that when a video recording command is received, the system is in video recording mode; when a video pause or end command is received, the system is in video pause recording mode.

[0117] It can be understood that the i-th frame of synthesized data is encoded in the case that the difference between the current time and the target acquisition time is less than or equal to the time threshold and the current video recording state is in the video recording state. The i-th frame of synthesized data is discarded in the case that the difference between the current time and the target acquisition time is greater than the time threshold or the current video recording state is in the video pause recording state.

[0118] It can be understood that the difference between the current time and the target acquisition time is greater than the time threshold, indicating that there is no writing operation for a long time, and the data corresponding to the whiteboard data buffer is not updated for a long time. Therefore, the synthesized data can be discarded, and the resource consumed by the encoding process is saved.

[0119] In some embodiments of the present application, it is judged whether the first condition is met, the first condition being that the difference between the current time and the target acquisition time is less than or equal to the time threshold and the current video recording state is in the video recording state. In the case that the first condition is met, the i-th frame of synthesized data is encoded. In the case that the first condition is not met, the i-th frame of synthesized data is discarded. In this way, in the case that the user has no writing operation for a long time, the synthesized data is directly discarded, and the resource consumed by the subsequent encoding process is saved.

[0120] In some embodiments of the present application, the first condition is met in combination with Figure 10 As shown in FIG. 6, the step 606a can be implemented by the following steps 606b to 606d, and the step 608 can be implemented by the following steps 608a to 608c. Figure 11 The i-th frame of synthesized data subjected to the encoding process comprises a time stamp, the time stamp being used to indicate the difference between the time of encoding the i-th frame of synthesized data and the target time.

[0121] 606b, detecting whether the current is in the data synthesis state.

[0122] If yes, the step 606c is performed, otherwise the step 606d is performed.

[0123] The step 606b is performed in the case that the first condition is met.

[0124] 606c, encoding the i-th frame of synthesized data, and determining the sum of the first difference and the system time when the screen recording instruction is received as the target time.

[0125] The first difference is 0, or the first difference is used to indicate the difference between the time of encoding the i-th frame of synthesized data and the time when the data synthesis state is last switched to the data pause synthesis state.

[0126] ​

[0127] It can be understood that, after receiving the screen recording instruction, if there is no data synthesis state switching operation, the first difference value is 0; if there is a data synthesis state switching operation, the first difference value is used to indicate the difference between the time of encoding processing of the i-th frame of synthesis data and the time of last switching from the data synthesis state to the data pause synthesis state.

[0128] It can be understood that, the time of encoding processing of the i-th frame of synthesis data can be the time of receiving the i-th frame of synthesis data by the encoder, can be the time of starting to process the i-th frame of synthesis data by the encoder, or can be the time of detecting that the current is in the data synthesis state, which is determined according to actual needs, and the embodiment of the present application is not limited.

[0129] 606d, encoding processing of the i-th frame of synthesis data, and switching the data pause state to the data synthesis state, determining the sum of the second difference value and the system time of receiving the screen recording instruction as the target time.

[0130] The second difference value is used to indicate the difference between the time of encoding processing of the i-th frame of synthesis data and the time of last switching from the data synthesis state to the data pause synthesis state.

[0131] It can be understood that, the i-th frame of synthesis data is encoded and processed, and the data pause state is switched to the data synthesis state, and the first time corresponding to the switching is the time of last switching from the data synthesis state to the data pause synthesis state, and the second difference value is used to indicate the difference between the time of encoding processing of the i-th frame of synthesis data and the first time.

[0132] 608a, detecting whether the current is in the data synthesis state.

[0133] If yes, step 608b is executed, otherwise step 608c is executed.

[0134] The step 608a is executed under the condition that the first condition is not met.

[0135] 608b, setting the data synthesis state to the data pause synthesis state, recording the time of switching the data synthesis state to the data pause synthesis state, and discarding the i-th frame of synthesis data.

[0136] 608c, discarding the i-th frame of synthesis data.

[0137] It can be understood that, the recorded time of switching the data synthesis state to the data pause synthesis state is used to calculate the first difference value and the second difference value.

[0138] It can be understood that in the case that the user has no writing operation for a long time or is in a pause recording state, the recorded video will be stuck in one frame, which seriously affects the viewing experience of the subsequent user. Therefore, in the process of encoding the recorded video, the timestamp can be changed to avoid the occurrence of frame sticking phenomenon.

[0139] Exemplarily, the time length threshold is 30 ms, the target time is 08:00:00:00, the target collection time is 08:00:00:00, the first frame of synthesis data is obtained, the current time is 08:00:00:10, the difference between the current time and the target collection time is 10 ms, which is less than 30 ms, the first frame of synthesis data is encoded and processed, and the added timestamp is 10. The target collection time is 08:00:00:15, the second frame of synthesis data is obtained, the current time is 08:00:00:30, the difference between the current time and the target collection time is 15 ms, which is less than 30 ms, the second frame of synthesis data is encoded and processed, and the added timestamp is 30. The target collection time is 08:00:00:15, the third frame of synthesis data is obtained, the current time is 08:00:00:50, the difference between the current time and the target collection time is 40 ms, which is greater than 30 ms, then the third frame of data is discarded, the data pause state is switched to the data synthesis state, and the current time of switching 08:00:00:50 is recorded. The target collection time is 08:00:00:160, the fourth frame of synthesis data is obtained, the current time is 08:00:00:170, the difference between the current time and the target collection time is 10 ms, which is less than 30 ms, the fourth frame of synthesis data is encoded and processed, the second difference is the difference between 08:00:00:170 and 08:00:00:50, the result is 120 ms, the target time is the sum of 08:00:00:00 and 120 ms, the result is 08:00:00:120, and the timestamp of the fourth frame of synthesis data after encoding processing is 50. Then the recorded video after encoding processing: the first frame timestamp is 10, the second frame timestamp is 30, and the third frame timestamp is 50, that is, the first frame plays for 30-10=20 ms, the second frame plays for 50-30=20 ms, and so on. If not processed, the first frame timestamp is 10, the second frame timestamp is 30, and the third frame timestamp is 170 (the difference between 08:00:00:170 and 08:00:00:00), that is, the first frame plays for 30-10=20 ms, the second frame plays for 170-30=140 ms, and there is a 140 ms pause at the second frame, which appears to the user as a frame sticking phenomenon. This is only an example for illustration, and in actual use, the time length of stopping at the previous frame will be longer when the user has no writing operation.

[0140] In some embodiments of the present application, the i-th frame of encoded composition data includes a timestamp, which is used to indicate the difference between the time of encoding the i-th frame of composition data and the target time; in the case of meeting the first condition, it is detected whether the current is in the data composition state; in the case of the current being in the data composition state, the i-th frame of composition data is encoded, and the sum of the first difference and the system time of receiving the screen recording instruction is determined as the target time, the first difference is 0, or the first difference is used to indicate the difference between the time of encoding the i-th frame of composition data and the time of switching from the data composition state to the data pause composition state last time; in the case of the current being in the data pause state, the i-th frame of composition data is encoded, and the data pause state is switched to the data composition state, and the sum of the second difference and the system time of receiving the screen recording instruction is determined as the target time, the second difference is used to indicate the difference between the time of encoding the i-th frame of composition data and the time of switching from the data composition state to the data pause composition state last time. In the case of not meeting the first condition, it is detected whether the current is in the data composition state; in the case of the current being in the data composition state, the data composition state is set to the data pause composition state, the time of switching the data composition state to the data pause composition state is recorded, and the i-th frame of composition data is discarded; in the case of the current being in the data pause composition state, the i-th frame of composition data is discarded. In this way, in the case of no writing operation or pause recording of the user for a long time, by setting the encoded timestamp, the frame freezing phenomenon can be avoided when the user reviews the recorded video.

[0141] In some embodiments of the present application, after step 606, when playing the encoded video data at the target speed, the product of the timestamp and the reciprocal of the target speed is used as the time interval between playing the i-th frame of composition data and playing the next frame of composition data.

[0142] It can be understood that the target speed can be any speed selected by the user, such as 1.5, 2, 2.5, 3, etc.

[0143] It can be understood that the product of the timestamp and the reciprocal of the target speed is used as the time interval between playing the i-th frame of composition data and playing the next frame of composition data, that is, by changing the timestamp, such as changing the timestamp of a frame in the video to the product of the original timestamp and the reciprocal of the target speed.

[0144] Exemplarily, the target speed is 2 times, taking 3 frames in the recorded video as an example, the original timestamp of the 1st frame is 1 ms, the original timestamp of the 2nd frame is 2 ms, and the original timestamp of the 3rd frame is 4 ms, that is, the 1st frame is played for 2-1=1 ms, the 2nd frame is played for 4-2=2 ms, and it takes 3 ms to play the 1st frame and the 2nd frame; after being changed according to the speed of 2 times, the timestamp of the 1st frame is 1*0.5=0.5 ms, the timestamp of the 2nd frame is 2*0.5=1 ms, and the timestamp of the 3rd frame is 4*0.5=2 ms, that is, the 1st frame is played for 1-0.5=0.5 ms, the 2nd frame is played for 2-1=1 ms, and it takes 1.5 ms to play the 1st frame and the 2nd frame; compared with the time length of the original playing of the 1st frame and the 2nd frame, the time length is shortened by half, that is, the playing speed is improved by 2 times.

[0145] In the embodiments of the present application, when playing the encoded video data according to the target speed, the product of the timestamp and the reciprocal of the target speed is taken as the time interval between playing the i-th frame of synthesized data and playing the next frame of synthesized data. In this way, for the recorded video, in order to adapt to different needs of different users, the recorded video can be played at different speeds, further improving the user experience.

[0146] In the embodiments of the present application, as shown in Figure 12 A screen recording device is provided, which comprises an acquisition module 1201 and a synthesis module 1202; the acquisition module 1201 is configured to acquire the i-th frame of to-be-synthesized screen data of an OSD layer in response to a received screen recording instruction; the acquisition module 1201 is further configured to acquire target whiteboard data of a VIDEO layer corresponding to the i-th frame of to-be-synthesized screen data, and the target acquisition time of the target whiteboard data is closest to the acquisition time of the i-th frame of to-be-synthesized screen data; the synthesis module 1202 is configured to synthesize the i-th frame of to-be-synthesized screen data and the target whiteboard data to obtain the i-th frame of synthesized data, so as to record video data comprising OSD layer data and VIDEO layer data; wherein i is 0, 1, 2, 3, ….

[0147] In some embodiments of the present application, the device further comprises a creation module; the creation module is configured to create a custom GLSurfaceView and a custom OpenGL renderer corresponding to the custom GLSurfaceView before acquiring the i-th frame of to-be-synthesized screen data of the OSD layer, and the custom OpenGL renderer comprises an external screen recording texture pipeline and a texture superposition pipeline; and the synthesis module 1202 is specifically configured to process the i-th frame of to-be-synthesized screen data through the external screen recording texture pipeline to obtain the i-th frame of texture, and process the i-th frame of texture and the target whiteboard data through the texture superposition pipeline to obtain the i-th frame of synthesized data.

[0148] In some embodiments of the present application, the acquisition module 1201 is further configured to listen to a data available event of a data buffer of the OSD layer, and acquire the data of the OSD layer from the data buffer after listening to the data available event for a preset number of times.

[0149] In some embodiments of the present application, the device further includes a display module and an encoding module. The display module is configured to display the i-th frame of the synthesized data after synthesizing the i-th frame of the to-be-synthesized screen data and the target whiteboard data to obtain the i-th frame of the synthesized data, and recording the video data including the OSD layer data and the VIDEO layer data, or the encoding module is configured to encode the i-th frame of the synthesized data to generate the encoded video data.

[0150] In some embodiments of the present application, the device further includes a judgment module. The judgment module is configured to judge whether a first condition is met before the encoding module encodes the i-th frame of the synthesized data. The first condition is that a difference between a current time and a target time is less than or equal to a time threshold, and the current time is in a video recording state. The encoding module is specifically configured to encode the i-th frame of the synthesized data when the first condition is met, and discard the i-th frame of the synthesized data when the first condition is not met.

[0151] In some embodiments of the present application, the device further includes a detection module. The i-th frame of the synthesized data after the encoding processing includes a time stamp. The time stamp is used to indicate a difference between a time of encoding the i-th frame of the synthesized data and a target time. The detection module is configured to detect whether the current time is in a data synthesis state when the first condition is met. The encoding module is specifically configured to encode the i-th frame of the synthesized data when the current time is in the data synthesis state. A sum of the first difference and a system time of receiving the screen recording instruction is determined as the target time. The first difference is 0, or the first difference is used to indicate a difference between a time of encoding the i-th frame of the synthesized data and a time of switching from the data synthesis state to a data pause synthesis state last time. The encoding module is specifically configured to encode the i-th frame of the synthesized data when the current time is in the data pause state, switch the data pause state to the data synthesis state, and determine a sum of the second difference and the system time of receiving the screen recording instruction as the target time. The second difference is used to indicate a difference between a time of encoding the i-th frame of the synthesized data and a time of switching from the data synthesis state to the data pause synthesis state last time.

[0152] In some embodiments of the present application, the device further comprises a switching module; the detection module is further configured to detect whether the current is in a data synthesis state or not in the case that the first condition is not met; the switching module is configured to set the data synthesis state to a data pause synthesis state in the case that the current is in the data synthesis state, record a time when the data synthesis state is switched to the data pause synthesis state, and discard the i-th frame of synthesis data; and in the case that the current is in the data pause synthesis state, discard the i-th frame of synthesis data.

[0153] In some embodiments of the present application, after the i-th frame of synthesis data is encoded to generate the encoded video data, a product of the timestamp and a reciprocal of the target speed is used as a time interval between playing the i-th frame of synthesis data and playing a next frame of synthesis data when the encoded video data is played at the target speed.

[0154] In the embodiments of the present application, each module can implement the screen recording method provided in the method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0155] As shown in FIG. 13, Figure 13 The embodiments of the present application further provide an electronic device, which can include a processor 1301, a memory 1302, and a program or instruction stored in the memory 1302 and executable on the processor 1301. When executed by the processor 1301, the program or instruction can implement each process of the screen recording method provided in the method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0156] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. When executed by a processor, the computer program implements each process of the screen recording method and achieves the same technical effects. To avoid repetition, details are not described herein.

[0157] The computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0158] The present application provides a computer program product, which includes: when the computer program product is run on a computer, the computer program product causes the computer to implement the screen recording method.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0160] The foregoing description has been set forth in conjunction with a particular embodiment and implementations, for purposes of convenience and illustration. However, the foregoing discussion is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Various modifications and changes are possible in light of the teachings above. The selection and arrangement of various elements described or claimed in the foregoing embodiments are presented to provide the best description possible of the principles and the practical applications of the embodiments. It is intended that the described embodiments be considered as illustrative only and that the scope of the embodiments be determined not with reference to the foregoing description but with reference to the claims.

Claims

1. A screen recording method, characterized by, The method comprises: In the case of starting the whiteboard application, in response to the received screen recording instruction, the i-th frame of screen data to be synthesized of the screen display OSD layer is acquired: Target whiteboard data of a video VIDEO layer corresponding to the i-th frame of screen data to be synthesized is acquired, a target acquisition time of the target whiteboard data is closest to an acquisition time of the i-th frame of screen data to be synthesized; The i-th frame of screen data to be synthesized and the target whiteboard data are synthesized to obtain the i-th frame of synthesized data, so as to record video data comprising OSD layer data and VIDEO layer data; It is judged whether a first condition is met, the first condition being that a difference between a current time and the target acquisition time is less than or equal to a time threshold, and a current video recording state is met; In the case where the first condition is met, it is detected whether a current data synthesis state is met; In the case where the current data synthesis state is met, the i-th frame of synthesized data is encoded to generate encoded video data, a sum of a first difference and a system time when the screen recording instruction is received is determined as a target time, the first difference being 0, or the first difference being used to indicate a difference between a time when the i-th frame of synthesized data is encoded and a time when a data synthesis state is last switched to a data pause synthesis state; In the case where the current data pause synthesis state is met, the i-th frame of synthesized data is encoded to generate encoded video data, and the data pause synthesis state is switched to the data synthesis state, a second difference is used to indicate a difference between a time when the i-th frame of synthesized data is encoded and a time when a data synthesis state is last switched to a data pause synthesis state, and a sum of the second difference and a system time when the screen recording instruction is received is determined as a target time; Wherein, the i-th frame of synthesized data after encoding processing comprises a time stamp, the time stamp being used to indicate a difference between a time when the i-th frame of synthesized data is encoded and the target time; i is 0, 1, 2, 3, ….

2. The method of claim 1, wherein, Before the i-th frame of screen data to be synthesized of the screen display OSD layer is acquired, the method further comprises: A custom GLSurfaceView and a custom OpenGL renderer corresponding to the custom GLSurfaceView are created, the custom OpenGL renderer comprising an external screen recording texture pipeline and a texture superposition pipeline; The i-th frame of screen data to be synthesized and the target whiteboard data are synthesized to obtain the i-th frame of synthesized data, comprising: The i-th frame of screen data to be synthesized is processed by the external screen recording texture pipeline to obtain the i-th frame of texture; The i-th frame of texture and the target whiteboard data are processed by the texture superposition pipeline to obtain the i-th frame of synthesized data.

3. The method of claim 1, wherein, The i-th frame of screen data to be synthesized of the screen display OSD layer is acquired, comprising: A data available event of a data buffer area of the OSD layer is listened to; After a preset number of data available events are listened to, data of the OSD layer is acquired from the data buffer area.

4. The method of claim 1, wherein, The method further comprises: displaying the i-th frame of the synthesized data to play the video data.

5. The method of claim 1, wherein, The method further comprises: discarding the i-th frame of the synthesized data when the first condition is not met.

6. The method of claim 5, wherein, The method further comprises: detecting whether the current is in a data synthesis state when the first condition is not met; setting the data synthesis state to a data pause synthesis state, recording a time when the data synthesis state is switched to the data pause synthesis state, and discarding the i-th frame of the synthesized data when the current is in the data synthesis state; discarding the i-th frame of the synthesized data when the current is in the data pause synthesis state.

7. The method of claim 1, wherein, After the i-th frame of the synthesized data is encoded to generate the encoded video data, the product of the timestamp and the reciprocal of the target speed is used as the time interval between playing the i-th frame of the synthesized data and playing the next frame of the synthesized data when the encoded video data is played at the target speed.

8. A screen recording apparatus, characterized by comprising: The device comprises an acquisition module, a synthesis module, a judgment module, and an encoding module. The acquisition module is configured to, in response to a received screen recording instruction, acquire i-th frame of screen data to be synthesized of an OSD layer when a whiteboard application is started. The acquisition module is further configured to acquire target whiteboard data of a VIDEO layer corresponding to the i-th frame of screen data to be synthesized, wherein a target acquisition time of the target whiteboard data is closest to an acquisition time of the i-th frame of screen data to be synthesized. The synthesis module is configured to synthesize the i-th frame of screen data to be synthesized and the target whiteboard data to obtain i-th frame of synthesized data to record video data comprising OSD layer data and VIDEO layer data. The judgment module is configured to judge whether a first condition is met, wherein the first condition is that a difference between a current time and the target acquisition time is less than or equal to a time threshold, and the current is in a video recording state. The encoding module is configured to detect whether the current is in a data synthesis state when the first condition is met. When the current is in the data synthesis state, the i-th frame of the synthesized data is encoded to generate the encoded video data, and a sum of a first difference and a system time when the screen recording instruction is received is determined as a target time, wherein the first difference is 0, or the first difference is used to indicate a difference between a time when the i-th frame of the synthesized data is encoded and a time when the data synthesis state is last switched to a data pause synthesis state. In a case that the current is in the data pause synthesis state, the i-th frame synthesis data is encoded to generate encoded video data, and the data pause synthesis state is switched to a data synthesis state. A second difference value and a system time when the screen recording instruction is received are added to determine a target time. The second difference value is used to indicate a difference between a time when the i-th frame synthesis data is encoded and a time when the data synthesis state is switched to the data pause synthesis state last time. The i-th frame synthesis data after encoding includes a time stamp, which is used to indicate a difference between a time when the i-th frame synthesis data is encoded and the target time. i is 0, 1, 2, 3, and so on.

9. The apparatus of claim 8, wherein, The device further includes a creating module. The creating module is configured to create a custom GLSurfaceView and a custom OpenGL renderer corresponding to the custom GLSurfaceView before the i-th frame screen data to be synthesized of the OSD layer of the screen display is acquired. The custom OpenGL renderer includes an external screen recording texture pipeline and a texture superposition pipeline. The synthesizing module is specifically configured to process the i-th frame screen data to be synthesized through the external screen recording texture pipeline to obtain an i-th frame texture, and process the i-th frame texture and the target whiteboard data through the texture superposition pipeline to obtain the i-th frame synthesis data.

Images