Screen recording method, device, electronic equipment and storage medium

Abstract

The application provides a screen recording method and device, electronic equipment and storage medium. The screen recording method comprises the following steps: obtaining a screen recording instruction for a picture of a terminal target application; in response to the screen recording instruction, capturing a target IOSurface object created by a system in a rendering process of the picture of the target application; reading rendering data of the picture of the target application from the target IOSurface object to complete a recording operation. The embodiment of the application is beneficial to improving the screen recording efficiency of the picture of the target application on an IOS terminal.

Description

Technical Field

[0001] This application relates to the field of document protection technology, and in particular to a screen recording method, apparatus, electronic device and storage medium. Background Technology

[0002] Screen recording refers to the capture of multimedia data such as screen images and audio from a terminal to obtain a video file that can be saved and shared. With the development of screen recording technology, most current technologies utilize corresponding screen recording applications for terminal screen recording. However, screen recording applications are not suitable for all situations. For example, iOS uses ReplayKit to record the screen, but it records the entire terminal screen, which can easily record information involving user privacy into the video file; therefore, its use is not widespread. Furthermore, the open graphics library interface glReadPixels is the most widely used technology for screen recording on iOS. However, due to the special underlying architecture of open graphics library interfaces, each frame is executed asynchronously after being submitted by the processor, while glReadPixels needs to synchronously read screen data, resulting in relatively low efficiency for screen recording on iOS. Summary of the Invention

[0003] To address the above issues, this application proposes a screen recording method, apparatus, electronic device, and storage medium, which helps improve the screen recording efficiency of target applications on iOS devices.

[0004] A first aspect of this application provides a screen recording method, the method comprising:

[0005] Obtain screen recording commands for the target application on the terminal;

[0006] In response to the screen recording command, the screen of the target application is captured by the target IOSurface object created by the system during the rendering process;

[0007] The recording operation is completed by reading the rendering data of the target application's screen from the target IOSurface object.

[0008] In conjunction with the first aspect, in one possible implementation, capturing the screen of the target application during the rendering process using a target IOSurface object created by the system includes:

[0009] Hook the InitWithProperties method of the target IOSurface object;

[0010] When the target IOSurface object is initialized, the target IOSurface object is captured according to preset conditions.

[0011] In conjunction with the first aspect, in one possible implementation, capturing the target IOSurface object according to preset conditions includes:

[0012] Set the condition variable shouldCapture to true, and execute the block code in the system parameters;

[0013] If the execution of the block code causes the system to allocate a new IOSurface object, then the newly allocated IOSurface object is identified as the target IOSurface object and captured.

[0014] In conjunction with the first aspect, in one possible implementation, reading the rendering data of the target application's screen from the target IOSurface object includes:

[0015] When the current frame of the target application ends, the target attribute value of the target IOSurface object is detected;

[0016] The rendering data of the current frame is read from the target IOSurface object based on the target attribute value.

[0017] In conjunction with the first aspect, in one possible implementation, reading the rendering data of the current frame from the target IOSurface object based on the target attribute value includes:

[0018] If the target attribute value is updated, it is determined that the rendering data in the target IOSurface object has also been updated. When recording and reading data for the next frame, the updated rendering data is read from the target IOSurface object, wherein the updated rendering data is the rendering data of the current frame.

[0019] In conjunction with the first aspect, in one possible implementation, the method further includes:

[0020] While detecting the target attribute value, the liveness status of the target IOSurface object is also detected. If the target IOSurface object has been destroyed, it is removed from the management list.

[0021] When the system recreates the target IOSurface object, the operation of capturing the recreated target IOSurface object is performed again to read the rendering data of the current frame from the recreated target IOSurface object;

[0022] Add the recreated target IOSurface object to the management list.

[0023] In conjunction with the first aspect, in one possible implementation, the target IOSurface object holds a weak reference mechanism; the detection of the liveness of the target IOSurface object includes:

[0024] Check if the reference pointer of the target IOSurface object is null;

[0025] If yes, then the target IOSurface object has been destroyed; otherwise, the target IOSurface object has not been destroyed.

[0026] A second aspect of this application provides a screen recording apparatus, the apparatus comprising:

[0027] The instruction acquisition module is used to acquire screen recording instructions for the target application on the terminal.

[0028] The target capture module is used to respond to the screen recording command and capture the target IOSurface object created by the system during the rendering process of the target application's screen.

[0029] The data reading module is used to read the rendering data of the target application's screen from the target IOSurface object in order to complete the recording operation.

[0030] A third aspect of this application provides an electronic device including an input device and an output device, and further including a processor adapted to implement one or more instructions; and a computer storage medium storing one or more instructions adapted to be loaded by the processor and executed as follows:

[0031] Obtain screen recording commands for the target application on the terminal;

[0032] In response to the screen recording command, the screen of the target application is captured by the target IOSurface object created by the system during the rendering process;

[0033] The recording operation is completed by reading the rendering data of the target application's screen from the target IOSurface object.

[0034] A fourth aspect of this application provides a computer storage medium storing one or more instructions adapted for loading and execution by a processor of the following steps:

[0035] Obtain screen recording commands for the target application on the terminal;

[0036] In response to the screen recording command, the screen of the target application is captured by the target IOSurface object created by the system during the rendering process;

[0037] The recording operation is completed by reading the rendering data of the target application's screen from the target IOSurface object.

[0038] The above-described solution of this application includes at least the following beneficial effects: Compared with the prior art, the embodiments of this application obtain a screen recording instruction for the screen of the target application on the terminal; in response to the screen recording instruction, capture the target IOSurface object created by the system during the rendering process of the target application's screen; and read the rendering data of the target application's screen from the target IOSurface object to complete the recording operation. Since the rendering data generated by all applications in the iOS system must be transmitted using shared memory, when a user needs to screen record the screen of the target application, based on the underlying characteristics of the iOS system, the target IOSurface object storing the rendering data is directly captured, and then the rendering data of each frame of the target application is read from the target IOSurface object, thereby improving the screen recording efficiency of the target application's screen on the iOS device. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 An example diagram illustrating an application scenario provided in this application embodiment;

[0041] Figure 2 A flowchart illustrating a screen recording method provided in an embodiment of this application;

[0042] Figure 3 An example diagram illustrating a trigger signal for screen recording provided in an embodiment of this application;

[0043] Figure 4 An example diagram illustrating another trigger signal for screen recording provided in an embodiment of this application;

[0044] Figure 5 A flowchart illustrating another screen recording method provided in an embodiment of this application;

[0045] Figure 6 An example diagram of screen recording for a WeChat mini-game provided in this application embodiment;

[0046] Figure 7 This is a schematic diagram of the structure of a screen recording device provided in an embodiment of this application;

[0047] Figure 8 This is a schematic diagram of another screen recording device provided in an embodiment of this application;

[0048] Figure 9 This is a schematic diagram of another screen recording device provided in an embodiment of this application;

[0049] Figure 10 This is a schematic diagram of another screen recording device provided in an embodiment of this application;

[0050] Figure 11 This is a schematic diagram of another screen recording device provided in an embodiment of this application;

[0051] Figure 12 This is a schematic diagram of another screen recording device provided in an embodiment of this application;

[0052] Figure 13 This is a schematic diagram of another screen recording device provided in an embodiment of this application;

[0053] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0055] The terms "comprising" and "having," and any variations thereof, appearing in this specification, claims, and drawings, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Furthermore, the terms "first," "second," and "third," etc., are used to distinguish different objects and are not used to describe a specific order.

[0056] This application proposes a screen recording scheme to improve the screen recording efficiency of target applications on iOS devices. This screen recording scheme can be applied to terminals, primarily referring to iOS-based terminals, including but not limited to mobile phones, tablets, wearable devices, e-readers, etc. The terminal can implement the screen recording scheme through corresponding applications, such as WeChat mini-games or other WeChat mini-programs. The following describes the application of this screen recording scheme... Figure 1 Taking the scenario shown as an example, Figure 1 The scenario depicted involves a terminal and a server, which communicate via a network. This network can be a wired network, a wireless network, a virtual private network (VPN), a local area network (LAN), a metropolitan area network (MAN), and so on. The terminal runs a related application, which can also be understood as a client, such as... Figure 1 As shown, when the terminal receives a screen recording command for the screen to be recorded (i.e., the screen of the relevant application), it obtains the shared memory occupied by the rendering data of the screen to be recorded in the system through a certain capture mechanism, and then reads the rendering data of the screen from the shared memory to realize the recording of the screen. The user can share the recorded video file to social friends or other terminals or third-party platforms through the server. The server can be a single server, a server cluster composed of several servers, or a cloud server, which is not limited in this application.

[0057] In some embodiments of this application, the terminal can automatically record the screen of the running application when the playback function of the application is enabled, or record the screen during the playback process of the application, or record the screen when a relevant prompt box (e.g., highlight clip, perfect operation) pops up during the operation of the application. In some embodiments of this application, when the terminal switches from the foreground to the background of the application, the shared memory occupied by the application during the screen rendering process may be destroyed. However, when the terminal switches the application back to the foreground, the system will recreate the shared memory for the rendering data of the application. At this time, the operation of capturing and reading the rendering data will be performed again to record the screen of the application.

[0058] As can be seen from the above application scenarios, compared with the screen recording solution using ReplayKit, this application only records the screen of the part that needs to be recorded, rather than recording the entire terminal screen, which is more secure and better protects user privacy. Furthermore, compared with the most widely used solution that uses the open graphics library interface glReadPixels for screen recording, this application directly reads rendering data from the shared memory allocated by the system for the relevant application for screen recording, without waiting for the current frame to finish rendering, resulting in higher recording efficiency.

[0059] Based on the above description, the screen recording method provided in the embodiments of this application will be described in detail below with reference to other accompanying drawings. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a screen recording method provided in an embodiment of this application, as shown below. Figure 2 As shown, steps S21-S23 are included:

[0060] S21, Obtain the screen recording command for the target application on the terminal.

[0061] In specific embodiments of this application, the target application refers to an application running on the terminal and in the foreground, which can be a video player, WeChat, WeChat mini-games, WeChat mini-programs, etc. Correspondingly, the screen of the target application can be a video playback interface, a chat interface, a game interface, etc., optionally, such as... Figure 3As shown, when the target application is running in the foreground, the terminal interface displays a control to enable screen recording. This control can be presented as a button or a slider, and its position can be fixed or floating. Taking a button as an example, when the user triggers the screen recording button by tapping or long-pressing, the server, after detecting the trigger signal of the control, sends a screen recording command to the terminal. The terminal receives the screen recording command and performs subsequent screen recording operations. Of course, the screen recording command can also be generated by the terminal after detecting the trigger signal of the control. In some implementations, the control can also be a button in a pop-up prompt box on the target application interface, such as... Figure 4 As shown, when the target application interface pops up a "Record a wonderful moment" prompt box, the user can trigger the relevant button below to start screen recording. This application does not impose any restrictions on the time or form in which the user triggers the screen recording signal.

[0062] S22, in response to the screen recording command, capture the target IOSurface object created by the system during the rendering process of the target application's screen.

[0063] In this specific embodiment, the target IOSurface object refers to the IOSurface object to be captured. After receiving the screen recording instruction, the terminal begins to record the screen of the target application. Specifically, for terminals using the iOS system, in order to display the rendered data generated by its application on the screen, the rendered data needs to be transmitted to the system window compositor. However, rendering data often occupies a large amount of memory space. Therefore, the system uses shared memory to transmit this rendering data. All memory related to rendering is created and managed by the public interface IOSurface class exposed by the system. However, the system creates IOSurface objects in various situations such as image resources, videos, and CoreGraphics (an advanced graphics engine). Among the many IOSurface objects, it is only necessary to capture the target IOSurface object created by the system when a specific behavior occurs. For example, in the process of rendering using the open graphics library in WeChat mini games, the target IOSurface object is created by the system when behaviors such as RenderBuffer allocation, calling the rendering interface, and submitting rendering occur.

[0064] To accurately capture these target IOSurface objects, this embodiment uses the Method Swizzle hook provided by the Objective-C language to hook the InitWithProperties method of the target IOSurface object. This allows the target IOSurface object to be captured based on preset conditions during its initialization. Specifically, the core code first sets the condition variable shouldCapture to true, then executes the block code in the system parameters, and then sets the condition variable shouldCapture to false. If the execution of the block code causes the system to allocate a new IOSurface object, the newly allocated IOSurface object is marked as the target IOSurface object for capture.

[0065] S23, read the rendering data of the target application's screen from the target IOSurface object to complete the recording operation.

[0066] In a specific embodiment of this application, for the target IOSurface object captured in step S22, taking the current frame of the target application as an example, when the current frame of the target application ends on the terminal screen, the target attribute value (i.e., seed value) of the target IOSurface object is detected, and the rendering data of the current frame is read from the target IOSurface object according to the status of the target attribute value. Specifically, if the target attribute value has been updated, it means that the rendering data in the target IOSurface object has also been updated. In order to ensure that the current frame has been drawn, the rendering data is read from the target IOSurface object with updated target attribute value when recording and reading data in the next frame of the current frame. The rendering data read at this time is the rendering data of the current frame of the target application.

[0067] In addition to detecting the target attribute values, the liveness of the target IOSurface object is also checked. Since this application uses Objective-C's weak reference mechanism for all IOSurface objects, if the reference pointer of the target IOSurface object is set to null, it indicates that the target IOSurface object has been destroyed by the system. For example, when the user switches the target application to the background, the system will destroy the target application's IOSurface object to save memory. If the target IOSurface object is determined to be destroyed, it is removed from the IOSurface management list. When the user switches the target application back to the foreground, during the rendering of the target application's current frame, the system will recreate the target IOSurface object for the target application. At this time, the terminal will again perform the operation of capturing the recreated target IOSurface object, detecting the target attribute values ​​of the recreated target IOSurface object, and reading the rendering data of the current frame from the recreated target IOSurface object to complete the recording of the current frame. It is understandable that this solution uses the aforementioned method to read the rendering data for each frame of the target application.

[0068] As can be seen, this embodiment obtains a screen recording command for the target application's screen on the terminal; responds to the screen recording command, captures the target IOSurface object created by the system during the rendering process of the target application's screen; and reads the rendering data of the target application's screen from the target IOSurface object to complete the recording operation. Since all application rendering data in the iOS system must be transmitted using shared memory, when a user needs to record the screen of a target application, based on the underlying characteristics of the iOS system, the target IOSurface object storing the rendering data is directly captured, and then the rendering data of each frame of the target application is read from the target IOSurface object, thereby improving the screen recording efficiency of the target application's screen on the iOS device.

[0069] based on Figures 2-4 For a description of the illustrated embodiment, please refer to [link to description]. Figure 5 , Figure 5 A flowchart illustrating another screen recording method provided in this application embodiment is shown below. Figure 5 As shown, steps S51-S54 are included:

[0070] S51, obtain the screen recording command for the target application on the terminal;

[0071] S52, in response to the screen recording command, Hook the InitWithProperties method of the target IOSurface object, the target IOSurface object is created by the system during the rendering process of the screen of the target application;

[0072] S53, when the target IOSurface object is initialized, the target IOSurface object is captured according to preset conditions;

[0073] S54, read the rendering data of the target application's screen from the target IOSurface object to complete the recording operation.

[0074] In one possible implementation, capturing the target IOSurface object according to preset conditions includes:

[0075] Set the condition variable shouldCapture to true, and execute the block code in the system parameters;

[0076] If the execution of the block code causes the system to allocate a new IOSurface object, then the newly allocated IOSurface object is identified as the target IOSurface object and captured.

[0077] In this implementation, capturing the IOSurface object created by the system according to preset conditions helps ensure that only the target IOSurface object needed for screen recording is captured.

[0078] In one possible implementation, reading the rendering data of the target application's screen from the target IOSurface object includes:

[0079] When the current frame of the target application ends, the target attribute value of the target IOSurface object is detected;

[0080] The rendering data of the current frame is read from the target IOSurface object based on the target attribute value.

[0081] In this implementation, the content of the target IOSurface object is determined based on the target attribute value of the target IOSurface object. If it has changed, the rendering data can be read. This effectively solves the problem of difficulty in reading rendering data caused by the FrameBuffer creating multiple IOSurface objects for rotation.

[0082] In one possible implementation, reading the rendering data of the current frame from the target IOSurface object based on the target attribute value includes:

[0083] If the target attribute value is updated, it is determined that the rendering data in the target IOSurface object has also been updated. When recording and reading data for the next frame, the updated rendering data is read from the target IOSurface object, wherein the updated rendering data is the rendering data of the current frame.

[0084] In this implementation, delaying the reading of rendering data for each frame of the target application by one frame ensures that the content of each frame has been drawn completely, without causing waiting.

[0085] In one possible implementation, the method further includes:

[0086] While detecting the target attribute value, the liveness status of the target IOSurface object is also detected. If the target IOSurface object has been destroyed, it is removed from the management list.

[0087] When the system recreates the target IOSurface object, the operation of capturing the recreated target IOSurface object is performed again to read the rendering data of the current frame from the recreated target IOSurface object;

[0088] Add the recreated target IOSurface object to the management list.

[0089] In this implementation, when the system destroys the target IOSurface object, the target IOSurface object is deleted from the IOSurface object management list. The system only captures the target IOSurface object again to read the rendering data when it reconstructs the target IOSurface object, which helps to ensure the system's storage space.

[0090] In one possible implementation, the target IOSurface object holds a weak reference mechanism; detecting the liveness of the target IOSurface object includes:

[0091] Check if the reference pointer of the target IOSurface object is null;

[0092] If yes, then the target IOSurface object has been destroyed; otherwise, the target IOSurface object has not been destroyed.

[0093] In this implementation, a weak reference mechanism is used for all IOSurface objects in the system, which can accurately determine the liveness of the target IOSurface object based on its reference pointer.

[0094] Specifically, the above steps S51-S54 are in Figures 2-4 The examples shown already contain relevant descriptions, and will not be repeated here to avoid duplication.

[0095] As can be seen, this embodiment of the application obtains a screen recording command for the screen of the target application on the terminal; responds to the screen recording command, captures the target IOSurface object created by the system during the rendering process of the target application's screen; and reads the rendering data of the target application's screen from the target IOSurface object to complete the recording operation. Since the rendering data generated by all applications in the iOS system must be transmitted using shared memory, when a user needs to screen record the screen of the target application, based on the underlying characteristics of the iOS system, the target IOSurface object storing the rendering data is directly captured, and then the rendering data of each frame of the target application is read from the target IOSurface object, thereby improving the screen recording efficiency of the target application's screen on the iOS device.

[0096] With the rapid development of the WeChat Mini Games platform and its integrated social attributes, to better understand the screen recording solution proposed in this application, the following explanation uses WeChat Mini Games on iOS as an example. Figure 6As shown, when a user runs a WeChat mini-game on their terminal, a screen recording signal is triggered at the start of the mini-game, during a highlight moment, or at the end of the mini-game. For example, triggering the "Game Replay" control. After detecting the screen recording signal, the terminal captures the target IOSurface object created by the system during the rendering process of the mini-game according to a preset capture mechanism. The terminal detects the seed value of the captured target IOSurface object and checks its survival status. If the target IOSurface object has not been destroyed and its seed value has been updated, the rendering data of the mini-game is read from the target IOSurface object. If the target IOSurface object has been destroyed, the terminal waits for the system to reconstruct the target IOSurface object, performs the capture operation again, and then reads the rendering data of the mini-game from the reconstructed target IOSurface object to complete the recording of the mini-game screen. The resulting screen recording video of the mini-game can then be shared with the corresponding friend in WeChat contacts via the server. The terminal uses a method of directly capturing the target IOSurface object and reading the rendering data in the target IOSurface object to record the screen of WeChat mini games. This method is not only highly efficient, but also has almost no impact on the performance of the WeChat mini game environment, which helps to ensure the basic functions of the WeChat mini game ecosystem.

[0097] Based on the description of the above screen recording method embodiments, this application also provides a screen recording device, which may be a computer program (including program code) running on a terminal. This screen recording device can execute... Figure 2 or Figure 5 The method shown. Please refer to [link / reference]. Figure 7 The device includes:

[0098] The instruction acquisition module 71 is used to acquire screen recording instructions for the target application on the terminal.

[0099] The target capture module 72 is used to respond to the screen recording command and capture the target IOSurface object created by the system during the rendering process of the target application's screen.

[0100] The data reading module 73 is used to read the rendering data of the target application's screen from the target IOSurface object in order to complete the recording operation.

[0101] In one implementation, such as Figure 8 As shown, the target acquisition module 72 includes:

[0102] The first capture unit 7201 is used to hook the InitWithProperties method of the target IOSurface object;

[0103] The second capture unit 7202 captures the target IOSurface object according to preset conditions when the target IOSurface object is initialized.

[0104] In one implementation, such as Figure 9 As shown, the second capture unit 7202 includes:

[0105] Code execution unit 72021 is used to set the condition variable shouldCapture to true and execute the block code in the system parameters;

[0106] The capture execution unit 72022 is used to set the condition variable shouldCapture to false. If the execution of the block code causes the system to allocate a new IOSurface object, the newly allocated IOSurface object is identified as the target IOSurface object and captured.

[0107] In one implementation, such as Figure 10 As shown, the data reading module 73 includes:

[0108] The attribute value detection unit 7301 is used to detect the target attribute value of the target IOSurface object when the current frame of the target application ends.

[0109] The data reading unit 7302 is used to read the rendering data of the current frame from the target IOSurface object according to the target attribute value.

[0110] In one implementation, such as Figure 11 As shown, the data reading unit 7302 includes:

[0111] The first data reading unit 73021 is used to determine that the rendering data in the target IOSurface object has also been updated if the target attribute value has been updated, and to read the updated rendering data from the target IOSurface object when recording and reading data for the next frame, wherein the updated rendering data is the rendering data of the current frame.

[0112] In one implementation, such as Figure 12 As shown, the device further includes:

[0113] The survival detection module 74 is used to detect the survival status of the target IOSurface object while detecting the target attribute value. If the target IOSurface object has been destroyed, the target IOSurface object is deleted from the management list.

[0114] The object reconstruction module 75 is used to re-capture the recreated target IOSurface object when the system recreates the target IOSurface object, so as to read the rendering data of the current frame from the recreated target IOSurface object.

[0115] The object management module 76 is used to add the recreated target IOSurface object to the management list.

[0116] In one implementation, such as Figure 13 As shown, the survival detection module 74 includes:

[0117] Pointer detection unit 7401 is used to detect whether the reference pointer of the target IOSurface object is null;

[0118] The survival determination unit 7402 is used to determine if the target IOSurface object has been destroyed, and otherwise to determine if the target IOSurface object has not been destroyed.

[0119] As can be seen, the screen recording device provided in this application obtains a screen recording instruction for the screen of a target application on the terminal; responds to the screen recording instruction by capturing a target IOSurface object created by the system during the rendering process of the target application's screen; and reads the rendering data of the target application's screen from the target IOSurface object to complete the recording operation. Since the rendering data generated by all applications in the iOS system must be transmitted using shared memory, when a user needs to record the screen of a target application, based on the underlying characteristics of the iOS system, the target IOSurface object storing the rendering data is directly captured, and then the rendering data of each frame of the target application is read from the target IOSurface object, thereby improving the screen recording efficiency of the target application's screen on the iOS device.

[0120] According to one embodiment of this application, Figure 7The various modules of the screen recording device shown can be individually or entirely combined into one or more other units, or some of the modules can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of the present invention. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, the screen recording device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0121] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 2 or Figure 5 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 7 The screen recording apparatus shown herein, and the screen recording method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0122] Based on the description of the above method and apparatus embodiments, please refer to... Figure 14 , Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 14 As shown, the electronic device includes at least a processor 1401, an input device 1402, an output device 1403, and a computer storage medium 1404. The processor 1401, input device 1402, output device 1403, and computer storage medium 1404 within the electronic device can be connected via a bus or other means.

[0123] The computer storage medium 1404 can be stored in the memory of the electronic device. The computer storage medium 1404 is used to store computer programs, which include program instructions. The processor 1401 is used to execute the program instructions stored in the computer storage medium 1404. The processor 1401 (or CPU (Central Processing Unit)) is the computing and control core of the electronic device, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions.

[0124] In one embodiment, the processor 1401 of the electronic device provided in this application embodiment can be used to perform a series of screen recording processes: acquiring a screen recording instruction for the screen of a target application on the terminal; in response to the screen recording instruction, capturing a target IOSurface object created by the system during the rendering process of the screen of the target application; and reading the rendering data of the screen of the target application from the target IOSurface object to complete the recording operation.

[0125] This application embodiment obtains a screen recording command for the screen of a target application on a terminal; responds to the screen recording command by capturing a target IOSurface object created by the system during the rendering process of the target application's screen; and reads the rendering data of the target application's screen from the target IOSurface object to complete the recording operation. Since all application rendering data in the iOS system must be transmitted using shared memory, when a user needs to record the screen of a target application, based on the underlying characteristics of the iOS system, the target IOSurface object storing the rendering data is directly captured, and then the rendering data of each frame of the target application is read from the target IOSurface object, thereby improving the screen recording efficiency of the target application's screen on the iOS device.

[0126] In one implementation, the processor 1401 performs the capture of the target IOSurface object created by the system during the rendering process of the screen of the target application, including:

[0127] Hook the InitWithProperties method of the target IOSurface object;

[0128] When the target IOSurface object is initialized, the target IOSurface object is captured according to preset conditions.

[0129] In one implementation, the processor 1401 performs the capture of the target IOSurface object according to preset conditions, including:

[0130] Set the condition variable shouldCapture to true, and execute the block code in the system parameters;

[0131] If the execution of the block code causes the system to allocate a new IOSurface object, then the newly allocated IOSurface object is identified as the target IOSurface object and captured.

[0132] In one implementation, the processor 1401 executes the rendering data of the screen of the target application read from the target IOSurface object, including:

[0133] When the current frame of the target application ends, the target attribute value of the target IOSurface object is detected;

[0134] The rendering data of the current frame is read from the target IOSurface object based on the target attribute value.

[0135] In one implementation, the processor 1401 executes the step of reading the rendering data of the current frame from the target IOSurface object based on the target attribute value, including:

[0136] If the target attribute value is updated, it is determined that the rendering data in the target IOSurface object has also been updated. When recording and reading data for the next frame, the updated rendering data is read from the target IOSurface object, wherein the updated rendering data is the rendering data of the current frame.

[0137] In one embodiment, the processor 1401 is further configured to: detect the liveness of the target IOSurface object while detecting the target attribute value; if the target IOSurface object has been destroyed, then delete the target IOSurface object from the management list.

[0138] When the system recreates the target IOSurface object, the operation of capturing the recreated target IOSurface object is performed again to read the rendering data of the current frame from the recreated target IOSurface object;

[0139] Add the recreated target IOSurface object to the management list.

[0140] In one implementation, the target IOSurface object holds a weak reference mechanism; the processor 1401 performs the detection of the liveness of the target IOSurface object, including:

[0141] Check if the reference pointer of the target IOSurface object is null;

[0142] If yes, then the target IOSurface object has been destroyed; otherwise, the target IOSurface object has not been destroyed.

[0143] For example, the aforementioned electronic device may be a mobile phone, tablet computer, PDA, personal digital assistant, or similar device. The electronic device may include, but is not limited to, a processor 1401, an input device 1402, an output device 1403, and a computer storage medium 1404. Those skilled in the art will understand that the schematic diagram is merely an example of an electronic device and does not constitute a limitation on the electronic device; it may include more or fewer components than illustrated, or combine certain components, or use different components.

[0144] It should be noted that since the processor 1401 of the electronic device executes the computer program to implement the steps in the screen recording method described above, all embodiments of the screen recording method described above are applicable to the electronic device and can achieve the same or similar beneficial effects.

[0145] This application embodiment also provides a computer storage medium (Memory), which is a memory device in an electronic device used to store programs and data. It is understood that the computer storage medium here can include the built-in storage medium in a terminal, or it can include an extended storage medium supported by the terminal. The computer storage medium provides storage space, which stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1401. These instructions can be one or more computer programs (including program code). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor 1401. In one embodiment, the processor 1401 can load and execute one or more instructions stored in the computer storage medium to implement the corresponding steps of the screen recording method described above; specifically, the one or more instructions in the computer storage medium are loaded and executed by the processor 1401 in the following steps:

[0146] Obtain screen recording commands for the target application on the terminal;

[0147] In response to the screen recording command, the screen of the target application is captured by the target IOSurface object created by the system during the rendering process;

[0148] The recording operation is completed by reading the rendering data of the target application's screen from the target IOSurface object.

[0149] This application embodiment obtains a screen recording command for the screen of a target application on a terminal; responds to the screen recording command by capturing a target IOSurface object created by the system during the rendering process of the target application's screen; and reads the rendering data of the target application's screen from the target IOSurface object to complete the recording operation. Since all application rendering data in the iOS system must be transmitted using shared memory, when a user needs to record the screen of a target application, based on the underlying characteristics of the iOS system, the target IOSurface object storing the rendering data is directly captured, and then the rendering data of each frame of the target application is read from the target IOSurface object, thereby improving the screen recording efficiency of the target application's screen on the iOS device.

[0150] In one example, when one or more instructions in the computer storage medium are loaded by the processor 1401, the following steps are also performed:

[0151] Hook the InitWithProperties method of the target IOSurface object;

[0152] When the target IOSurface object is initialized, the target IOSurface object is captured according to preset conditions.

[0153] In one example, when one or more instructions in the computer storage medium are loaded by the processor 1401, the following steps are also performed:

[0154] Set the condition variable shouldCapture to true, and execute the block code in the system parameters;

[0155] If the execution of the block code causes the system to allocate a new IOSurface object, then the newly allocated IOSurface object is identified as the target IOSurface object and captured.

[0156] In one example, when one or more instructions in the computer storage medium are loaded by the processor 1401, the following steps are also performed:

[0157] When the current frame of the target application ends, the target attribute value of the target IOSurface object is detected;

[0158] The rendering data of the current frame is read from the target IOSurface object based on the target attribute value.

[0159] In one example, when one or more instructions in the computer storage medium are loaded by the processor 1401, the following steps are also performed:

[0160] If the target attribute value is updated, it is determined that the rendering data in the target IOSurface object has also been updated. When recording and reading data for the next frame, the updated rendering data is read from the target IOSurface object, wherein the updated rendering data is the rendering data of the current frame.

[0161] In one example, when one or more instructions in the computer storage medium are loaded by the processor 1401, the following steps are also performed:

[0162] While detecting the target attribute value, the liveness status of the target IOSurface object is also detected. If the target IOSurface object has been destroyed, it is removed from the management list.

[0163] When the system recreates the target IOSurface object, the operation of capturing the recreated target IOSurface object is performed again to read the rendering data of the current frame from the recreated target IOSurface object;

[0164] Add the recreated target IOSurface object to the management list.

[0165] In one example, when one or more instructions in the computer storage medium are loaded by the processor 1401, the following steps are also performed:

[0166] Check if the reference pointer of the target IOSurface object is null;

[0167] If yes, then the target IOSurface object has been destroyed; otherwise, the target IOSurface object has not been destroyed.

[0168] For example, a computer program on a computer-readable storage medium includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0169] It should be noted that since the computer program on the computer storage medium is executed by the processor to implement the steps in the screen recording method described above, all embodiments of the screen recording method described above are applicable to the computer storage medium and can achieve the same or similar beneficial effects.

[0170] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A screen recording method, characterized in that, The method includes: Obtain screen recording commands for the target application on the terminal; In response to the screen recording command, the InitWithProperties method of the target IOSurface object is hooked. The target IOSurface object is an IOSurface object created by the system during the rendering process of the target application's screen, which is triggered by a target event. The target event includes RenderBuffer allocation, calling the rendering interface, and submitting rendering. When the target IOSurface object is initialized, the condition variable shouldCapture is set to true, and the block code in the system parameters is executed; If the execution of the block code causes the system to allocate a new IOSurface object, then the newly allocated IOSurface object is identified as the target IOSurface object and captured. The recording operation is completed by reading the rendering data of the target application's screen from the target IOSurface object. The step of reading the rendering data of the target application's screen from the target IOSurface object includes: When the current frame of the target application ends, the target attribute value of the target IOSurface object is detected. The target attribute value is updated as the rendering data in the target IOSurface object is updated. The update status of the target attribute value is used to indicate whether the rendering data in the target IOSurface object has been updated. The rendering data of the current frame is read from the target IOSurface object whose target attribute value has been updated.

2. The method according to claim 1, characterized in that, The step of reading the rendering data of the current frame from the target IOSurface object whose target attribute value has been updated includes: If the target attribute value is updated, it is determined that the rendering data in the target IOSurface object has also been updated. When recording and reading data for the next frame, the updated rendering data is read from the target IOSurface object, wherein the updated rendering data is the rendering data of the current frame.

3. The method according to claim 1, characterized in that, The method further includes: While detecting the target attribute value, the liveness status of the target IOSurface object is also detected. If the target IOSurface object has been destroyed, it is removed from the management list. When the system recreates the target IOSurface object, the operation of capturing the recreated target IOSurface object is performed again to read the rendering data of the current frame from the recreated target IOSurface object; Add the recreated target IOSurface object to the management list.

4. The method according to claim 3, characterized in that, The target IOSurface object holds a weak reference mechanism; detecting the liveness of the target IOSurface object includes: Check if the reference pointer of the target IOSurface object is null; If yes, then the target IOSurface object has been destroyed; otherwise, the target IOSurface object has not been destroyed.

5. A screen recording device, characterized in that, The device includes: The instruction acquisition module is used to acquire screen recording instructions for the target application on the terminal. The target capture module is used to respond to the screen recording command and capture at least one target IOSurface object created by the system during the rendering process of the target application's screen. The target acquisition module includes a first acquisition unit and a second acquisition unit; The first capture unit is used to hook the InitWithProperties method of the target IOSurface object. The target IOSurface object is an IOSurface object created by the system during the rendering process of the target application's screen, which is triggered by a target event. The target event includes RenderBuffer allocation, calling the rendering interface, and submitting rendering. The second capturing unit is used to capture the target IOSurface object according to preset conditions when the target IOSurface object is initialized; The second capture unit includes a code execution unit and a capture execution unit; The code execution unit is used to set the condition variable shouldCapture to true and execute the block code in the system parameters; The capture execution unit is used to set the condition variable shouldCapture to false. If the execution of the block code causes the system to allocate a new IOSurface object, the newly allocated IOSurface object is identified as the target IOSurface object and captured. The data reading module is used to read the rendering data of the target application's screen from the target IOSurface object in order to complete the recording operation; The data reading module includes: An attribute value detection unit is used to detect the target attribute value of the target IOSurface object at the end of the current frame of the target application. The target attribute value is updated as the rendering data in the target IOSurface object is updated. The update status of the target attribute value is used to indicate whether the rendering data in the target IOSurface object has been updated. The data reading unit is used to read the rendering data of the current frame from the target IOSurface object whose target attribute value has been updated.

6. An electronic device comprising an input device and an output device, characterized in that, Also includes: A processor, adapted to implement one or more instructions; as well as, A computer storage medium storing one or more instructions adapted to be loaded by the processor and executed as described in any one of claims 1-4.

7. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which are adapted to be loaded by a processor and executed as described in any one of claims 1-4.

Images