Screen recording method, electronic device and computer-readable storage medium

Abstract

An embodiment of the present application provides a screen recording method, which relates to the field of terminals. In response to a first operation of starting a screen recording function, the electronic device obtains identification information of an application associated with the first operation, wherein the application associated with the first operation includes an application running in the background of the electronic device. The electronic device obtains the media stream of the application based on the identification information, and generates a screen recording file corresponding to the application based on the media stream of the application. An embodiment of the present application also provides an electronic device and a computer-readable storage medium. The present application can realize screen recording of applications running in the background of an electronic device, thereby improving the user's screen recording experience.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a screen recording method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Screen recording is a common feature of electronic devices such as mobile phones and tablets. Existing screen recording methods record the current screen interface and save it as a screenshot or video file. However, existing screen recording methods cannot record application interfaces that are not displayed on the screen, which reduces the user experience. Summary of the Invention

[0003] In view of this, it is necessary to provide a screen recording method that can record the interface of an application running in the background or record the application window projected and displayed on other electronic devices on the local device side.

[0004] A first aspect of an embodiment of the present application discloses a screen recording method, which is applied to a first electronic device. The screen recording method includes: in response to a first operation of starting a screen recording function, obtaining identification information of an application associated with the first operation, wherein the application associated with the first operation includes an application running in the background of the first electronic device; obtaining a media stream of the application based on the identification information; and generating a screen recording file corresponding to the application based on the media stream of the application.

[0005] By adopting this technical solution, by obtaining the identification information of the application to be recorded, and then generating a screen recording file based on the media stream obtained from the identification information, compared with the existing method that can only record the interface displayed on the front end, this application can realize the screen recording of the interface of one or more background applications of the electronic device.

[0006] In some embodiments, the identification information includes the process identity identification number (Identity Document, ID) of the application or the package name of the application.

[0007] In some embodiments, acquiring the media stream of the application according to the identification information includes: determining a screen data buffer corresponding to the application according to the identification information; and acquiring the media stream of the application from the screen data buffer.

[0008] By adopting this technical solution, the interface of one or more applications of the electronic device can be recorded by extracting the media stream from the screen data buffer instead of recording the interface currently displayed on the screen. The recorded application can run in the foreground or the background of the electronic device.

[0009] In some embodiments, generating a screen recording file corresponding to the application according to the media stream of the application includes: performing image synthesis on the media stream of the application to generate the screen recording file corresponding to the application.

[0010] With this technical solution, the electronic device calls SurfaceFlinger or Direct Digital Synthesizer (DDS) algorithm to perform image synthesis on the media stream of the application and generate a screen recording file.

[0011] In some embodiments, the application corresponds to a first screen data buffer and a second screen data buffer, and a screen recording file corresponding to the application is generated according to the media stream of the application, including: performing image synthesis on the first media stream obtained from the first screen data buffer to generate a first screen recording sub-file; performing image synthesis on the second media stream obtained from the second screen data buffer to generate a second screen recording sub-file; splicing the first screen recording sub-file and the second screen recording sub-file to generate a screen recording file corresponding to the application.

[0012] By adopting this technical solution, the display contents of two or more screens in a distributed display scenario can be recorded on the electronic device side. Recording multiple screen display contents by the same device can ensure the synchronization of the recording clock. At the same time, multiple screen recording sub-files can be spliced ​​on the electronic device side to generate a screen recording file, so that users can view the contents of multiple display interfaces in one screen recording file.

[0013] In some embodiments, the first screen recording sub-file and the second screen recording sub-file are spliced ​​together, including: time-aligning the first screen recording sub-file and the second screen recording sub-file according to their timestamps; and splicing the first screen recording sub-file and the second screen recording sub-file that have undergone time alignment.

[0014] By adopting this technical solution, recording multiple screen display contents by the same device can ensure the synchronization of the recording clock. At the same time, multiple screen recording sub-files can be spliced ​​together on the electronic device side based on the timestamps of the screen recording sub-files to obtain a screen recording file, so that users can view the contents of multiple display interfaces with time synchronization in one screen recording file.

[0015] In some embodiments, the application corresponds to a first screen data buffer and a second screen data buffer, and a screen recording file corresponding to the application is generated according to the media stream of the application, including: obtaining the first media stream from the first screen data buffer, and obtaining the second media stream from the second screen data buffer; performing image synthesis and splicing on the first media stream and the second media stream to generate a screen recording file corresponding to the application.

[0016] By adopting this technical solution, the display contents of two or more screens in a distributed display scenario can be recorded on the electronic device side. Recording multiple screen display contents by the same device can ensure the synchronization of the recording clock. At the same time, multiple media streams can be time-aligned on the electronic device side, and then the images can be synthesized and spliced ​​to generate a screen recording file, so that users can view the contents of multiple display interfaces in one screen recording file.

[0017] In some embodiments, performing image synthesis and splicing on the first media stream and the second media stream includes: performing time alignment processing on the first media stream and the second media stream according to the timestamps of the first media stream and the second media stream; performing image synthesis and splicing on the first media stream and the second media stream that have undergone time alignment processing.

[0018] By adopting this technical solution, recording multiple screen display contents by the same device can ensure the synchronization of the recording clock. At the same time, multiple media streams can be time-aligned based on the timestamps of the media streams on the electronic device side, and then the images can be synthesized and spliced ​​to obtain a screen recording file, so that users can view the contents of multiple display interfaces with time synchronization in one screen recording file.

[0019] In some embodiments, the screen recording file includes a first interface and a second interface, the first interface and the second interface are displayed in a left-right split screen, or the first interface and the second interface are displayed in an upper-lower split screen, or part or all of the first interface is displayed floating on the second interface.

[0020] By adopting this technical solution, the multiple interfaces in the screen recording file can be set to be displayed in top and bottom split screens, left and right split screens, or in the form of floating windows according to the actual needs of the user, thereby improving the user experience.

[0021] In some embodiments, the screen recording method further includes: in response to a screen recording instruction sent by the second electronic device, obtaining identification information of an application associated with the screen recording instruction.

[0022] With this technical solution, for screen projection scenarios, users can select screen recording on the source device or the destination device, and specify the application on the source device to record the screen.

[0023] In some embodiments, the application associated with the first operation also includes an application that is projected and displayed on the second electronic device.

[0024] By adopting this technical solution, it is possible to record the screen of an application projected and displayed on a second electronic device on the first electronic device side, and the projected application can run in the foreground or background of the first electronic device.

[0025] In the second aspect, an embodiment of the present application provides a screen recording method, using a second electronic device, and the first electronic device projects one or more application windows on the second electronic device for display. The screen recording method includes: receiving a media stream sent by the first electronic device; in response to a first operation of starting a screen recording function, obtaining identification information of the application window associated with the first operation; obtaining a media stream corresponding to the identification information of the application window associated with the first operation from the received media stream; and generating a screen recording file based on the obtained media stream.

[0026] By adopting this technical solution, by obtaining the identification information of the application window to be recorded, and then extracting the media stream corresponding to the identification information from the received media stream based on the identification information to generate a screen recording file, it is possible to record the application interface displayed on the screen of the second electronic device.

[0027] In some embodiments, the identification information includes a process ID of the application window and a device ID of the first electronic device.

[0028] By adopting this technical solution, the process ID of the application window and the device ID of the first electronic device can be used as the unique identifier of the application window for screen recording on the second electronic device side.

[0029] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the screen recording method as described in the first aspect or the second aspect.

[0030] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory so that the electronic device executes the screen recording method described in the first aspect or the second aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the screen recording method as described in the first aspect or the second aspect.

[0032] In a sixth aspect, a device is provided, which has the function of implementing the behavior of the first electronic device in the method provided in the first or second aspect. The function can be implemented through hardware or through hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0033] It can be understood that the computer-readable storage medium described in the third aspect, the electronic device described in the fourth aspect, the computer program product described in the fifth aspect, and the device described in the sixth aspect all correspond to the methods of the first aspect or the second aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic structural diagram of an electronic device provided in one embodiment of the present application;

[0035] Figure 2 A schematic diagram of the software structure of an electronic device provided in one embodiment of the present application;

[0036] Figure 3 A schematic diagram of an application scenario of the screen recording method provided in one embodiment of the present application;

[0037] Figure 4 A schematic diagram of an application scenario of a screen recording method provided in another embodiment of the present application;

[0038] Figure 5 A schematic diagram of an application scenario of a screen recording method provided in another embodiment of the present application;

[0039] Figure 6 A flowchart of a screen recording method provided in one embodiment of the present application;

[0040] Figure 7 A flowchart of a screen recording method provided in another embodiment of the present application;

[0041] Figure 8 A schematic structural diagram of a possible first electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0043] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0044] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given for reference.

[0045] Multiple electronic devices can be combined into a hyperterminal via a communication network. A hyperterminal can be defined as presenting multiple electronic devices as a unified whole, organically integrating them to enhance the user experience in a multi-device environment. Each electronic device is a component of a hyperterminal. Background applications on one electronic device can be used on other electronic devices within the hyperterminal, and operating information of other electronic devices within the hyperterminal can be viewed on one electronic device.

[0046] The electronic device may be at least one of a mobile phone, a foldable electronic device, a tablet computer, a personal computer (PC), a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device. The communication network may be a wired network or a wireless network. For example, the communication network may be a local area network (LAN) or a wide area network (WAN), such as the Internet. When the communication network is a local area network, illustratively, the communication network may be a short-range communication network such as a wifi hotspot network, a wifi P2P network, a Bluetooth network, a zigbee network, or a near field communication (NFC) network. When the communication network is a wide area network, illustratively, the communication network can be a third-generation wireless telephone technology (3G) network, a fourth-generation mobile communication technology (4G) network, a fifth-generation mobile communication technology (5G) network, a future evolved public land mobile network (PLMN) or the Internet, etc.

[0047] An electronic device can install one or more applications. An application can be referred to as an application, which is a software program that can realize one or more specific functions. For example, instant messaging applications, video applications, audio applications, image capture applications, cloud desktop applications, etc. Among them, instant messaging applications can include SMS applications, Photo Sharing Image capture applications, for example, can include camera applications (system camera or third-party camera applications). Video applications, for example, can include Etc. Audio applications, for example, may include The applications mentioned in the following embodiments may be system applications installed on the electronic device when it leaves the factory, or third-party applications downloaded from the Internet or obtained from other electronic devices by the user while using the electronic device.

[0048] Electronic equipment including but not limited to or other operating systems.

[0049] Figure 1 A schematic structural diagram of an electronic device 10 is shown.

[0050] The electronic device 10 may include a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 130, a wireless communication module 140, an audio module 150, a sensor module 160, a camera module 170, a display screen 180, etc.

[0051] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0053] The processor can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0054] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 may be a cache memory. This memory can store instructions or data that have been used or are frequently used by processor 110. When processor 110 needs to use the instruction or data, it can directly access it from this memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0055] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface. The processor 110 may be connected to modules such as an audio module, a wireless communication module, a display, and a camera through at least one of the above interfaces.

[0056] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely illustrative and does not constitute a structural limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0057] The wireless communication function of the electronic device 10 can be implemented through the antenna 1, the antenna 2, the mobile communication module 130, the wireless communication module 140, the modem processor and the baseband processor.

[0058] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 10 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0059] The mobile communication module 130 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 10. The mobile communication module 130 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 130 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 130 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 130 can be set in the same device as at least some of the modules of the processor 110.

[0060] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 180. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 130 or other functional modules.

[0061] The wireless communication module 140 can provide wireless communication solutions for the electronic device 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 140 can be one or more devices that integrate at least one communication processing module. The wireless communication module 140 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 140 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0062] In some embodiments, antenna 1 of electronic device 10 is coupled to mobile communication module 130, and antenna 2 is coupled to wireless communication module 140, so that electronic device 10 can communicate with a network and other electronic devices via wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0063] Electronic device 10 can implement display functions using a GPU, display screen 180, and an application processor. A GPU is a microprocessor for image processing that connects display screen 180 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0064] The sensor module includes a touch sensor, a pressure sensor, a fingerprint sensor, etc. The camera module 170 includes a camera. The display screen 180 is used to display images, videos, etc. The display screen 180 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 10 may include one or more display screens 180.

[0065] The electronic device 10 can realize the camera function through the camera module 170, ISP, video codec, GPU, display screen 180, application processor AP, neural network processor NPU, etc.

[0066] The digital signal processor is used to process digital signals and can also process other digital signals. For example, when the electronic device 10 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0067] Video codecs are used to compress or decompress digital video. Electronic device 10 may support one or more video codecs. This allows electronic device 10 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0068] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 10, such as image recognition, face recognition, speech recognition, and text comprehension.

[0069] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 10. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be saved on the external memory card or transferred from the electronic device to the external memory card.

[0070] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 10 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional methods or data processing of the electronic device 10 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0071] The audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 150 can also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be provided in the processor 110, or some functional modules of the audio module 150 can be provided in the processor 110.

[0072] The software system of the electronic device 10 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 10.

[0073] Figure 2 It is a software structure block diagram of the electronic device 10 according to an embodiment of the present application.

[0074] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: from top to bottom: the application layer, the application framework layer, the Android runtime (ART) and native C / C++ libraries, the hardware abstraction layer (HAL), and the kernel layer.

[0075] The application layer can include a series of application packages.

[0076] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0077] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0078] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and the like.

[0079] The window manager provides window management services (WMS). WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.

[0080] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0081] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0082] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0083] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0084] The Activity Manager can provide Activity Management Service (AMS), which can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers) as well as manage and schedule application processes.

[0085] The Input Manager provides Input Management Service (IMS), which manages system inputs, such as touch screen input, key input, and sensor input. The IMS retrieves events from input device nodes and, through interaction with the WMS, distributes the events to the appropriate window.

[0086] The Android runtime consists of the core libraries and the Android runtime. The Android runtime is responsible for converting source code into machine code. It primarily utilizes ahead-of-time (AOT) and just-in-time (JIT) compilation technologies.

[0087] The core library is mainly used to provide basic Java class library functions, such as basic data structures, mathematics, IO, tools, databases, networks, etc. The core library provides an API for users to develop Android applications.

[0088] Native C / C++ libraries can include multiple functional modules, such as surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0089] Among them, the surface manager is used to manage the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media framework supports playback and recording of a variety of commonly used audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides the drawing and operation of 2D and 3D graphics in applications. SQLite provides a lightweight relational database for applications of electronic device 10.

[0090] The hardware abstraction layer runs in user space, encapsulates kernel layer drivers, and provides a calling interface to the upper layer.

[0091] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0092] The following describes the workflow of the software and hardware of the electronic device 10 in conjunction with capturing a photo scene.

[0093] When the display receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the application framework layer interface to start the camera application, which then calls the kernel layer to start the camera driver and use the camera to capture still images or video.

[0094] The following combination Figure 3 A schematic diagram of an application scenario of the screen recording method provided by an embodiment of the present invention is exemplarily introduced.

[0095] This embodiment may include two electronic devices (e.g. Figure 1 For ease of description, the two electronic devices are hereinafter referred to as the first electronic device 100 and the second electronic device 200. The first electronic device 100 and the second electronic device 200 may be in the same local area network. For example, the first electronic device 100 and the second electronic device 200 establish a communication connection via Wi-Fi. The first electronic device 100 includes a first display screen 1001, and the second electronic device 200 includes a second display screen 2001. The first electronic device 100 is a source device, and the second electronic device 200 is a destination device. An application running on the first electronic device 100 can be projected and displayed on the second electronic device 200. The application may be a desktop application, a gallery application, a call application, a map application, a music application, a video application, etc.

[0096] The first electronic device 100 and the second electronic device 200 include but are not limited to mobile phones, tablet computers, PCs, laptop computers, etc. The following description will be made by taking the first electronic device 100 as a mobile phone and the second electronic device 200 as a laptop computer as an example.

[0097] The first electronic device 100 can open n applications APP1~APP n ,n is a positive integer. When the first electronic device 100 adds m applications APP1~APP m The screen is projected to the second electronic device 200, and the second electronic device 200 can generate m applications APP1~APP m There are m corresponding application windows, where m is a positive integer. For example, there are multiple applications APP1~APP mThe frame data output by SurfaceFlinger during runtime can be encoded and compressed by the video encoder to generate multiple applications APP1~APP m The first electronic device 100 can process the encoded data corresponding to the application window of the video encoder, for example, the data in YUV format. m The coded data is sent to the second electronic device 200, and the second electronic device 200 can decode the coded data and draw multiple applications APP1~APP m application window.

[0098] The user can view one or more applications APP1~APP m For example, the user can use the mouse and keyboard of the second electronic device 200 to control the applications APP1 to APP m Perform control operations.

[0099] For example, the first electronic device 100 opens a calculator application APP1, a gallery application APP2 and a chat application APP3. The user can perform touch operations on the interface of the gallery application APP2 to call up the screen projection function icon. When the user clicks the screen projection function icon, a list of devices that can be projected pops up. The device list may include information such as the device identity document (ID) or the device name. When the user selects the second electronic device 200 as the destination device in the device list, the first electronic device 100 can project the application window of the gallery application APP2 to the second electronic device 200, and the second electronic device 200 can generate an application window corresponding to the gallery application APP2. The user can use a similar projection operation method as the gallery application APP2 on the first electronic device 100 side to project the application window of the chat application APP3 to the second electronic device 200.

[0100] like Figure 3 As shown, the calculator application APP1 is displayed in the foreground of the first electronic device 100, the gallery application APP2 and the chat application APP3 are running in the background of the first electronic device 100, and the gallery application APP2 and the chat application APP3 are projected and displayed on the second electronic device 200.

[0101] When it is necessary to record the screen on the first electronic device 100, the user can select the application that needs to be recorded on the first electronic device 100 or the second electronic device 200. For example, the operating system of the first electronic device 100 has a built-in screen recording function, and the user can click the screen recording icon on the first electronic device 100 to turn on the screen recording function. When the user clicks the screen recording icon, the operating system can obtain the application information currently opened on the first electronic device 100, and pop up a list of applications that can be selected for screen recording. The application list can display the name or thumbnail of the application. When the user selects one or more applications in the application list, the first electronic device 100 can obtain the identification information of the one or more applications based on the user's operation. The identification information can refer to the process ID of the application or the package name of the application. The first electronic device 100 can start recording the one or more applications based on the identification information of the application. The process ID is a value used by the kernel of the device operating system to uniquely identify the process of the application.

[0102] In some embodiments, each application running on an electronic device has a unique process ID and package name, and each electronic device has a unique device ID. Therefore, a process in a certain electronic device can be uniquely identified by the process ID and device ID as the unique identification information of the application. For example, the device ID of an electronic device is the International Mobile Equipment Identity, and the applications installed on the electronic device include QQ The device ID of the electronic device, the process ID of each application, and the package name are shown in Table 1 below:

[0103] Table 1

[0104]

[0105] In some embodiments, when the screen recording scene only includes the first electronic device 100, when the first electronic device 100 records the foreground application or the background application, it can only use the process ID or package name of the application as the unique identification information of the application. When the screen recording scene includes multiple electronic devices, such as a scene where the application of the first electronic device 100 is projected and displayed on the second electronic device 200, when the first electronic device 100 records the foreground application or the background application, it can also only use the process ID or package name of the application as the unique identification information of the application. When the second electronic device 200 records the projection application, it can use the process ID and the device ID of the first electronic device 100 as the unique identification information of the projection application.

[0106] The first electronic device 100 may also have a third-party screen recording application installed or a screen recording application pre-installed at the factory, and the first electronic device 100 may enable the screen recording application. The screen recording application may obtain information about applications currently enabled on the first electronic device 100. When a user selects one or more applications for screen recording in the screen recording application of the first electronic device 100, the first electronic device 100 may obtain identification information of the one or more applications based on the user's operation, and the first electronic device 100 may start recording the one or more applications based on the identification information of the applications.

[0107] In some embodiments, the user can also specify the screen projection application (such as the gallery application APP2 and the chat application APP3) that needs to be recorded on the second electronic device 200, and the specific screen recording operation is performed by the first electronic device 100. For example, the second electronic device 200 can generate a screen recording instruction in response to the user's operation, and the second electronic device 200 can send the screen recording instruction to the first electronic device 100 through the Wi-Fi channel, so that the first electronic device 100 records the application specified by the user. For example, the second electronic device 200 is installed with a screen recording application, and the screen recording application of the second electronic device 200 can obtain the application information of the first electronic device 100 currently displayed on the screen of the second electronic device 200. When the user selects one or more applications for screen recording in the screen recording application of the second electronic device 200, the second electronic device 200 can generate a screen recording instruction associated with the one or more applications in response to the user operation, and the screen recording instruction may include the identification information of the application. The second electronic device 200 can send the screen recording instruction to the first electronic device 100 to control the first electronic device 100 to start recording the one or more applications selected by the user.

[0108] When the first electronic device 100 opens an application, the process of the application can request WMS to create a Surface for the application window. Multiple application processes can correspond to multiple Surfaces. Surface can refer to a data buffer provided by a screen data consumer to a screen data producer. The screen data producer can produce image content on the Surface, and the screen data consumers can draw the produced data on the screen or convert it into the data they need. For example, in the screen recording scenario of the first electronic device 100, a screen data producer, such as a virtual display (VirtualDisplay), can generate data on the Surface, and a screen data consumer, such as a multimedia recorder (MediaRecoder), can obtain data from the Surface. After obtaining the data, the multimedia recorder can use encoding methods such as the fast forward moving Picture expert group (FFMPEG) to encode the data into a video.

[0109] Assume that the first electronic device 100 has turned on the calculator application APP1, the gallery application APP2 and the chat application APP3, the calculator application APP1 is displayed in the foreground of the first electronic device 100, the gallery application APP2 and the chat application APP3 are running in the background of the first electronic device 100, and the user chooses to record the gallery application APP2 and the chat application APP3. The first electronic device 100 can obtain the identification information of the gallery application APP2 and the identification information of the chat application APP3 according to the user's screen recording operation. The first electronic device 100 obtains the media stream of the gallery application APP2 from the corresponding Surface according to the identification information of the gallery application APP2, and the first electronic device 100 can generate a screen recording file f1 corresponding to the gallery application APP2 based on the media stream of the gallery application APP2. For example, the first electronic device 100 can call SurfaceFlinger or Direct Digital Synthesizer (DDS) algorithm to perform image synthesis on the media stream of the gallery application APP2 to generate a screen recording file f1.

[0110] Similarly, the first electronic device 100 can obtain the media stream of the chat application APP3 from the corresponding Surface according to the identification information of the chat application APP3, and the first electronic device 100 can generate a screen recording file f2 corresponding to the chat application APP3 according to the media stream of the chat application APP3.

[0111] In some embodiments, the first electronic device 100 can encode the media stream on the Surface to form a screen recording file during or before the media stream in the Surface is sent to the second electronic device 200.

[0112] Assume that the user chooses to record the screen of the calculator application APP1. The first electronic device 100 can obtain the identification information of the calculator application APP1 based on the user's screen recording operation. The first electronic device 100 can obtain the media stream of the calculator application APP1 from the corresponding Surface based on the identification information of the calculator application APP1. The first electronic device 100 can generate a screen recording file f3 corresponding to the calculator application APP1 based on the media stream of the calculator application APP1. For example, the first electronic device 100 calls SurfaceFlinger or the DDS algorithm to perform image synthesis on the media stream of the calculator application APP1 to generate the screen recording file f3.

[0113] In some embodiments, the second electronic device 200 can also record the screen of the projection application (such as the gallery application APP2 and the chat application APP3). For example, the operating system of the second electronic device 200 has a built-in screen recording function, and the user can click the screen recording icon on the second electronic device 200 to turn on the screen recording function. When the user clicks the screen recording icon, the operating system can obtain the application information currently displayed on the screen of the second electronic device 200, and pop up a list of applications that can be selected for screen recording. The application list can display the name or thumbnail of the application. When the user selects one or more applications in the application list, the second electronic device 200 can obtain the identification information of the one or more applications according to the user's operation, and the second electronic device 200 can record the one or more applications according to the identification information of the application.

[0114] In the screen projection display scenario, the first electronic device 100 sends the media stream of the gallery application APP2 and the media stream of the chat application APP3 to the second electronic device 200. The second electronic device 200 decodes and displays the received media stream to display the gallery application APP2 and the chat application APP3 on the second display screen 2001.

[0115] The second electronic device 200 can obtain the media stream of the chat application APP3 from the received media stream based on the identification information of the chat application APP3. The identification information may include a process ID and a device ID (the device ID is used to determine that the media stream is sent by the first electronic device 100). The second electronic device 200 can call SurfaceFlinger or the DDS algorithm to perform image synthesis on the media stream of the chat application APP3 and generate a screen recording file corresponding to the chat application APP3. For example, the second electronic device 200 can perform image synthesis on the media stream of the chat application APP3 before sending it for display.

[0116] The second electronic device 200 can also obtain the media stream of the picture application APP2 from the received media stream according to the identification information of the picture application APP2. The second electronic device 200 can call SurfaceFlinger or DDS algorithm to perform image synthesis on the media stream of the picture application APP2 to generate a screen recording file corresponding to the picture application APP2.

[0117] The following combination Figure 4 A schematic diagram of an application scenario of a screen recording method provided by another embodiment of the present invention is exemplarily introduced.

[0118] This embodiment includes a first electronic device 100 and a second electronic device 200. The following description uses the example of a mobile phone as the first electronic device 100 and a vehicle-mounted device as the second electronic device 200. The first electronic device 100 can establish a communication connection with the second electronic device 200 via Bluetooth. The first electronic device 100 and the second electronic device 200 use an in-vehicle mobile phone mapping solution to establish a communication channel between the mobile phone and the vehicle, thereby extending the mobile phone's applications and services to the vehicle. In-vehicle mobile phone mapping solutions include, but are not limited to, HiCar, CarPlay, or CarLife.

[0119] like Figure 4 As shown, the first electronic device 100 uses the map application APP 11 、Phone application APP 12 , music application APP 13 The screen is projected and displayed on the second electronic device 200. In existing in-vehicle screen projection scenarios, when an application on a mobile phone is projected onto the screen of the vehicle-mounted device, the application is running in the background of the mobile phone and cannot be opened or brought to the foreground for screen recording. Since vehicle-mounted devices generally do not support screen recording, users cannot record the screen of the projected application on their mobile phone or vehicle-mounted device.

[0120] In the present application, in order to realize the screen recording of the projection application in the in-vehicle projection scenario, the first electronic device 100 can obtain the identification information of the application to be recorded based on the user's screen recording operation (operation on the first electronic device 100 side), and the identification information can refer to the process ID of the application or the package name of the application. The first electronic device 100 can obtain the media stream of the application from the corresponding Surface based on the identification information of the application, and the first electronic device 100 can generate a screen recording file corresponding to the application based on the media stream of the application. For example, the first electronic device 100 calls SurfaceFlinger or DDS algorithm to perform image synthesis on the media stream of the application to generate a screen recording file.

[0121] When the first electronic device 100 opens the map application APP 11 、Phone application APP 12 and music application APP 13 When, map application APP 11 The process can request WMS to create a Surface for its application window, the phone application APP 12 The process can request WMS to create a Surface for its application window, such as the music application APP 13 The process can request WMS to create a Surface for its application window. 11 With phone application APP 12Take screen recording as an example. Assuming that the operating system of the first electronic device 100 has a screen recording function, the user can click the screen recording icon on the first electronic device 100 to turn on the screen recording function. When the user clicks the screen recording icon, the operating system can obtain the application information currently opened by the first electronic device 100, and pop up a list of applications that can be selected for screen recording. For example, the application list can display the name of the application. When the user selects the map application APP in the application list 11 With phone application APP 12 When the name of the first electronic device 100 is displayed, the first electronic device 100 can obtain the map application APP 11 With phone application APP 12 The first electronic device 100 uses the map application APP 11 Get the map application APP from the corresponding Surface 11 The first electronic device 100 uses the map application APP 11 Media streams are synthesized to generate images and map applications 11 The first electronic device 100 can also use the phone application APP 12 Get the phone application APP from the corresponding Surface 12 The first electronic device 100 uses the phone application APP 12 The media stream is synthesized to generate images and the phone application APP 12 The corresponding screen recording file.

[0122] In some embodiments, the first electronic device 100 may also be installed with a third-party screen recording application or pre-installed with a screen recording application at the factory. The user can also open the screen recording application in the car screen projection scene to record the screen. After the first electronic device 100 opens the screen recording application, the screen recording application can obtain the application information currently opened by the first electronic device 100. When the user selects the map application APP in the screen recording application, 11 With phone application APP 12 When recording the screen, the first electronic device 100 can obtain the map application APP according to the user's operation 11 With phone application APP 12 The first electronic device 100 can use the map application APP 11 With phone application APP 12 Get the identification information of the map application APP 11 With phone application APP 12 The first electronic device 100 then respectively processes the map application APP 11 Media streaming and phone application APP 12 Media streams are synthesized to generate images and map applications11 Corresponding screen recording files and phone application APP 12 The corresponding screen recording file.

[0123] The following combination Figure 5 A schematic diagram of an application scenario of a screen recording method provided in another embodiment of the present invention is exemplarily introduced.

[0124] This embodiment includes a first electronic device 100 and a second electronic device 200. The first electronic device 100 can be a small-screen terminal device such as a mobile phone or a tablet, and the second electronic device 200 can be a large-screen display device (such as a TV, a smart screen, etc.). The following is an example in which the first electronic device 100 is a mobile phone and the second electronic device 200 is a TV. The first electronic device 100 can establish a communication connection with the second electronic device 200 via Wi-Fi. In a distributed gaming scenario, the first electronic device 100 can be used as an operating handle, and the second electronic device 200 can be used as a display terminal to display the game screen projected by the first electronic device 100.

[0125] In existing distributed gaming scenarios, screen recording involves recording the screen display of the first electronic device 100 on the first electronic device 100 side and the screen display of the second electronic device 200 on the second electronic device 200 side. Simultaneous recording of the screen display of both devices is not possible. Currently, the only option is to record the screen display of both devices separately and then splice the two recorded files together. Due to network latency, the clocks of the two devices may not be precisely synchronized, resulting in a time difference in the spliced ​​screen recording file.

[0126] In this application, in order to realize the synchronization (clock synchronization) of screen display contents on both sides in a distributed game scene, the screen display contents of the first electronic device 100 and the second electronic device 200 are recorded on the first electronic device 100 side. Figure 5 In the distributed gaming scenario shown, a mobile phone serves as the game controller and a TV serves as the game display. The two devices (mobile phone and TV) have different device IDs. However, since the game applications running on both devices are from the same developer and share the same package name, the two devices have the same package name. However, the kernels of the operating systems of the different devices generate different values ​​to uniquely identify the processes of the applications running on them, resulting in different process IDs for the two devices' game applications.

[0127] In a distributed gaming scenario, when the first electronic device 100 projects a gaming application onto the second electronic device 200, the first electronic device 100 can automatically turn on the gaming control controls. At this point, the first electronic device 100 displays the gaming control interface, and the second electronic device 200 displays the gaming screen. Users can customize the position and size of the buttons in the gaming control interface. WMS creates two independent Surfaces for the gaming application process within the first electronic device 100. The two independent Surfaces can correspond to the same process ID, and screen data producers can produce gaming control interface content on one Surface and gaming screen content on another Surface.

[0128] In some embodiments, when the first electronic device 100 projects a game application onto the second electronic device 200, the first electronic device 100 may display a controller function icon. When a user clicks the controller function icon, the first electronic device 100 enters "controller mode" and displays the game control interface.

[0129] The first electronic device 100 can respond to the user's screen recording operation and obtain media streams from two Surfaces. The first electronic device 100 can time-align the two media streams based on the timestamps of the two media streams, and then perform image synthesis and splicing processing on the two time-aligned media streams to generate a screen recording file including the game operation interface and the game screen interface. The user can then view the time-synchronized screen recording of the game operation interface and the game screen interface, thereby improving the user experience. For example, the splicing processing can refer to the game operation interface and the game screen interface in the screen recording file being arranged in a top-down split screen, or in a left-right split screen.

[0130] In some embodiments, the game operation interface in the screen recording file can also be displayed in a floating position in a specified area of ​​the game screen interface, and the position of the area can be customized by the user. For example, when the user performs a screen recording operation on the first electronic device 100, the screen recording operation includes setting the game operation interface to be displayed in a floating position in the lower left corner area or the lower right corner area of ​​the game screen interface.

[0131] In some embodiments, the first electronic device 100 can also respond to the user's screen recording operation and obtain media streams from two Surfaces. The first electronic device 100 can perform image synthesis on the two media streams respectively to generate a first screen recording sub-file corresponding to the game operation interface and a second screen recording sub-file corresponding to the game screen interface. The first electronic device 100 can then perform time alignment and splicing processing on the first screen recording sub-file and the second screen recording sub-file based on the timestamp of the first screen recording sub-file and the timestamp of the second screen recording sub-file to generate a screen recording file including the game operation interface and the game screen interface, so that the user can view the screen recording of the game operation interface and the game screen interface that are synchronized in time.

[0132] Assuming that the operating system of the first electronic device 100 has a built-in screen recording function, the user can click the screen recording icon on the first electronic device 100 to turn on the screen recording function. When the user clicks the screen recording icon, the operating system can obtain the game application information currently opened by the first electronic device 100, and the first electronic device 100 obtains the media stream of the game application from the corresponding Surface based on the identification information of the game application. For example, the identification information of the game application includes the process ID of the game application, and the first electronic device 100 can determine the two surfaces corresponding to the game application based on the process ID of the game application. The first electronic device 100 can obtain the media stream from the two Surfaces, and the first electronic device 100 can time-align the two media streams based on the timestamps of the two media streams, and then perform image synthesis and splicing processing on the two media streams that have been time-aligned to generate a screen recording file including the game operation interface and the game screen interface.

[0133] Reference Figure 6 As shown, an embodiment of the present application provides a screen recording method, which is applied to a first electronic device 100. In this embodiment, the screen recording method may include:

[0134] 61. In response to a first operation of starting a screen recording function, obtain identification information of an application associated with the first operation.

[0135] In some embodiments, when a user wishes to record the screen, the user can activate the screen recording function of the first electronic device 100, for example, by performing a first operation to activate the screen recording function of the first electronic device 100. The first operation can be, for example, an operation in which the user presses a physical button (for example, simultaneously pressing the power button and the volume up button), the user performs a predefined gesture (for example, sliding down the screen with three fingers, tapping the screen with knuckles, shaking the phone), the user operates a corresponding button on the screen (clicking the screen recording control in the notification bar), the user enters a voice command, the user operates on the screen recording application, etc. The embodiments of the present application do not specifically limit the screen recording operation.

[0136] In some embodiments, the application associated with the first operation may include one or more applications, and the one or more applications may be running in the foreground or background of the first electronic device 100, or partially in the foreground and partially in the background. Identification information may refer to information that can uniquely identify an application, for example, the identification information may include the process ID of the application or the package name of the application. In other embodiments, the identification information may also include the process ID of the application and the ID of the device running the application, or the package name of the application and the ID of the device running the application.

[0137] In some embodiments, in a screen projection scenario, when one or more applications running on the first electronic device 100 are projected and displayed on the second electronic device 200, the user can also specify the screen projection application that needs to be recorded on the second electronic device 200, and the specific screen recording operation is performed by the first electronic device 100. The first electronic device 100 can also obtain the identification information of the application associated with the screen recording instruction in response to the screen recording instruction sent by the second electronic device 200. For example, the second electronic device 200 can generate a screen recording instruction associated with one or more applications in response to a user operation, and the screen recording instruction can include the identification information of the application. The second electronic device 200 can send the screen recording instruction to the first electronic device 100, and then the first electronic device 100 can obtain the identification information of the application to be recorded.

[0138] 62. Obtain the media stream of the application according to the identification information.

[0139] In some embodiments, when the first electronic device 100 opens an application, the process of the application can request WMS to create a Surface for the application window. Each application process can correspond to at least one Surface. Surface can refer to a data buffer provided by a screen data consumer to a screen data producer. The screen data producer can produce image content on the Surface, and the screen data consumers can consume the produced data on the screen (draw it) or convert it into the data they need. The first electronic device 100 can determine the Surface corresponding to the application based on the identification information of the application, and the first electronic device 100 can obtain the media stream of the application from the Surface corresponding to the application.

[0140] 63. Generate a screen recording file corresponding to the application based on the media stream of the application.

[0141] In some embodiments, when the first electronic device 100 obtains the media stream of an application, the first electronic device 100 can call SurfaceFlinger or DDS algorithm to perform image synthesis on the media stream of the application to generate a screen recording file corresponding to the application.

[0142] In some embodiments, in certain application scenarios, the first electronic device 100 may be able to obtain two media streams based on the identification information of a certain application. For example, in a distributed game scenario, when the first electronic device 100 projects the game application to the second electronic device 200, the first electronic device 100 can automatically turn on the game control controls. At this time, the first electronic device 100 displays the game control interface and the second electronic device 200 displays the game screen. WMS creates two independent Surfaces (for example, a first Surface and a second Surface) for the process of the game application in the first electronic device 100. The two independent Surfaces can correspond to the same process ID. The screen data producer can produce the game control interface content on the first Surface and the game screen content on the second Surface.

[0143] The first electronic device 100 can respond to the user's screen recording operation and obtain media streams from the first Surface and the second Surface. The first electronic device 100 can time-align the two media streams based on the timestamps of the two media streams, and then perform image synthesis and splicing processing on the two time-aligned media streams to generate a screen recording file including the game operation interface and the game screen interface. The user can then view the time-synchronized recording of the game operation interface and the game screen interface, thereby improving the user experience. For example, the splicing processing can mean that the game operation interface and the game screen interface in the screen recording file are arranged in a top-down split screen, or in a left-right split screen.

[0144] In some embodiments, the game operation interface in the screen recording file can also be displayed in a floating position in a specified area of ​​the game screen interface, and the position of the area can be customized by the user. For example, when the user performs a screen recording operation on the first electronic device 100, the screen recording operation includes setting the game operation interface to be displayed in a floating position in the lower left corner area or the lower right corner area of ​​the game screen interface.

[0145] In some embodiments, the first electronic device 100 may also respond to the user's screen recording operation and obtain media streams from the first Surface and the second Surface. The first electronic device 100 may perform image synthesis on the two media streams to generate a first screen recording sub-file corresponding to the game operation interface and a second screen recording sub-file corresponding to the game screen interface. The first electronic device 100 may then perform time alignment and splicing processing on the first screen recording sub-file and the second screen recording sub-file based on the timestamps of the first screen recording sub-file and the second screen recording sub-file to generate a screen recording file including the game operation interface and the game screen interface, so that the user can view the time-synchronized recording of the game operation interface and the game screen interface.

[0146] The above-mentioned screen recording method obtains the identification information of the application to be recorded, and then generates a screen recording file based on the media stream obtained from the identification information. Compared with the existing screen recording of the local desktop, this application can only record the interface of one or more applications of the electronic device, and the recorded application can run in the foreground or the background of the electronic device. At the same time, two or more screen display contents in a distributed display scenario can be recorded on the local electronic device side. Recording multiple screen display contents by the same device can ensure the synchronization of the recording clock, and enable users to view the contents of multiple display interfaces with time synchronization in one screen recording file.

[0147] Reference Figure 7 As shown, an embodiment of the present application provides a screen recording method, which is applied to a first electronic device 100. The first electronic device 100 and the second electronic device 200 may be in the same local area network. For example, the first electronic device 100 and the second electronic device 200 establish a communication connection via Wi-Fi or Bluetooth. The first electronic device 100 includes a first display screen 1001, and the second electronic device 200 includes a second display screen 2001. The first electronic device 100 is a source device, and the second electronic device 200 is a destination device. One or more applications running on the first electronic device 100 can be projected and displayed on the second electronic device 200. In this embodiment, the screen recording method may include:

[0148] 71. When the first electronic device 100 detects a first operation for starting a screen recording function, the first electronic device 100 obtains identification information of an application associated with the first operation, where the application associated with the first operation includes an application for projecting the screen to the second electronic device.

[0149] In some embodiments, in a screen projection scenario, when a user wishes to record the screen, the user can activate the screen recording function of the first electronic device 100, for example, by performing a first operation to activate the screen recording function of the first electronic device 100. The first operation can be, for example, any one of the following: the user pressing a physical button, the user performing a predefined gesture, the user operating a corresponding button on the screen, the user inputting a voice command, the user operating a screen recording application, etc. The embodiments of the present application do not specifically limit the screen recording operation.

[0150] In some embodiments, the application associated with the first operation may include one or more applications, which may be running in the foreground of the first electronic device 100, or in the background (for example, casting the screen on the second electronic device 200), or partially in the foreground and partially in the background (for example, casting the screen on the second electronic device 200). Identification information may refer to information that can uniquely identify an application, for example, identification information may include the process ID of the application or the package name of the application. In other embodiments, the identification information may also include the device ID of the device running the application.

[0151] 72. When the first electronic device 100 receives a screen recording instruction sent by the second electronic device 200, the first electronic device 100 obtains identification information of an application associated with the screen recording instruction, where the application associated with the screen recording instruction includes an application for projecting the screen to the second electronic device.

[0152] In some embodiments, the user can also specify the screen projection application that needs to be recorded on the second electronic device 200, and the specific screen recording operation is performed by the first electronic device 100. When the first electronic device 100 receives the screen recording instruction sent by the second electronic device 200, the first electronic device 100 obtains the identification information of the application associated with the screen recording instruction. For example, the second electronic device 200 can generate a screen recording instruction associated with one or more applications in response to the user operation, and the screen recording instruction can include the identification information of the application. The second electronic device 200 can send the screen recording instruction to the first electronic device 100, and then the first electronic device 100 can obtain the identification information of the application to be recorded.

[0153] In some embodiments, the first electronic device 100 may choose to execute step 71 or step 72.

[0154] 73. The first electronic device 100 obtains the media stream of the application according to the identification information.

[0155] In some embodiments, the first electronic device 100 can determine the Surface corresponding to the application based on the identification information of the application, and the first electronic device 100 can obtain the media stream of the application from the Surface corresponding to the application.

[0156] 74. The first electronic device 100 generates a screen recording file corresponding to the application based on the media stream of the application.

[0157] In some embodiments, when the first electronic device 100 obtains the media stream of an application, the first electronic device 100 can call SurfaceFlinger or DDS algorithm to perform image synthesis on the media stream of the application to generate a screen recording file corresponding to the application.

[0158] In some embodiments, in certain application scenarios, the first electronic device 100 may be able to obtain two media streams based on the identification information of a certain application. For example, in a distributed gaming scenario, the first electronic device 100 displays the game control interface, and the second electronic device 200 displays the game screen. The WMS creates two independent Surfaces (e.g., a first Surface and a second Surface) for the gaming application process within the first electronic device 100. The two independent Surfaces can correspond to the same process ID, and the screen data producer can produce the game control interface content on the first Surface and the game screen content on the second Surface.

[0159] The first electronic device 100 can respond to the user's screen recording operation and obtain media streams from the first Surface and the second Surface. The first electronic device 100 can time-align the two media streams based on the timestamps of the two media streams, and then perform image synthesis and splicing processing on the two time-aligned media streams to generate a screen recording file including the game operation interface and the game screen interface. The user can then view the time-synchronized recording of the game operation interface and the game screen interface, thereby improving the user experience. For example, the splicing processing can mean that the game operation interface and the game screen interface in the screen recording file are arranged in a top-down split screen, or in a left-right split screen.

[0160] In some embodiments, the game operation interface in the screen recording file can also be displayed in a floating position in a specified area of ​​the game screen interface, and the position of the area can be customized by the user. For example, when the user performs a screen recording operation on the first electronic device 100, the screen recording operation includes setting the game operation interface to be displayed in a floating position in the lower left corner area or the lower right corner area of ​​the game screen interface.

[0161] In some embodiments, the first electronic device 100 may also respond to the user's screen recording operation and obtain media streams from the first Surface and the second Surface. The first electronic device 100 may perform image synthesis on the two media streams to generate a first screen recording sub-file corresponding to the game operation interface and a second screen recording sub-file corresponding to the game screen interface. The first electronic device 100 may then perform time alignment and splicing processing on the first screen recording sub-file and the second screen recording sub-file based on the timestamps of the first screen recording sub-file and the second screen recording sub-file to generate a screen recording file including the game operation interface and the game screen interface, so that the user can view the time-synchronized recording of the game operation interface and the game screen interface.

[0162] In some embodiments, the second electronic device 200 can also record the screen of the screen projection application. In the existing screen projection display scenario, the second electronic device 200 can receive the media stream sent by the first electronic device 100, and decode the received media stream for display. When the user wants to record the screen on the second electronic device 200 side, the user can start the screen recording function of the second electronic device 200, such as performing a second operation to start the screen recording function of the second electronic device 200. Among them, the second operation can be, for example, any one of the operations of the user pressing a physical button, the user performing a predefined gesture, the user operating a corresponding button on the screen, the user inputting a voice command, and the user operating on the screen recording application. The embodiment of the present application does not specifically limit the screen recording operation.

[0163] When the second electronic device 200 detects the second operation for starting the screen recording function, the second electronic device 200 obtains the identification information of the application associated with the second operation, wherein the identification information of the application associated with the second operation may include the process ID of the application and the device ID of the first electronic device 100 (the device ID of the first electronic device 100 is used to determine that the media stream indeed originates from the first electronic device 100). The second electronic device 200 can obtain the media stream corresponding to the identification information of the application associated with the second operation from the received media stream, and the second electronic device 200 generates a screen recording file based on the obtained media stream. For example, the second electronic device 200 can perform image synthesis on the media stream to generate a screen recording file before sending the received media stream for display.

[0164] The above-mentioned screen recording method obtains the identification information of the application to be recorded, and then generates a screen recording file based on the media stream obtained from the identification information. Compared with the existing inability to record the projection application, this application can realize the recording of the background application of the electronic device or the projection application that is projected to other electronic devices. At the same time, it can record the display contents of two or more screens in a distributed display scenario on the local electronic device side. Recording multiple screen display contents by the same device can ensure the synchronization of the recording clock, and allow users to view the contents of multiple display interfaces with time synchronization in one screen recording file.

[0165] refer to Figure 8 , is a schematic diagram of the hardware structure of the first electronic device 100 provided in an embodiment of the present application. Figure 8As shown, the first electronic device 100 may include a first display screen 1001, a first processor 1002, a first memory 1003, and a first communication bus 1004. The first memory 1003 is used to store one or more first computer programs 1005. The one or more first computer programs 1005 are configured to be executed by the first processor 1002. The one or more first computer programs 1005 include instructions, which can be used to implement the execution of the following in the first electronic device 100: Figure 6 or Figure 7 The screen recording method.

[0166] It is understood that the structure shown in this embodiment does not constitute a specific limitation on the first electronic device 100. In other embodiments, the first electronic device 100 may include more or fewer components than shown, or combine or separate some components, or arrange the components differently.

[0167] This embodiment also provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the screen recording method in the above-mentioned embodiment.

[0168] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the screen recording method in the above-mentioned embodiment.

[0169] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the screen recording method in the above-mentioned method embodiments.

[0170] Among them, the first electronic device, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0171] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0174] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0176] The above description is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A screen recording method, applied to a first electronic device, characterized in that: The screen recording method includes: In response to a first operation of starting a screen recording function, obtaining identification information of an application associated with the first operation, wherein the application associated with the first operation includes an application for projecting a screen to a second electronic device, and the application corresponds to a first screen data buffer and a second screen data buffer; Acquire a first media stream from the first screen data buffer and acquire a second media stream from the second screen data buffer according to the identification information; A screen recording file corresponding to the application is generated according to the first media stream and the second media stream.

2. The screen recording method according to claim 1, wherein: The identification information includes the process identity ID of the application or the package name of the application.

3. The screen recording method according to claim 1 or 2, wherein: The generating a screen recording file corresponding to the application according to the first media stream and the second media stream includes: Performing image synthesis on the first media stream to generate a first screen recording sub-file; Performing image synthesis on the second media stream to generate a second screen recording sub-file; The first screen recording sub-file and the second screen recording sub-file are spliced ​​together to generate a screen recording file corresponding to the application.

4. The screen recording method according to claim 3, wherein: The step of splicing the first screen recording sub-file and the second screen recording sub-file includes: Performing time alignment processing on the first screen recording subfile and the second screen recording subfile according to the timestamps of the first screen recording subfile and the second screen recording subfile; The first screen recording sub-file and the second screen recording sub-file that have undergone time alignment processing are spliced ​​together.

5. The screen recording method according to claim 1 or 2, wherein: The generating a screen recording file corresponding to the application according to the first media stream and the second media stream includes: Perform image synthesis and splicing on the first media stream and the second media stream to generate a screen recording file corresponding to the application.

6. The screen recording method according to claim 5, wherein: The synthesizing and splicing the first media stream and the second media stream includes: Performing time alignment processing on the first media stream and the second media stream according to the timestamps of the first media stream and the second media stream; Perform image synthesis and splicing on the first media stream and the second media stream that have undergone time alignment processing.

7. The screen recording method according to any one of claims 1 to 6, wherein: The screen recording file includes a first interface and a second interface, the first interface and the second interface are displayed in a left-right split screen, or the first interface and the second interface are displayed in an upper-lower split screen, or part or all of the first interface is displayed floating on the second interface.

8. The screen recording method according to any one of claims 1 to 7, wherein: The screen recording method further includes: In response to the screen recording instruction sent by the second electronic device, identification information of an application associated with the screen recording instruction is obtained.

9. The screen recording method according to any one of claims 1 to 7, wherein: The application associated with the first operation also includes an application running in the background of the first electronic device.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the screen recording method according to any one of claims 1 to 9.

11. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the screen recording method according to any one of claims 1 to 9.

Images