Virtual screen processing method and electronic device

Abstract

A virtual screen processing method and an electronic device, applied to the technical field of terminals. The method comprises: when GI synthesis of a first frame is performed in an image processing process, using a communication thread in the image processing process to communicate with an application process to obtain a communication result; when the communication result is Normal, sending a synthesized GI to a virtual screen created by the application process; and when the communication result is Abnormal, deleting a virtual screen object. When the application process response is not timely or the communication between an application process and an image processing process is abnormal, according to the method, a task thread of the image processing process is prevented from being blocked, such that the impact of abnormality is controlled in a communication thread without affecting other task threads; and the communication thread is split from an image synthesis thread, and the communication thread is used for communicating with the application process, thereby solving the problem of a frozen screen of a display interface of an electronic device to a certain extent, and improving the device stability and user experience.

Description

Virtual screen processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 18, 2023, with application number 202311051429.9 and invention name “A method for processing a virtual screen and an electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a virtual screen processing method and electronic device. Background Art

[0003] When an electronic device implements functions such as screen recording or screen projection, an application (App) is required to create a virtual screen. After the creation is completed, the Graphic Interface (GI) on the virtual screen is synthesized and sent to the virtual screen for display. The App can also be called an application process.

[0004] Related solutions use the image synthesis thread within the image processing process, such as the SurfaceFlinger process, to simultaneously synthesize the GI on the virtual screen and the GI on the electronic device's display interface. However, the display interface of the electronic device often freezes. In this state, the display interface becomes unresponsive to user triggers, resulting in a poor user experience.

[0005] Summary of the Invention

[0006] The present application provides a virtual screen processing method and electronic device, the purpose of which is to solve the problem of frozen screen state of the display interface of the electronic device to a certain extent and improve the user experience.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a virtual screen processing method, specifically comprising:

[0009] When the image processing process performs GI synthesis of the first frame, the communication thread in the image processing process is used to communicate with the application process to obtain a communication result. When the communication result is normal, the synthesized GI is sent to the application process to be displayed on the virtual screen created by the application process. When the communication result is abnormal, the virtual screen object is deleted. At this time, when the application process does not respond in time, or the communication with the image processing process is abnormal, the task thread of the image processing process is prevented from being blocked, so that the impact of the abnormality is controlled in the communication thread, and other task threads are not affected. Since the communication thread is only used for cross-process communication with the application process, it is not used for GI synthesis of the electronic device display interface. Therefore, the blocking of the communication thread will not affect the GI synthesis of the electronic device display interface. The GI synthesis of the electronic device display interface is related to the frozen screen of the display interface. When the GI of the electronic device display interface is synthesized normally, the display interface will not be frozen. However, when the image processing process uses the communication part of the image synthesis thread to communicate with the application process, if the application process does not respond in time or the communication with the image processing process is abnormal, the communication part of the image processing process is blocked, and the other threads of the image processing process are blocked, resulting in the inability to synthesize the GI of the electronic device display interface and the GI on the virtual screen. In summary, this application uses the communication thread in the image processing process to communicate with the application process, effectively reducing the impact of a single application process abnormality on the underlying image processing process, and to a certain extent solves the problem of frozen screen and stuck of the electronic device display interface, thereby improving the device stability and user experience.

[0010] In one possible implementation, when the image synthesis thread in the image processing process sends a first event signal to the communication thread, the communication thread sends a memory acquisition message to the application process. The memory acquisition message is used to determine the memory for caching the synthesized GI. When the communication thread receives the memory parameters sent by the application process within a preset time and sends a first result signal to the image synthesis thread, the communication result is determined to be normal. The memory parameters are used to specify the memory. When the communication thread does not receive the memory parameters sent by the application process within the preset time, the communication result is determined to be abnormal. In this way, the image synthesis thread is separated from the communication thread, avoiding the blockage of the communication thread causing the blockage of the image synthesis thread, which may cause the GI on the electronic device display interface to be unable to be synthesized and cause a frozen screen.

[0011] In another possible implementation, when the first event signal is an event signal marked as DequeueBuffer, the memory acquisition message is DequeueBuffer(). Thus, DequeueBuffer() specifies the order in which memory is acquired, prioritizing the target GraphicBuffer (hereinafter referred to as Buffer) from the slot bound to the memory. This improves determination efficiency.

[0012] Optionally, the memory acquisition message also includes but is not limited to memory size and pixel format information. The App can determine the target Buffer from the Buffer Queue based on the width, height and pixel format specified by the first event signal. In another achievable method, when the communication result is that the communication is normal, the synthesized GI is cached in the memory and a second event signal is sent to the communication thread; in response to the communication thread receiving the second event signal, a return message is sent to the application process using the communication thread; when the communication thread receives the return success instruction sent by the application process and sends the second result signal to the image synthesis thread, it is determined that the synthesized GI is cached in the memory.

[0013] Optionally, after the image processing process synthesizes the GI and the communication thread is determined to be terminated, the virtual screen object is destroyed.

[0014] Optionally, the second event signal is an event signal marked QueueBuffer, and the return message is QueueBuffer().

[0015] Optionally, when a communication anomaly occurs, a termination thread is constructed and the GI synthesis of the second frame is entered. The termination thread is used to terminate the communication thread, and the second frame is the next frame of the first frame; determine whether the communication thread is terminated; when it is determined that the communication thread has not been terminated, enter the GI synthesis of the third frame, and again determine whether the communication thread is terminated. The third frame is the next frame of the second frame. The cycle continues until it is determined that the communication thread is terminated and the virtual screen object is deleted. In this way, when communication anomalies occur or the App does not respond in a timely manner, the electronic device display interface can perform GI synthesis normally.

[0016] Optionally, when the system service process detects that the application process has failed and sends a close instruction to the application process to close the application process, the communication thread receives a failure instruction of the memory acquisition instruction sent by the application process to determine that the communication thread is terminated.

[0017] Optionally, when the application process creates a virtual screen, a communication thread is created in the image processing process, and a virtual screen object is created to maintain a corresponding relationship between the virtual screen and the communication thread.

[0018] Optionally, the communication thread is terminated and the virtual screen object is destroyed. In this way, the virtual screen is created when it is needed and deleted when it is no longer needed, which can prevent the virtual screen from affecting other functions.

[0019] Optionally, when the application process is a screen recording process, the synthesized GI is sent to the application process so that the application process encodes the synthesized GI into a video format for storage; when the application process is a screen projection process, the synthesized GI is sent to the application process so that the application process sends the synthesized GI to the screen projection device for display.

[0020] Alternatively, a communication thread can be used to communicate with the application process based on the Binder mechanism. Compared to other transmission mechanisms, the Binder mechanism only requires one copy, which is faster, more secure, and easier to use.

[0021] A second aspect provides an electronic device, which includes a memory and a processor, wherein the memory is coupled to the processor; the memory stores a program, and when the program is executed by the processor, the electronic device executes any method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of the internal structure of BufferQueueCore;

[0023] FIG2A is a schematic diagram of an application scenario of a virtual screen provided in an embodiment of the present application;

[0024] FIG2B is a schematic diagram of another application scenario of a virtual screen provided in an embodiment of the present application;

[0025] FIG3 is an interactive diagram of a method for performing virtual screen synthesis in a related solution;

[0026] FIG4 is a schematic diagram showing the principle of blocking the image processing process in a related solution;

[0027] FIG5 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application;

[0028] FIG6 is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0029] FIG7 is a flow chart of a method for processing a virtual screen according to an embodiment of the present application;

[0030] FIG8 is an interactive diagram of a method for synthesizing a virtual screen provided in an embodiment of the present application;

[0031] FIG9 is an interactive diagram of a virtual screen synthesis failure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The terms "first", "second" and "third" in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0033] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0034] The following is an introduction to the professional terms involved in the embodiments of this application.

[0035] BufferQueue: It consists of three parts: BufferQueueCore, which is responsible for maintaining the basic data structure of BufferQueue; BufferQueueProducer and BufferQueueConsumer, which are used to provide the basic interface for operating BufferQueue.

[0036] See Figure 1, which is a schematic diagram of the internal structure of BufferQueueCore. BufferQueueCore includes five queues: mQueue, mSlots, mFreeSlots, mFreeBuffers, and mActiveBuffers.

[0037] Among them, mQueue is used to store the First Input First Output (FIFO) queue of BufferItem.

[0038] mSlots is the BufferSlot structure data, and the array length is 64, from slot 0 to slot 64. Below, the slot parameter is used to replace the BufferSlot array.

[0039] The slot state of mFreeSlots is FREE and no Buffer is bound to it, that is, the slot state is FREE and NULL. The FREE state indicates that the current owner of this BufferSlot is BufferQueue and the current state can be removed from the queue by calling the Dequeue method of BufferQueueProducer. The NULL state indicates that no Buffer is bound to the slot.

[0040] The slot status of mFreeBuffers is FREE and a Buffer is bound to it. That is, the slot status is FREE and the buffer status. The buffer status indicates that a Buffer is bound to the slot.

[0041] The slot status of mActiveBuffers is NON-FREE. The NON-FREE status is at least one of DEQUEUED, QUEUED, ACQUIRED, and SHARED.

[0042] The DEQUEUED state indicates that the current slot has been dequeued from the queue by the BufferQueueProducer. However, the BufferQueueProducer cannot fill or modify the buffer contents until the last consumer using the buffer sends a Fence signal to release the buffer. At this point, the buffer is owned by the BufferQueueProducer.

[0043] The QUEUED state indicates that the buffer is queued by the BufferQueueProducer by calling the Queue method and is owned by the BufferQueue. At this time, the buffer described by this slot can be taken away by the BufferQueueConsumer.

[0044] ACQUIRED: After the Buffer is taken from the queue by BufferQueueConsumer calling the Acquire method, the state changes to the Acquired state. At this time, BufferQueueConsumer cannot access the contents of the Buffer until it receives the relevant Fence signal.

[0045] Unused Slots is a collection of slots that are not used.

[0046] Binder mechanism: A cross-process communication model based on the client and server. The server will register with the Service Manager, and the client will achieve inter-process communication with the server by obtaining services.

[0047] A thread is the most basic unit of program execution. It doesn't exist independently, but rather within a process. It's the smallest unit involved in system scheduling and the actual execution unit within a process. When an application is launched, the system creates a process. While processes don't run, it's the threads within them that can. A process is merely a container, containing the data structures and environment variables necessary for thread execution. Multiple threads within the same process share all system resources, such as the virtual address space, file descriptors, and signal handlers. However, each thread within a process has its own call stack, register context, and thread-local storage (TLS). A thread is a single, sequential flow of control within a process (i.e., execution path). Multiple threads can be created within a process, allowing them to run concurrently, with each thread performing a different task. For example, if an application has two tasks that need to run concurrently, Task 1 and Task 2, Task 1 and Task 2 can be placed in separate threads to achieve concurrent execution.

[0048] A virtual screen, also known as a virtual screen, corresponds to the display screen of an electronic device and is used for virtual display of a GI, mainly for screen recording, or screen projection, etc. In the embodiment of the present application, the virtual screen is created and destroyed by the App.

[0049] See FIG2A , which is a schematic diagram of an application scenario of a virtual screen provided in an embodiment of the present application.

[0050] As shown in FIG2A (a), the system is currently running a game app corresponding to a game interface 11. While the game app is running, if the user wants to record the screen, they can pull down to trigger the system to display a pull-down notification bar 12 containing a "Record Screen" button 102, as shown in FIG2A (b).

[0051] It should be noted that (b) in FIG. 2A is merely an example of how a "record screen" button can be set in the pull-down notification bar 12. In actual use, the "record screen" button is not limited to being set in the pull-down notification bar. In addition, the pull-down notification bar may also include other function buttons, such as a wireless network button 101, a flashlight on / off button 103, and a screen projection button 104.

[0052] The user can perform a touch operation on the "record screen" button 102, where the touch operation may include but is not limited to a single click, double click, or long press operation. After the electronic device receives the user's touch on the "record screen" button in the pull-down notification bar, it records the screen, resulting in a recorded screen 13 as shown in FIG2A (c).

[0053] The recorded image is the synthesized GI. The specific synthesis method is as follows:

[0054] The screen recording app initiates the creation of a virtual screen. Once the virtual screen is created, the image processing process is triggered to synthesize each frame. Specifically, the GI on the virtual screen is synthesized with the GI on the electronic device's display interface. The synthesized GI is sent to the electronic device's display interface for display, representing the recorded image 13 in (c). It is then sent to the screen recording app, which encodes the image into a video format for storage.

[0055] In actual use, the Media Recorder in the Android.media.API in the application architecture layer can be called, that is, the multimedia creation interface in the media manager to create a data buffer, which is used to receive the captured screen images. The Media Projection or screen recording interface in the Android.media.API is called to create a virtual screen Virtual Display, where the Virtual Display is a mirror copy of the physical screen of the electronic device in the memory. The Virtual Display saves the captured screen images to the data buffer. The Media Recorder integrates the video encoding and recording of the screen images in the data buffer to generate video clips.

[0056] See FIG. 2B , which is a schematic diagram of another application scenario of a virtual screen provided in an embodiment of the present application.

[0057] Regarding (b) in Figure 2A above, when the user performs a touch operation on the "Cast Screen" button 104, where the touch operation may include but is not limited to "single click, double click, or long press operation, etc.", the electronic device receives the user's touch on the "Cast Screen" button in the drop-down notification bar and casts the screen, obtaining the casting prompt interface 21 as shown in (a) in Figure 2B.

[0058] The prompt box prompts the screen projection service: This service requires WLAN and Bluetooth to be turned on for the wireless screen projection function. Is it turned on? In addition, the screen projection service also includes a "Cancel" button 211 and a "Turn on" button 212. The user can touch the "Turn on" button 212. The touch operation may include but is not limited to "single click, double click, or long press operation, etc.

[0059] After the electronic device receives the user's touch of the "On" button, it enters the screen projection device search interface 22 and searches for an available screen projection device 213. The user can click "ON" on the switch button to turn on the screen projection device, thereby projecting the display interface 11 in (a) of Figure 2A onto the display interface of the voting device 213. The display interface 23 of the screen projection device 213 shown in (c) is obtained.

[0060] The projection screen is the synthesized GI, and the specific synthesis method is as follows:

[0061] The screen casting app initiates the creation of a virtual screen. After the virtual screen is created, the image processing process is triggered to perform image synthesis on each frame. Specifically, the GI on the virtual screen is synthesized with the GI on the electronic device's display interface. The synthesized GI is sent to the electronic device's display interface for display and is displayed on the display interface 23 of the screen casting device 213(c).

[0062] In addition to the screen projection and screen recording application scenarios shown in Figures 2A and 2B, the virtual screen can also be applied to other scenarios. This application does not specifically limit the application scenarios of the virtual screen.

[0063] When recording on a virtual screen, the relevant schemes shown in Figures 3 and 4 can be used for GI synthesis.

[0064] See Figure 3, which shows an interactive diagram of a method for performing virtual screen synthesis in a related solution. The App in Figure 3 is located in the application layer of the electronic device software layer. The image processing process nodes involved are located in the application framework layer of the electronic device software layer.

[0065] The image processing process node is used to call the image processing process to implement the entire process of layer rendering, compositing, and display. The image processing process includes the image compositing thread, and the communication part of the image processing process that communicates with the app is located in the image compositing thread.

[0066] S11: The App creates a virtual screen.

[0067] S12: When the creation is successful, the App sends a start synthesis message to the image processing process node.

[0068] The start synthesis message is generated when the app successfully creates a virtual screen. It is used to enable the image processing process node to start the image processing process and perform layer rendering, synthesis, and display.

[0069] S13: The image processing process node is started.

[0070] After receiving the start synthesis message, the image processing process node starts the image processing process and calls the image synthesis thread to perform layer rendering, synthesis, and display.

[0071] S14: The image processing process node sends a memory acquisition message to the App.

[0072] The memory acquisition message refers to the image processing process obtaining the memory buffer from the app to cache the synthesized GI in the buffer. The buffer exists in the BufferQueue in the app.

[0073] S15: APP obtains memory.

[0074] After the APP receives the memory acquisition message sent by the image processing process node, it obtains the Buffer used to cache the synthesized GI from the Buffer Queue.

[0075] S16: The App sends memory parameters to the image processing process node.

[0076] The memory parameter is used to specify a buffer in the app's buffer queue, such as a slot parameter in a buffer. This allows the image processing node to cache the synthesized GI image into the buffer corresponding to the slot parameter.

[0077] S17: The image processing process node caches the synthesized GI in the buffer corresponding to the memory parameter.

[0078] The image processing process node caches the synthesized GI on the specified Buffer. At this time, the GI can be displayed on the virtual screen corresponding to the App. According to the functions of the application process, such as the screen projection function or the screen recording function, the corresponding operations of the application process can be performed, such as encoding the image corresponding to the screen projection function into a video format for storage.

[0079] S18: The image processing process node sends a return message to the App.

[0080] The return message is used to add the synthesized GI to the App's Buffer Queue, with the purpose of triggering the App to receive the synthesized GI and perform subsequent operations.

[0081] S19: The App returns a success message to the image processing process node.

[0082] When the App receives the return message sent by the image processing process, it modifies the memory state, generates a Buffer event object for the synthesized GI, adds it to the BufferQueue, and returns a return success message to the image processing process node.

[0083] In this way, GIs of other frames are synthesized according to steps S14 to S19 until the App initiates the destruction of the virtual screen.

[0084] S110: The App sends a destroy virtual screen message to the image processing process node.

[0085] When the user closes the executing App, or when the executing App is closed by other means, the App initiates the destruction of the virtual screen and sends a destroy virtual screen message to the image processing process node.

[0086] S111: The image processing process node destroys the virtual screen and the virtual screen object.

[0087] The virtual screen object refers to the thread that performs virtual screen synthesis in the image processing process node.

[0088] However, if the App does not respond in time, or if there is a communication anomaly between the App and the image processing process node, the image processing process will be blocked.

[0089] See FIG4 , which is a schematic diagram showing the principle of blocking the image processing process in a related solution.

[0090] S21: The App creates a virtual screen.

[0091] S22: When the creation is successful, the App sends a start synthesis message to the image processing process node.

[0092] S23: The image processing process node sends a memory acquisition message to the App.

[0093] S24: When the image processing process node does not receive a message sent by the App within a preset time period, the system service process node detects that the image processing process node has a communication anomaly.

[0094] For example, the preset duration is 1 second. If the image processing process node does not receive a message sent by the App within 1 second, the system service process node detects that the image processing process node has a communication anomaly.

[0095] S25: The system service process restarts the electronic device.

[0096] After the system service process detects a communication anomaly with the image processing process node, it restarts the electronic device and sends a restart instruction to the image processing process. The image processing process waits for the electronic device to restart before restarting.

[0097] In the above solution, the image processing process communicates directly with the app. If the app doesn't respond in a timely manner or encounters communication errors with the image processing process, the image processing process becomes blocked, causing the electronic device's display interface to freeze. In this frozen state, the user interface becomes unresponsive to user actions, resulting in a poor user experience.

[0098] In view of the above problems, embodiments of the present application provide a method for triggering a virtual screen. When an application process fails to respond in a timely manner or experiences communication anomalies with an image processing process, the image processing process's task thread is prevented from being blocked, thereby limiting the impact of the anomaly to the communication thread and unaffecting other task threads. Since the communication thread is only used for cross-process communication with the application process and is not used for GI synthesis on the electronic device's display interface, a blocked communication thread does not affect GI synthesis on the electronic device's display interface. GI synthesis on the electronic device's display interface is related to frozen screens; when the GI synthesis on the electronic device's display interface is normal, the display interface will not experience a frozen screen state. However, when the image processing process uses the communication portion of the image synthesis thread to communicate with the application process, if the application process fails to respond in a timely manner or experiences communication anomalies with the image processing process, the communication portion of the image processing process is blocked, and other threads of the image processing process are also blocked, resulting in the inability to synthesize the GI on the electronic device's display interface and the GI on the virtual screen. In summary, the present application utilizes the communication thread in the image processing process to communicate with the application process, effectively reducing the impact of a single application process anomaly on the underlying image processing process, to a certain extent resolving issues such as frozen screens and stuck displays on electronic devices, and improving device stability and user experience.

[0099] In order to better illustrate the virtual screen processing method provided by the present application, the electronic device 100 involved in the embodiment of the present application is first introduced.

[0100] In some embodiments, the electronic device 100 may be a wearable electronic device such as a wearable sports watch, wearable smart watch, wearable wristband, wearable audio headset, etc. that requires the creation of a virtual screen; it may also be a mobile phone, laptop computer, tablet computer, augmented reality (AR) device, virtual reality (VR) device, and in-vehicle device. It should be noted that the embodiments of the present application do not impose any particular limitation on the specific type of the electronic device 100.

[0101] See FIG5 , which is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0102] As shown in FIG5 , the electronic device 100 may include a processor 110 , an internal memory 120 , an antenna 1 , an antenna 2 , a mobile communication module 130 , a wireless communication module 140 , a display screen 160 , a touch sensor 180K, and the like.

[0103] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0104] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0105] For example, in the present application, when the image processing process is performing GI synthesis for the first frame, the processor 110 can communicate with the application process using a communication thread within the image processing process to obtain a communication result. If the communication result is normal, the synthesized GI is sent to the application process for display on the virtual screen created by the application process. If the communication result is abnormal, the virtual screen is destroyed. In this case, if the application process does not respond in a timely manner or if communication with the image processing process is abnormal, the communication thread is blocked, but this does not affect other threads within the image processing process. Because the communication thread is only used for cross-process communication with the application process and is not used for GI synthesis on the electronic device's display interface, a blocked communication thread does not affect GI synthesis on the electronic device's display interface. GI synthesis on the electronic device's display interface is related to frozen screens. When the GI synthesis on the electronic device's display interface is normal, the display interface will not freeze. However, when the image processing process communicates with the application process using the communication portion of the image synthesis thread, if the application process does not respond in a timely manner or if communication with the image processing process is abnormal, the communication portion is blocked, and thus neither the GI on the electronic device's display interface nor the GI on the virtual screen can be synthesized by the image processing process. In summary, this application solves the problem of frozen screens on electronic device displays to a certain extent by utilizing the communication thread within the image processing process to communicate with the application process, thereby improving the user experience. The controller can serve as the nerve center and command center of the electronic device. Based on the instruction opcode and timing signals, the controller can generate operation control signals to control instruction fetching and execution.

[0106] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0107] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0108] The wireless communication function of the electronic device can be implemented through the antenna 1, the antenna 2, the mobile communication module 130, the wireless communication module 140, the modem processor and the baseband processor.

[0109] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0110] The internal memory 120 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 120. The internal memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 120, and / or the instructions stored in the memory provided in the processor.

[0111] A series of GIs can be displayed on the display screen 160 of the electronic device, and these GIs are the main screen of the electronic device. Generally speaking, the size of the display screen 160 of the electronic device is fixed, and only a limited number of controls can be displayed on the display screen 160 of the electronic device. A control is a GI element, which is a software component included in an application. It controls all data processed by the application and interactive operations on this data. Users can interact with the control through direct manipulation to read or edit relevant information of the application. Generally speaking, controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0112] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0113] In addition, an operating system runs on the above components, such as the iOS operating system, the Android open source operating system, the Windows operating system, etc. Applications can be installed and run on this operating system.

[0114] The operating system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device.

[0115] 6 , which is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application. The present application embodiment takes the Android system with a layered architecture as an example to exemplify the software structure of the electronic device 100.

[0116] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: applications (Apps), framework (FWK), system libraries, and kernel.

[0117] The application layer can include a series of application packages. In some embodiments, the application layer can include applications such as screen recording apps, screen casting apps, and game apps.

[0118] The framework layer provides an application programming interface (API) and a programming framework for the application layer's applications. The framework layer may include some predefined functions. In some specific implementations, the framework layer may include a GI architecture and a display framework. Specifically, the framework corresponding to the virtual screen processing method provided in the embodiment of the present application is stored in the display framework of the application framework layer. The GI drawing thread is located in the GI architecture.

[0119] Optionally, an image processing process node is located in the display framework and is used to perform layer rendering, compositing, and display. The image processing process node is used to call the image processing process. The image processing process includes at least one thread. Optionally, the image processing process provided in embodiments of the present application includes an image synthesis thread, a communication thread, and a termination thread.

[0120] Optionally, the image synthesis thread is used to perform GI synthesis on the virtual screen and the GI synthesis of the electronic device display interface. The communication thread is used to communicate with the app and synchronize the obtained result signal to the image synthesis thread. The termination thread is used to terminate the communication thread.

[0121] Optionally, the application framework layer includes a window manager, a multimedia manager, a view system, etc.

[0122] The window manager is used to manage window programs. It can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, and send interface information display instructions to the view system.

[0123] The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build apps. A display interface can consist of one or more views. The view system can also invoke the display driver of the corresponding electronic device's display screen based on display interface display instructions sent by the window manager, and use the display driver to drive the first display screen to display interface information.

[0124] Multimedia Manager, used to manage multimedia calling interfaces. The calling interfaces include but are not limited to: Media Recorder, which is a multimedia creation interface, and Media Projection, which is a screen recording interface. The Media Recorder in the Multimedia Manager is used to create a data buffer. Media Projection is used to create a Virtual Display. A Virtual Display is a virtual reality display in the memory of an electronic device's display screen, i.e., a virtual screen, used to mirror the display screen. The Virtual Display saves the captured screen image to the data buffer, and the Media Recorder then generates a video clip from the screen image in the data buffer.

[0125] The system library may include multiple functional modules. In some specific implementations, the system library layer includes functional modules such as SurfaceFlinger, communication thread, and termination thread.

[0126] SurfaceFlinger is used for layer creation, control, and management. The communication thread is used to communicate with the app, and the termination thread is used to terminate the communication thread.

[0127] The kernel layer is responsible for managing the system's threads, memory, device drivers, files, and network systems, and determines the system's performance and stability. In some specific implementations, the kernel layer hardware driver layer, such as display driver, GPU driver, HWC driver, and CPU driver.

[0128] It should be noted that although the embodiment of the present application is described using the Android system as an example, its basic principles are also applicable to electronic devices 100 based on operating systems such as iOS and Windows.

[0129] 7 , a flowchart of a method for processing a virtual screen according to an embodiment of the present application is provided.

[0130] As shown in FIG7 , the method includes an image processing process and an App, wherein the image processing process includes a communication thread. The method specifically includes:

[0131] S31: When the image processing process performs GI synthesis of the first frame, the communication thread in the image processing process communicates with the App to obtain a communication result.

[0132] In the embodiment of the present application, the first frame is the current frame, and the image processing process synthesizes the GI of the first frame, that is, the image processing process synthesizes the GI of the current frame.

[0133] Optionally, the image processing process is a SurfaceFlinger process.

[0134] The communication thread is the thread within the image processing process that communicates with the app. This embodiment of the present application creates an independent communication thread for cross-process communication with the app. If the app doesn't respond in a timely manner or cross-process communication is blocked, only the communication thread within the image processing process is blocked, while other threads, such as the electronic device's display image synthesis thread, remain unblocked. This prevents the display interface of the electronic device from freezing due to a blocked image synthesis thread.

[0135] Optionally, the communication thread can be created simultaneously in the image processing process when the app creates a virtual screen. Furthermore, when the app creates a virtual screen, it also creates a virtual screen object in the image processing process. The virtual screen object is used to maintain the correspondence between the virtual screen and the communication thread. This way, when the app's functionality does not involve a virtual screen, it is not created, which saves memory to a certain extent and improves GI synthesis efficiency.

[0136] Optionally, the Binder mechanism can be used for cross-process communication between the app and the communication thread. Compared to other cross-process transmission mechanisms, the Binder mechanism only requires one copy, is faster, and is more secure and easy to use.

[0137] S32: Determine whether the communication result is normal communication. If the communication is abnormal, execute S33; if the communication is normal, execute S34.

[0138] In the embodiment of the present application, the communication result refers to the cross-process communication between the App and the communication thread, including normal communication and abnormal communication.

[0139] Normal communication means that the App and the communication thread are communicating normally. In this embodiment, normal communication means that the image processing process receives the message sent by the App within a preset time. For example, the image processing process receives the memory acquisition success message sent by the App within 1 second, or receives the return success message within 1 second.

[0140] A communication anomaly refers to abnormal communication between the app and the communication thread. In this embodiment of the present application, a communication anomaly refers to the image processing process not receiving a message from the app within a preset time. For example, the image processing process does not receive any messages from the app within 1 second. In this embodiment of the present application, a communication anomaly refers to an untimely response from the app or abnormal communication between the app and the image processing process.

[0141] If the image processing process does not receive a message from the App within the preset time, it indicates a communication anomaly. Otherwise, it indicates normal communication.

[0142] S33: The image processing process deletes the virtual screen object.

[0143] The virtual screen object refers to threads and other contents related to virtual screen synthesis that exist in the image processing process, such as the GI synthesis part on the virtual screen in the image synthesis thread, the communication thread, etc.

[0144] When communication is abnormal, cross-process communication blocks the communication thread. At this time, the GI synthesis part on the virtual screen cannot be executed normally. In order to achieve normal execution of GI synthesis of the electronic device display interface, the virtual screen object is deleted, such as deleting the GI synthesis part on the virtual screen in the image synthesis thread, and / or the communication thread, etc.

[0145] S34: The image processing process sends the synthesized GI to the App for display on the virtual screen.

[0146] When the communication is normal, the image processing process sends the synthesized GI to the App for display on the virtual screen of the application process.

[0147] Optionally, when the application process is a screen recording process, the synthesized GI is sent to the application process, so that the application process encodes the synthesized GI into a video format for storage.

[0148] Optionally, when the application process is a screen projection process, the synthesized GI is sent to the application process, so that the application process sends the synthesized GI to the screen projection device for display.

[0149] Embodiments of the present application provide a method for processing a virtual screen. When an application process fails to respond in a timely manner or experiences communication anomalies with an image processing process, the image processing process's task thread is prevented from being blocked, limiting the impact of the anomaly to the communication thread and leaving other task threads unaffected. Because the communication thread is only used for cross-process communication with the application process and is not used for GI synthesis on the electronic device's display interface, a blocked communication thread does not affect GI synthesis on the electronic device's display interface. GI synthesis on the electronic device's display interface is related to frozen screens; when the GI synthesis on the electronic device's display interface is normal, the display interface will not freeze. However, when the image processing process communicates with the application process using the communication portion of the image synthesis thread, if the application process fails to respond in a timely manner or experiences communication anomalies with the image processing process, the communication portion of the image processing process becomes blocked, blocking other threads in the image processing process as well. This results in the inability to synthesize the GI on the electronic device's display interface and the GI on the virtual screen. In summary, the present application utilizes the communication thread in the image processing process to communicate with the application process, effectively reducing the impact of a single application process anomaly on the underlying image processing process, addressing issues such as frozen screens and stuck displays on electronic devices to a certain extent, and improving device stability and user experience.

[0150] See Figure 8, which is an interactive diagram of a method for virtual screen synthesis provided in an embodiment of the present application. The method is illustrated by taking the SurfaceFlinger process as an example of an image processing process. The SurfaceFlinger process node involved in the method is used to implement the SurfaceFlinger process. The SurfaceFlinger process includes an image synthesis thread, a communication thread, and a termination thread. The image synthesis thread is used to implement GI synthesis of the electronic device display interface and GI synthesis on the virtual screen.

[0151] When the app creates a virtual screen, it creates a communication thread in the SurfaceFlinger process, creates a virtual screen object, and then performs the following operations.

[0152] In an embodiment of the present application, the communication thread is used to implement communication between the SurfaceFlinger process and the App, and includes at least a first sub-thread and a second sub-thread.

[0153] The first sub-thread is used to obtain memory parameters from the App, and the second sub-thread is used to return the memory parameters to the App.

[0154] S41: The image synthesis thread in the SurfaceFlinger process node sends a first event signal to the communication thread.

[0155] The first event signal is a signal that triggers the communication thread to specify memory from the App for caching the synthesized GI. In the embodiment of the present application, the first event signal is a signal obtained when the SurfaceFlinger process starts performing a frame synthesis, which is used to trigger the communication thread to determine the specified memory from the App, that is, the target buffer. For example, the first event signal is a signal that triggers the communication thread to call DequeueBuffer().

[0156] To ensure that the target buffer specified by the communication thread meets the memory size required for virtual screen synthesis, the first event signal can pre-set the width, height, and pixel format of the target buffer. The app obtains the corresponding target buffer based on the pre-set width, height, and pixel format.

[0157] Since the communication thread includes multiple sub-threads, in order to accurately call the first sub-thread of the communication thread, the first event signal may carry a first tag.

[0158] The first tag is a pre-set identifier, which can be a letter, number, or other identifier. The first tag corresponds to the first child thread. When the first event signal carries the first tag, the communication thread calls the first child thread and determines the target buffer from the App. For example, the first tag is DequeueBuffer, which is used to trigger the communication thread to call the first child thread DequeueBuffer() and determine the target buffer from the App.

[0159] S42: The communication thread sends a memory acquisition message to the App.

[0160] After receiving the first event signal from the image synthesis thread, the communication thread sends a memory acquisition message to the App. The memory acquisition message is used to specify the target buffer from the App's Buffer Queue.

[0161] The memory acquisition message includes at least the App identity and the memory acquisition instruction.

[0162] The app identity uniquely identifies the corresponding app. Because the software layer of an electronic device includes many apps, the communication thread can find the required app from multiple apps based on the app identity.

[0163] The memory acquisition instruction is used to trigger the app to determine the target buffer from the Buffer Queue. Optionally, the memory acquisition instruction can be a call to DequeueBuffer() to release the target buffer from the app's Buffer Queue.

[0164] DequeueBuffer() specifies the order in which memory is retrieved, prioritizing the target buffer from the slot bound to the memory. This improves the efficiency of the determination.

[0165] To ensure that the target buffer determined by the communication thread meets the memory requirements for virtual screen composition, the memory acquisition instruction can also carry the width, height, and pixel format specified by the first event signal. The app can then determine the target buffer from the Buffer Queue based on the width, height, and pixel format specified by the first event signal.

[0166] Alternatively, the communication thread can send memory retrieval messages to the app using the Binder mechanism. Compared to other transmission mechanisms, the Binder mechanism only requires a single copy, making it faster, more secure, and easier to use.

[0167] S43: The App obtains the target buffer.

[0168] The app determines the target buffer from the Buffer Queue based on the memory acquisition instruction in the received memory acquisition message.

[0169] When the memory access instruction is to call DequeueBuffer(), a slot is first obtained from mFreeBuffers and the corresponding Bufferslot status is changed from FREE to DEQUEUED. Since the slot in mFreeBuffers is already connected to the buffer, there is no need to allocate a new buffer for the slot, which improves the access speed.

[0170] In addition, if mFreeBuffers is empty, a slot is obtained from mFreeSlots, and a Buffer is allocated to it, and the Bufferslot state is changed from FREE to DEQUEUED.

[0171] The above mFreeSlots is a slot set whose BufferSlot structure is FREE and has no Buffer bound to it. mFreeBuffers is a slot set whose BufferSlot structure is FREE and has a Buffer bound to it.

[0172] When the memory acquisition instruction carries the width, height, and pixel format specified by the first event signal, the target buffer corresponding to the requirement is determined from the Buffer Queue.

[0173] S44: The App sends the memory parameters corresponding to the target buffer to the communication thread.

[0174] Memory parameters are used to specify the target buffer. That is, memory parameters correspond one-to-one with the buffers in the Buffer Queue, and different buffers correspond to different memory parameters.

[0175] In this embodiment of the present application, the App sends the memory parameters corresponding to the target Buffer to the communication thread, so that the communication thread synchronizes the memory parameters to the image synthesis thread.

[0176] Optionally, when the memory acquisition instruction is DequeueBuffer(), the app assigns the acquired slot as an output parameter to the communication thread. The slot is bound to the specified buffer.

[0177] Optionally, if the target buffer bound to the slot is reallocated, the communication thread receives the return value sent by the app and returns the status BUFFER_NEEDS_REALLOCATION. Otherwise, the communication thread returns the status NO_ERROR. This allows the communication thread to determine subsequent processing based on the return value.

[0178] Among them, if the return value is BUFFER_NEEDS_REALLOCATION status, the communication thread needs to send a new pointer acquisition message to the App, such as sending RequestBuffer, obtain the target Buffer pointer from the App, and then perform subsequent operations.

[0179] If the return value is No_ERROR, the communication thread directly performs subsequent operations.

[0180] Optionally, the App sends messages to the communication thread through the Binder mechanism, including memory acquisition messages, memory parameters, and / or new pointer acquisition messages.

[0181] S45: The communication thread sends the first result signal to SurfaceFlinger.

[0182] In this embodiment of the present application, the first result signal refers to the result signal triggered after the communication thread receives the memory parameters sent by the App, which is used to indicate that the SurfaceFlinger process can perform the synthesis operation on the target Buffer specified by the BufferQueue. The first result signal includes the memory parameters.

[0183] Optionally, the first result signal may include the width, height, and pixel format of the target buffer, so that the image synthesis thread determines whether synthesis can be completed based on the width, height, and pixel format of the buffer in the first result signal. It should be understood that when the width, height, and pixel format of the buffer do not meet the requirements, such as the width is lower than the width set by the first event signal, the height is lower than the width set by the first event signal, or the pixel format is inconsistent with the pixel format set by the first event signal, the SurfaceFlinger process cannot synthesize the synthesis result on the target buffer.

[0184] Furthermore, to composite the SurfaceFlinger virtual screen onto the specified target buffer, the first result signal includes a buffer pointer. The buffer pointer uniquely identifies the target buffer. At this point, the SurfaceFlinger process finds the target buffer based on the buffer pointer and composites the GI on the virtual screen onto the target buffer.

[0185] S46: The image synthesis thread performs GI synthesis on the target Buffer.

[0186] After receiving the first result signal sent by the communication thread, the image synthesis thread performs layer synthesis and synthesizes the synthesis result on the specified target Buffer.

[0187] The image synthesis thread includes hardware compositor (HWC) synthesis and / or GPU synthesis.

[0188] A HWC is a dedicated image processing device used for layer compositing and display. It can be a standalone device or integrated into a system-on-chip. The layer compositing method that uses the HWC to synthesize layer display data is called HWC compositing.

[0189] The GPU is a general-purpose image processing device. Besides compositing layers, it also performs other graphics processing tasks. The layer compositing method that uses the GPU to composite layer display data is called GPU compositing.

[0190] During actual use, the SurfaceFlinger process sends the layer parameters of the rendered layer to the HWC, and the HWC marks the rendered layer according to its own processing capabilities. When the number of rendered layers is greater than the maximum number of layers that the HWC can synthesize, some are marked as GPU synthesis and some are marked as HWC synthesis. The electronic device first calls the GPU to synthesize the layers marked as GPU synthesis to obtain a GPU synthesis layer, and then synthesizes the GPU synthesis layer and the layer marked as HWC synthesis to obtain an HWC synthesis layer, and synthesizes the HWC synthesis layer on the target buffer to complete a frame of synthesis processing.

[0191] Optionally, when the first result signal includes the width, height, and pixel format of the target buffer, SurfaceFlinger can first determine whether the obtained buffer meets the synthesis requirements based on the first result signal. If so, SurfaceFlinger performs synthesis. If not, SurfaceFlinger re-executes steps S41 to S46 until the synthesis requirements are met, and then executes S47.

[0192] S47: The image synthesis thread sends a second event signal to the communication thread.

[0193] The second event signal is an event signal for returning memory parameters to the app. In this embodiment of the present application, the signal obtained after the SurfaceFlinger process completes a frame synthesis is used to trigger the communication thread to return the memory parameters to the app. Exemplarily, the second event signal is a call to QueueBuffer().

[0194] Since the communication thread includes multiple sub-threads, in order to accurately call the second sub-thread of the communication thread, the second event signal may carry a second tag.

[0195] The second tag is a pre-set identifier, which can be a letter, number, or other identifier. The second tag corresponds to the second sub-thread. When the second event signal carries the second tag, the communication thread calls the second sub-thread to return the memory parameters to the App. For example, the second tag is QueueBuffer, which is used to trigger the communication thread to call the second sub-thread QueueBuffer() to return the memory parameters to the App's Buffer Queue.

[0196] S48: The communication thread sends a return message to the App.

[0197] The return message includes the App identity, the return instruction, and the second buffer.

[0198] The app identity uniquely identifies the corresponding app. Because the software layer of an electronic device includes many apps, the communication thread can find the required app from multiple apps based on the app identity.

[0199] The return instruction is used to trigger the App to perform subsequent operations.

[0200] Optionally, the second child thread is QueueBuffer(), and the memory can be returned to the App through QueueBuffer(). The specific implementation of QueueBuffer() is:

[0201] QueueBuffer() changes the Bufferslot state corresponding to the slot from DEQUEUED to QUEUED according to the slot parameter corresponding to the Buffer, generates a Buffer event object based on the Bufferslot information, and adds it to the Buffer Queue.

[0202] Optionally, the communication thread sends a return message to the App through the Binder mechanism.

[0203] S49: The App sends a return success message to the communication thread.

[0204] When the App receives the memory parameters corresponding to the target Buffer returned by the communication thread, it sends a return success message to the communication thread.

[0205] The return success message is used to indicate that the memory parameters corresponding to the target buffer are returned successfully. It can be a letter, a number, or other mark that can identify a successful return. For example, the return success message is Queue Buffer Success.

[0206] Optionally, the App sends a return success instruction to the communication thread through the Binder mechanism

[0207] S410: The communication thread sends the second result signal to the image synthesis thread.

[0208] In an embodiment of the present application, the second result signal refers to a result signal triggered after the communication thread receives a return success message sent by the App, which is used to notify the image synthesis thread that the memory parameters of the target buffer have been successfully returned.

[0209] S411: Determine whether the communication thread has terminated. If so, execute S41. If not, execute S412.

[0210] S412: After the App initiates the destruction of the virtual screen, the App sends a destroy virtual screen instruction to the image synthesis thread of the SurfaceFlinger process node.

[0211] The destroy virtual screen instruction is used to cause the image synthesis thread process in SurfaceFlinger to destroy the virtual screen object, and SurfaceFlinger does not perform the virtual screen synthesis operation.

[0212] Optionally, when the app terminates, it initiates the destruction of the virtual screen. The app termination operation can be closing the app. For example, the app automatically terminates when the running time reaches a preset length, clicks to close the space, or automatically closes the app through language, gestures, etc.

[0213] Optionally, the system service process detects that the application process has failed, that is, closes the App, sends a close instruction to the application process, and closes the application process.

[0214] S413: The image synthesis thread sends a termination instruction to the communication thread.

[0215] After receiving the app's instruction to destroy the virtual screen object, the image compositing thread sends a terminate instruction to the communication thread. This terminates the instruction and causes the image compositing thread to destroy the virtual screen object, terminating the communication thread and deleting the virtual screen object. At this point, the communication thread no longer occupies the device's memory and will not affect the next virtual screen composition.

[0216] S414: The communication thread sends an instruction to destroy the virtual screen object to the image synthesis thread.

[0217] S415: The image synthesis thread destroys the virtual screen object.

[0218] When the App does not respond to SurfaceFlinger's communication request in a timely manner, or when a communication anomaly occurs, the interaction method shown in FIG9 is executed.

[0219] See FIG. 9 , which is an interactive diagram of a virtual screen synthesis failure provided by an embodiment of the present application.

[0220] The SurfaceFlinger process synthesizes the first frame.

[0221] S51: The image synthesis thread sends a first event signal to the communication thread.

[0222] S52: The communication thread sends memory acquisition information to the App.

[0223] The process is the same as steps S31 to S32 and will not be described again here.

[0224] S53: If the image synthesis thread does not receive the first result signal sent by the communication thread within the preset time period, the communication thread is terminated.

[0225] In an embodiment of the present application, because the App did not respond to the communication request sent by the communication thread in a timely manner, or the communication with the communication thread was abnormal, etc., the image synthesis thread did not receive the first result signal sent by the communication thread within the preset time. At this time, it is determined that the current virtual screen is invalid, and the graphics synthesis thread creates a termination thread to terminate the communication thread, that is, the communication thread is no longer called.

[0226] The preset time duration is a time duration set by those skilled in the art based on experience, for example, 1 second. The image synthesis thread does not receive the first result signal sent by the communication thread within 1 second.

[0227] SurfaceFlinger directly synthesizes the second frame, which is the next frame after the first frame.

[0228] At this time, although the virtual screen has failed, the image synthesis thread still performs the GI synthesis operation of the electronic device display interface. In this way, the display interface of the electronic device can be prevented from being still due to the blocking of the image synthesis thread, which provides the user with a certain degree of visual experience.

[0229] S54: SurfaceFlinger determines whether the communication thread is terminated. If terminated, execute S55.

[0230] In an embodiment of the present application, SurfaceFlinger monitors the running status of the communication thread. When it is detected that the communication thread is not running, for example, when it is detected that the running status of the communication thread is FALSE, the communication thread is terminated.

[0231] If the communication thread is still running, it has not terminated. SurfaceFlinger will continue compositing the second frame. When the second frame is finished, it will compose the third frame. While compositing the third frame, it will continue to check whether the communication thread has terminated. If not, it will continue compositing the fourth frame and again checking whether the communication thread has terminated. This cycle repeats until the communication thread terminates.

[0232] In this embodiment of the present application, the communication thread termination condition is as follows: the system server detects that the app has expired and sends a process termination instruction to the app. The app receives the process termination instruction and sends a memory acquisition instruction failure instruction to the communication thread. Upon receiving the memory acquisition instruction failure instruction, the communication thread terminates.

[0233] S55: The SurfaceFlinger process deletes the virtual screen object.

[0234] The virtual screen synthesis failure provided in the embodiment of the present application occurs when the App fails to respond to the communication request sent by the communication thread in a timely manner, or when there is an abnormality in the communication with the communication thread. Although the virtual screen has failed, SurfaceFlinger still performs the synthesis operation. In this way, the display interface of the electronic device can be prevented from being frozen due to SurfaceFlinger being blocked, and the user's visual experience can be improved to a certain extent.

[0235] The technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (such as a personal computer, server, or network device) or a processor to execute all or part of the steps of the method described in each embodiment. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, a mobile hard disk, read-only memory, random access memory, a magnetic disk, or an optical disk.

[0236] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for processing a virtual screen, characterized in that: The method comprises: When the image processing process performs the GI synthesis of the image interface of the first frame, the communication thread in the image processing process is used to communicate with the application process to obtain a communication result, wherein the communication result includes normal communication and abnormal communication, and the normal communication means that the image processing process receives a message sent by the application process within a preset time; When the communication result is that the communication is abnormal, the virtual screen object in the image processing process is deleted; when the communication result is that the communication is normal, the synthesized GI is sent to the virtual screen created by the application process.

2. The method according to claim 1, characterized in that: The utilizing the communication thread in the image processing process to communicate with the application process to obtain a communication result includes: When the image synthesis thread in the image processing process sends a first event signal to the communication thread, the communication thread is used to send a memory acquisition message to the application process, wherein the memory acquisition message is used to determine a memory, and the memory is used to cache the synthesized GI; When the communication thread receives the memory parameter sent by the application process within the preset time and sends a first result signal to the image synthesis thread, it is determined that the communication result is that the communication is normal, and the memory parameter is used to specify the memory; When the communication thread does not receive the memory parameter sent by the application process within the preset time, it is determined that the communication result is a communication abnormality.

3. The method according to claim 2, characterized in that: When the first event signal is an event signal marked as DequeueBuffer, the memory acquisition message is DequeueBuffer().

4. The method according to claim 2 or 3, characterized in that: The memory acquisition message also includes memory size and pixel format.

5. The method according to any one of claims 2 to 4, characterized in that: When the communication result is that the communication is normal, sending the synthesized GI to the application process includes: When the communication result is that the communication is normal, the synthesized GI is cached in the memory, and a second event signal is sent to the communication thread; In response to the communication thread receiving the second event signal, using the communication thread to send a return message to the application process; When the communication thread receives the return success message sent by the application process and sends the second result signal to the image synthesis thread, it is determined to cache the synthesized GI into the memory.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: After the image processing process synthesizes the GI and it is determined that the communication thread is terminated, the virtual screen object is destroyed.

7. The method according to claim 5 or 6, characterized in that: The second event signal is an event signal marked as QueueBuffer, and the return message is QueueBuffer().

8. The method according to any one of claims 2 to 7, characterized in that: When the communication result is the communication abnormality, deleting the virtual screen object in the image processing process includes: When the communication is abnormal, a termination thread is constructed and the GI synthesis of the second frame is entered. The termination thread is used to terminate The communication thread, the second frame is the next frame of the first frame; Determine whether the communication thread is terminated; when it is determined that the communication thread is not terminated, enter the GI synthesis of the third frame, and determine again whether the communication thread is terminated, the third frame is the next frame of the second frame, and the cycle is repeated until it is determined that the communication thread is terminated, and the virtual screen object is deleted.

9. The method according to claim 8, characterized in that: The determining that the communication thread is terminated includes: When the system service process detects that the application process has failed and sends a close instruction to the application process to close the application process, the communication thread receives the failure instruction of the memory acquisition message sent by the application process to determine that the communication thread is terminated.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: When the application process creates the virtual screen, the communication thread is created in the image processing process, and the virtual screen object is created to maintain the corresponding relationship between the virtual screen and the communication thread.

11. The method according to claim 10, characterized in that: The destroying of the virtual screen comprises: The communication thread is terminated, and the virtual screen object is destroyed.

12. The method according to any one of claims 1 to 11, characterized in that: After sending the synthesized GI to the application process, the method further includes: When the application process is a screen recording process, sending the synthesized GI to the application process, so that the application process encodes the synthesized GI into a video format for storage; When the application process is a screen projection process, the synthesized GI is sent to the application process, so that the application process sends the synthesized GI to the screen projection device for display.

13. The method according to any one of claims 1 to 12, characterized in that: The utilizing the communication thread in the image processing process to communicate with the application process comprises: The communication thread is utilized to communicate with the application process based on the Binder mechanism.

14. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is coupled to the processor; The memory stores a program, and when the program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 13.

Images