Virtual screen processing method and electronic equipment

Abstract

The invention provides a virtual screen processing method and electronic equipment, and is applied to the technical field of terminals. The method comprises the following steps: when an image processing process performs GI synthesis of a first frame, using a communication thread in the image processing process to communicate with an application process to obtain a communication result. And when the communication result is normal, sending the synthesized GI to a virtual screen created by the application process. And when the communication result is abnormal, deleting the virtual screen object. At the moment, when the application process does not respond in time or communication with the image processing process is abnormal, the task thread of the image processing process is prevented from being blocked, so that the abnormal influence is controlled in the communication thread, and other task threads are not influenced. Thus, the communication thread is split from the image synthesis thread, the communication thread is used for communicating with the application process, the frozen screen problem of the display interface of the electronic equipment is solved to a certain extent, and the equipment stability and the user experience are improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a virtual screen processing method and electronic device. Background Art

[0002] When an electronic device implements functions such as screen recording or screen projection, an application (Application, App) is required to create a virtual screen, and after the creation is completed, the graphic interface (Graphic Interface, GI) on the virtual screen is synthesized and sent to the virtual screen for display. Among them, App can also be called an application process.

[0003] Related solutions use the image synthesis thread in the image processing process, such as the image synthesis thread of the SurfaceFlinger process, to synthesize the GI on the virtual screen and the GI on the display interface of the electronic device at the same time. However, the display interface of the electronic device often freezes. In the frozen state, the display interface does not respond to the user's trigger operation, which makes the user experience poor. Summary of the invention

[0004] The present application provides a virtual screen processing method and electronic device, which aims to solve the problem of frozen screen on the display interface of the electronic device to a certain extent and improve the user experience.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a virtual screen processing method, which specifically includes:

[0007] When the image processing process performs GI synthesis of the first frame, the communication thread in the image processing process is used to communicate with the application process to obtain the communication result. When the communication result is normal, the synthesized GI is sent to the application process to be displayed on the virtual screen created by the application process. When the communication result is abnormal, the virtual screen object is deleted. At this time, when the application process does not respond in time, or the communication with the image processing process is abnormal, the task thread of the image processing process is prevented from being blocked, so that the impact of the abnormality is controlled in the communication thread, and other task threads are not affected. Since the communication thread is only used for cross-process communication with the application process, it is not used for GI synthesis of the display interface of the electronic device. Therefore, the blocking of the communication thread will not affect the GI synthesis of the display interface of the electronic device, and the GI synthesis of the display interface of the electronic device is related to the frozen screen of the display interface. When the GI of the display interface of the electronic device is synthesized normally, the display interface will not appear in a frozen screen state. When the image processing process uses the communication part of the image synthesis thread to communicate with the application process, when the application process does not respond in time, or the communication with the image processing process is abnormal, the communication part of the image processing process is blocked, and other threads of the image processing process are blocked, resulting in the inability to synthesize the GI of the electronic device display interface and the GI on the virtual screen. In summary, this application uses the communication thread in the image processing process to communicate with the application process, effectively reducing the impact of a single application process abnormality on the underlying image processing process, and to a certain extent solves the problem of frozen screen and stuck of the electronic device display interface, and improves the device stability and user experience.

[0008] In a possible implementation, when the image synthesis thread in the image processing process sends a first event signal to the communication thread, the communication thread is used to send a memory acquisition message to the application process, and the memory acquisition message is used to determine the memory of the cached synthesized GI; when the communication thread receives the memory parameters sent by the application process within a preset time and sends the first result signal to the image synthesis thread, it is determined that the communication result is normal, and the memory parameters are used to specify the memory; when the communication thread does not receive the memory parameters sent by the application process within the preset time, it is determined that the communication result is abnormal. In this way, the image synthesis thread is separated from the communication thread, avoiding the blocking of the communication thread causing the blocking of the image synthesis thread, so that the GI of the electronic device display interface cannot be synthesized and the screen freezes.

[0009] In another possible implementation, when the first event signal is an event signal marked as DequeueBuffer, the memory acquisition message is DequeueBuffer(). In this way, DequeueBuffer() specifies the order of memory acquisition, and preferentially determines the target GraphicBuffer (hereinafter referred to as Buffer) from the slot bound to the memory. In this way, the determination efficiency can be improved.

[0010] Optionally, the memory acquisition message also includes but is not limited to memory size and pixel format information. The App can determine the target Buffer from the Buffer Queue based on the width, height and pixel format specified by the first event signal. In another achievable method, when the communication result is that the communication is normal, the synthesized GI is cached in the memory, and a second event signal is sent to the communication thread; in response to the communication thread receiving the second event signal, a return message is sent to the application process using the communication thread; when the communication thread receives the return success instruction sent by the application process and sends the second result signal to the image synthesis thread, it is determined that the synthesized GI is cached in the memory.

[0011] Optionally, after the image processing process synthesizes the GI and it is determined that the communication thread is terminated, the virtual screen object is destroyed.

[0012] Optionally, the second event signal is an event signal marked as QueueBuffer, and the return message is QueueBuffer().

[0013] Optionally, when communication is abnormal, a termination thread is constructed and the GI synthesis of the second frame is entered. The termination thread is used to terminate the communication thread, and the second frame is the next frame of the first frame; determine whether the communication thread is terminated; when it is determined that the communication thread is not terminated, enter the GI synthesis of the third frame, and again determine whether the communication thread is terminated, and the third frame is the next frame of the second frame, and cycle in sequence until it is determined that the communication thread is terminated and the virtual screen object is deleted. In this way, when communication is abnormal or the App does not respond in time, the electronic device display interface can perform GI synthesis normally.

[0014] Optionally, when the system service process detects that the application process has failed and sends a close instruction to the application process to close the application process, the communication thread receives a failure instruction of the memory acquisition instruction sent by the application process to determine that the communication thread is terminated.

[0015] Optionally, when the application process creates a virtual screen, a communication thread is created in the image processing process, and a virtual screen object is created to maintain a corresponding relationship between the virtual screen and the communication thread.

[0016] Optionally, the communication thread is terminated and the virtual screen object is destroyed. In this way, when the virtual screen is needed, it is created, and when it is not needed, it is deleted, which can prevent the virtual screen from affecting other functions.

[0017] Optionally, when the application process is a screen recording process, the synthesized GI is sent to the application process so that the application process encodes the synthesized GI into a video format for storage; when the application process is a screen projection process, the synthesized GI is sent to the application process so that the application process sends the synthesized GI to the screen projection device for display.

[0018] Optionally, the communication thread is used to communicate with the application process based on the Binder mechanism. Compared with other transmission mechanisms, the Binder mechanism only needs to be copied once, which is faster, safer and easier to use.

[0019] A second aspect provides an electronic device, the electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; the memory stores a program, and when the program is executed by the processor, the electronic device executes any method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the internal structure of BufferQueueCore;

[0021] Figure 2A A schematic diagram of an application scenario of a virtual screen provided in an embodiment of the present application;

[0022] Figure 2B A schematic diagram of another application scenario of a virtual screen provided in an embodiment of the present application;

[0023] Figure 3 Interactive diagram of methods for virtual screen synthesis for related schemes;

[0024] Figure 4 A schematic diagram of the principle of blocking the image processing process for the related scheme;

[0025] Figure 5 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 7 A flowchart of a method for processing a virtual screen is provided for an embodiment of the present application;

[0028] Figure 8 An interactive diagram of a method for synthesizing a virtual screen provided in an embodiment of the present application;

[0029] Fig. 9 An interactive diagram of a virtual screen synthesis failure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0031] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0032] The following is an introduction to the professional terms involved in the embodiments of the present application.

[0033] BufferQueue: It consists of three parts: BufferQueueCore, which is responsible for maintaining the basic data structure of BufferQueue; BufferQueueProducer and BufferQueueConsumer, which are used to provide the basic interface for operating BufferQueue.

[0034] See also Figure 1 , is a schematic diagram of the internal structure of BufferQueueCore. BufferQueueCore includes five queues, namely mQueue, mSlots, mFreeSlots, mFreeBuffers and mActiveBuffers.

[0035] Among them, mQueue is used to store the First Input First Output (FIFO) queue of BufferItem.

[0036] mSlots is the BufferSlot structure diagram data, and the array length is 64, that is, from slot0 to slot64. The following uses the slot parameter to replace the BufferSlot array.

[0037] The slot state of mFreeSlots is FREE, and no Buffer is bound to it, that is, the slot state is FREE and NULL. The FREE state means that the current owner of this BufferSlot is BufferQueue, and the current state can be taken out of the queue by BufferQueueProducer calling the Dequeue method. The NULL state means that no Buffer is bound to the slot.

[0038] The slot status of mFreeBuffers is FREE, and a Buffer is bound to it. That is, the slot status is FREE and the buffer status, where the buffer status indicates that a Buffer is bound to the slot.

[0039] The slot state of mActiveBuffers is NON-FREE state. NON-FREE state is at least one of DEQUEUED, QUEUED, ACQUIRED and SHARED.

[0040] The DEQUEUED state means that the current slot is dequeued from the queue by the BufferQueueProducer, but the BufferQueueProducer still cannot fill or modify the Buffer content until the last consumer using the Buffer sends a Fence signal to release the Buffer. At this time, the Buffer belongs to the Buffer Queue Producer.

[0041] The QUEUED state means that the Buffer is put into the queue by BufferQueueProducer calling the Queue method and belongs to BufferQueue. At this time, the Buffer described by this slot can be taken away by BufferQueueConsumer.

[0042] ACQUIRED: After the Buffer is taken from the queue by BufferQueueConsumer calling the Acquire method, the state changes to the Acquired state. At this time, BufferQueueConsumer cannot access the contents of the Buffer until it receives the relevant Fence signal.

[0043] Unused Slots is a collection of slots that are not used.

[0044] Binder mechanism: a cross-process communication mode based on the client and server. The server will register with the service manager, and the client can achieve inter-process communication with the server by obtaining services.

[0045] Thread is the most basic running unit of a program. Thread does not exist independently, but is contained in a process. It is the smallest unit participating in system scheduling and the actual running unit in a process. When the application is started, the system creates a process, but the process cannot run. What can run is the thread in the process. A process is just a container, which contains the data structure, environment variables and other information required for the thread to run. Multiple threads in the same process will share all system resources in the process, such as virtual address space, file descriptors and signal processing, etc. However, multiple threads in the same process have their own call stacks, register contexts and thread-local storage. A thread refers to a single sequential control flow (i.e., execution route, execution flow) in a process. Multiple threads can be created in a process to realize concurrent operation of multiple threads, and each thread performs different tasks. For example, an application is designed with two tasks that need to run concurrently, namely, task one and task two. Task one and task two can be placed in two threads respectively to realize concurrent operation of the two tasks.

[0046] The virtual screen, also known as the virtual screen, corresponds to the display screen of the electronic device and is used for virtual display of the GI, mainly for screen recording, or screen projection, etc. In the embodiment of the present application, the virtual screen is created and destroyed by the App.

[0047] See also Figure 2A , is a schematic diagram of an application scenario of a virtual screen provided in an embodiment of the present application.

[0048] like Figure 2A As shown in (a) of FIG. 1 , the system foreground is running the game interface 11 corresponding to the game App. When the user wants to record the screen during the running of the game App, the user can perform a pull-down operation to trigger the system to display the following Figure 2A The drop-down notification bar 12 including the “Screen Recording” button 102 shown in (b) .

[0049] It should be noted that Figure 2A (b) is only an exemplary description of how to set a "screen recording" button in the pull-down notification bar 12. In actual use, the "screen recording" button is not limited to being set in the pull-down notification bar. In addition, the pull-down notification bar may also include other function buttons, such as a wireless network button 101, a flashlight switch button 103, and a screen projection button 104.

[0050] The user can perform a touch operation on the "record screen" button 102, where the touch operation may include but is not limited to "single click, double click, or long press operation, etc." After the electronic device receives the user's touch on the "record screen" button in the drop-down notification bar, it records the video. Figure 2AThe recording screen 13 shown in (c) in FIG.

[0051] The recorded image is the synthesized GI, and the specific synthesis method is as follows:

[0052] The screen recording app initiates the creation of a virtual screen. After the screen recording app creates the virtual screen, it triggers the image processing process to synthesize each frame, specifically synthesizing the GI on the virtual screen and the GI on the display interface of the electronic device. The synthesized GI is sent to the display interface of the electronic device for display, which is the recording screen 13 in (c), and is sent to the screen recording app, which encodes the image and stores it in a video format.

[0053] In actual use, you can call the MediaRecorder in the Android.media.API in the application architecture layer, that is, the multimedia creation interface in the media manager to create a data buffer, which is used to receive the captured screen images. Call the Media Projection in the Android.media.API, that is, the screen recording interface, to create a virtual screen Virtual Display, where the Virtual Display is a mirror copy of the physical screen of the electronic device in the memory. The Virtual Display saves the captured screen images to the data buffer. The Media Recorder integrates the video encoding and recording of the screen images in the data buffer to generate video clips.

[0054] See also Figure 2B , is a schematic diagram of another application scenario of a virtual screen provided in an embodiment of the present application.

[0055] Regarding the above Figure 2A In (b), when the user performs a touch operation on the "Cast Screen" button 104, where the touch operation may include but is not limited to "single click, double click, or long press operation, etc.", the electronic device receives the user's touch of the "Cast Screen" button in the pull-down notification bar and casts the screen, and obtains the following Figure 2B The screen projection prompt interface 21 shown in (a).

[0056] The prompt box prompts the screen projection service: This service requires turning on WLAN and Bluetooth for the wireless screen projection function. Is it turned on. In addition, the screen projection service also includes a "Cancel" button 211 and an "On" button 212. The user can touch the "On" button 212. The touch operation may include but is not limited to "single click, double click, or long press operation, etc.

[0057] After the electronic device receives the user's touch of the "on" button, it enters the screen projection device search interface 22 and searches for an available screen projection device 213. The user can click "ON" on the switch button to turn on the screen projection device to Figure 2A The display interface 11 in (a) is projected onto the display interface of the voting device 213. The display interface 23 of the projection device 213 shown in (c) is obtained.

[0058] The projection screen is the synthesized GI, and the specific synthesis method is as follows:

[0059] The screen casting App initiates the creation of a virtual screen. After the screen casting App creates the virtual screen, it triggers the image processing process to synthesize the image for each frame, specifically synthesizing the GI on the virtual screen and the GI on the display interface of the electronic device. The synthesized GI is sent to the display interface of the electronic device for display. It is also displayed on the display interface 23 of the screen casting device 213 (c).

[0060] Apart from Figure 2A and Figure 2B In addition to the screen projection and screen recording application scenarios shown, the virtual screen can also be applied to other scenarios. This application does not specifically limit the application scenarios of the virtual screen.

[0061] When recording on a virtual screen, you can use Figure 3 and Figure 4 The relevant schemes for GI synthesis are shown.

[0062] See also Figure 3 , is an interactive diagram of the method of virtual screen synthesis for related solutions. Figure 3 The App involved is located in the application layer of the electronic device software layer. The image processing process node involved is located in the application framework layer of the electronic device software layer.

[0063] The image processing process node is used to call the image processing process to implement the whole process of layer rendering, synthesis, and display. Among them, the image processing process includes an image synthesis thread, and the communication part of the image processing process that communicates with the App is located in the image synthesis thread.

[0064] S11: App creates a virtual screen.

[0065] S12: When the creation is successful, the App sends a start synthesis message to the image processing process node.

[0066] The start synthesis message is a message generated when the App successfully creates a virtual screen. It is used to enable the image processing process node to start the image processing process and perform layer rendering, synthesis, and display.

[0067] S13: The image processing process node is started.

[0068] After receiving the start synthesis message, the image processing process node starts the image processing process and calls the image synthesis thread to perform layer rendering, synthesis and display.

[0069] S14: The image processing process node sends a memory acquisition message to the App.

[0070] The memory acquisition message refers to the image processing process obtaining the memory buffer from the App to cache the synthesized GI in the buffer. The buffer exists in the BufferQueue in the App.

[0071] S15: APP obtains memory.

[0072] After receiving the memory acquisition message sent by the image processing process node, the APP obtains the Buffer used to cache the synthesized GI from the Buffer Queue.

[0073] S16: The App sends memory parameters to the image processing process node.

[0074] The memory parameter is used to indicate the specified buffer in the App's Buffer Queue, such as binding the slot parameter of the Buffer. In this way, the image processing process node can cache the synthesized GI into the buffer corresponding to the slot parameter through the slot parameter.

[0075] S17: The image processing process node caches the synthesized GI in the Buffer corresponding to the memory parameter.

[0076] The image processing process node caches the synthesized GI on the specified Buffer. At this time, the GI can be displayed on the virtual screen corresponding to the App, and the corresponding operations of the application process can be performed according to the functions of the application process, such as the screen projection function or the screen recording function, such as encoding the image corresponding to the screen projection function into a video format for storage.

[0077] S18: The image processing process node sends a return message to the App.

[0078] The return message is used to add the synthesized GI to the App's Buffer Queue, in order to trigger the App to receive the synthesized GI and perform subsequent operations.

[0079] S19: The App returns a success message to the image processing process node.

[0080] When the App receives the return message sent by the image processing process, it modifies the memory state, generates a Buffer event object for the synthesized GI, adds it to the BufferQueue, and returns a successful return message to the image processing process node.

[0081] In this way, the GIs of other frames are synthesized according to steps S14 to S19 until the App initiates the destruction of the virtual screen.

[0082] S110: The App sends a destroy virtual screen message to the image processing process node.

[0083] When the user closes the executing App, or when the executing App is closed by other means, the App initiates the destruction of the virtual screen and sends a destroy virtual screen message to the image processing process node.

[0084] S111: The image processing process node destroys the virtual screen and the virtual screen object.

[0085] The virtual screen object refers to a thread that performs virtual screen synthesis in an image processing process node.

[0086] However, if the App does not respond in time, or there is a communication anomaly between the App and the image processing process node, the image processing process will be blocked.

[0087] See also Figure 4 , which is a schematic diagram of the principle of blocking the image processing process in the related scheme.

[0088] S21: App creates a virtual screen.

[0089] S22: When the creation is successful, the App sends a start synthesis message to the image processing process node.

[0090] S23: The image processing process node sends a memory acquisition message to the App.

[0091] S24: When the image processing process node does not receive a message sent by the App within a preset time period, the system service process node detects that the image processing process node has a communication anomaly.

[0092] For example, the preset duration is 1 second. If the image processing process node does not receive a message sent by the App within 1 second, the system service process node detects that the image processing process node has a communication anomaly.

[0093] S25: The system service process restarts the electronic device.

[0094] After the system service process detects that the communication of the image processing process node is abnormal, the electronic device is restarted and a restart instruction is sent to the image processing process. The image processing process waits for the electronic device to restart before opening.

[0095] In the above solution, the image processing process communicates directly with the App. When the App does not respond in time or the communication with the image processing process is abnormal, the image processing process is blocked, causing the display interface of the electronic device to be in a frozen state. In the frozen state, the user display interface does not respond to the user's trigger operation, which makes the user experience poor.

[0096] In view of the above problems, the embodiment of the present application provides a method for triggering a virtual screen. When the application process does not respond in time, or the communication with the image processing process is abnormal, the task thread of the image processing process is prevented from being blocked, so that the impact of the abnormality is controlled in the communication thread, and other task threads are not affected. Since the communication thread is only used for cross-process communication with the application process, it is not used for GI synthesis of the display interface of the electronic device. Therefore, the blocking of the communication thread will not affect the GI synthesis of the display interface of the electronic device, and the GI synthesis of the display interface of the electronic device is related to the freezing of the display interface. When the GI of the display interface of the electronic device is synthesized normally, the display interface will not appear in a frozen screen state. When the image processing process uses the communication part of the image synthesis thread to communicate with the application process, when the application process does not respond in time, or the communication with the image processing process is abnormal, the communication part of the image processing process is blocked, and other threads of the image processing process are blocked, resulting in the GI of the display interface of the electronic device and the GI on the virtual screen cannot be synthesized. In summary, the present application uses the communication thread in the image processing process to communicate with the application process, effectively reducing the impact of a single application process abnormality on the underlying image processing process, to a certain extent, solves the problem of freezing of the display interface of the electronic device, and improves the stability of the device and the user experience.

[0097] In order to better illustrate the virtual screen processing method provided by the present application, the electronic device 100 involved in the embodiment of the present application is first introduced.

[0098] In some embodiments, the electronic device 100 may be a wearable electronic device such as a wearable sports watch, a wearable smart watch, a wearable bracelet, a wearable audio headset, etc. that needs to create a virtual screen; it may also be a mobile phone, a laptop computer, a tablet computer, an augmented reality (AR) device, a virtual reality (VR) device, and a vehicle-mounted device, etc. It should be noted that the embodiment of the present application does not impose any special restrictions on the specific type of the electronic device 100.

[0099] See also Figure 5 , is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application.

[0100] like Figure 5As shown, the electronic device 100 may include a processor 110, an internal memory 120, an antenna 1, an antenna 2, a mobile communication module 130, a wireless communication module 140, a display screen 160, a touch sensor 180K, etc.

[0101] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0102] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0103] For example, in the present application, when the image processing process performs GI synthesis of the first frame, the processor 110 can use the communication thread in the image processing process to communicate with the application process to obtain the communication result. When the communication result is normal, the synthesized GI is sent to the application process to be displayed on the virtual screen created by the application process. When the communication result is abnormal, the virtual screen is destroyed. At this time, when the application process does not respond in time or the communication with the image processing process is abnormal, the communication thread is blocked, but it will not affect other threads in the image processing process. Since the communication thread is only used for cross-process communication with the application process, it is not used for GI synthesis of the electronic device display interface. Therefore, the blocking of the communication thread will not affect the GI synthesis of the electronic device display interface, and the GI synthesis of the electronic device display interface is related to the frozen screen of the display interface. When the GI of the electronic device display interface is synthesized normally, the display interface will not appear in a frozen screen state. When the image processing process uses the communication part of the image synthesis thread to communicate with the application process, when the application process does not respond in time or the communication with the image processing process is abnormal, the communication part is blocked, and then the GI of the electronic device display interface of the image processing process and the GI on the virtual screen cannot be synthesized. In summary, this application solves the freezing problem of the electronic device display interface to a certain extent by using the communication thread in the image processing process to communicate with the application process, thereby improving the user experience. Among them, the controller can be the nerve center and command center of the electronic device. The controller can generate an operation control signal based on the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0104] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0105] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0106] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 130, wireless communication module 140, modem processor and baseband processor.

[0107] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0108] The internal memory 120 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 120. The internal memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 120, and / or the instructions stored in the memory provided in the processor.

[0109] A series of GIs can be displayed on the display screen 160 of the electronic device, and these GIs are the main screens of the electronic device. Generally speaking, the size of the display screen 160 of the electronic device is fixed, and only limited controls can be displayed on the display screen 160 of the electronic device. A control is a GI element, which is a software component included in an application program, and controls all data processed by the application program and interactive operations on these data. Users can interact with the control through direct manipulation to read or edit relevant information of the application program. Generally speaking, a control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0110] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0111] In addition, an operating system runs on the above components, such as the iOS operating system, the Android open source operating system, the Windows operating system, etc. Application programs can be installed and run on the operating system.

[0112] The operating system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0113] See also Figure 6 , which is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application. The present application embodiment takes the Android system of layered architecture as an example to exemplarily illustrate the software structure of the electronic device 100.

[0114] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer (Applications, App), framework layer (Framework, FWK), system library layer and kernel layer (Kernel).

[0115] The application layer may include a series of application packages. In some embodiments, the application layer may include screen recording apps, screen projection apps, game apps, and other applications.

[0116] The framework layer provides an application programming interface (API) and a programming framework for the application of the application layer, and the framework layer may include some predefined functions. In some specific implementations, the framework layer may include a GI framework and a display framework. Specifically, the framework corresponding to a virtual screen processing method provided in an embodiment of the present application is stored in the display framework of the application framework layer. The GI drawing thread is located in the GI framework.

[0117] Optionally, the image processing process node is located in the display framework and is used to perform layer rendering, synthesis and display. The image processing process node is used to call the image processing process. The image processing process includes at least one thread. Optionally, the image processing process provided in the embodiment of the present application includes an image synthesis thread, a communication thread, and a termination thread.

[0118] Optionally, the image synthesis thread is used to perform GI synthesis on the virtual screen and GI synthesis of the electronic device display interface. The communication thread is used to communicate with the App and synchronize the obtained result signal to the image synthesis thread. The termination thread is used to terminate the communication thread.

[0119] Optionally, the application framework layer includes a window manager, a multimedia manager, and a view system, etc.

[0120] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, and send interface information display instructions to the view system.

[0121] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build apps. The display interface can be composed of one or more views. The view system can also call the display driver of the display screen of the corresponding electronic device according to the display interface display instruction sent by the window manager, and drive the first display screen to display interface information through the display driver.

[0122] Multimedia manager, used to manage multimedia calling interfaces. The calling interfaces include but are not limited to: MediaRecorder, which is a multimedia creation interface, and Media Projection, which is a screen recording interface. MediaRecorder in the multimedia manager is used to create a data cache. Media Projection is used to create a Virtual Display. VirtualDisplay is a virtual reality display of the display screen of an electronic device in memory, that is, a virtual screen, which is used to mirror the display screen. The Virtual Display saves the captured screen images into the data cache, and then the MediaRecorder generates video clips from the screen images in the data cache.

[0123] The system library may include multiple functional modules. In some specific implementations, the system library layer includes functional modules such as SurfaceFlinger, communication thread, and termination thread.

[0124] Among them, SurfaceFlinger is used for functions such as layer creation, control and management. The communication thread is used to communicate with the App, and the termination thread is used to terminate the communication thread.

[0125] The kernel layer is responsible for managing the system's threads, memory, device drivers, files, and network systems, and determines the system's performance and stability. In some specific implementations, the kernel layer hardware driver layer, such as display driver, GPU driver, HWC driver, and CPU driver.

[0126] It should be noted that although the embodiment of the present application is described using the Android system as an example, its basic principles are also applicable to electronic devices 100 based on operating systems such as iOS and Windows.

[0127] Combine the following Figure 7 , a flowchart of a virtual screen processing method is provided for introduction to an embodiment of the present application.

[0128] like Figure 7 As shown, the method includes an image processing process and an App, wherein the image processing process includes a communication thread. The method specifically includes:

[0129] S31: When the image processing process performs GI synthesis of the first frame, the communication thread in the image processing process communicates with the App to obtain a communication result.

[0130] In the embodiment of the present application, the first frame is the current frame. The image processing process performs GI synthesis of the first frame, that is, the image processing process synthesizes the GI of the current frame.

[0131] Optionally, the image processing process is a SurfaceFlinger process.

[0132] The communication thread is a thread that communicates with the App in the image processing process. The embodiment of the present application creates an independent communication thread to communicate with the App across processes. When the App does not respond in time or the cross-process communication is blocked, only the communication thread of the image processing process is blocked, while other threads, such as the image synthesis thread of the display screen of the electronic device, will not be blocked. In this way, the freezing problem caused by the blocked image synthesis thread on the display interface of the electronic device is avoided.

[0133] Optionally, the communication thread can be a communication thread created in the image processing process when the App creates a virtual screen. In addition, when the App creates a virtual screen, it also creates a virtual screen object in the image processing process. Among them, the virtual screen object is used to maintain the correspondence between the virtual screen and the communication thread. In this way, when the App function does not involve the virtual screen, the virtual screen is not created, which saves memory to a certain extent and improves the efficiency of GI synthesis.

[0134] Optionally, the cross-process communication between the App and the communication thread can use the Binder mechanism. Compared with other cross-process transmission mechanisms, the Binder mechanism only needs to be copied once, has a faster transmission speed, and is more secure and easy to use.

[0135] S32: Determine whether the communication result is normal communication. If the communication is abnormal, execute S33; if the communication is normal, execute S34.

[0136] In the embodiment of the present application, the communication result refers to the cross-process communication between the App and the communication thread, including normal communication and abnormal communication.

[0137] Normal communication means that the App communicates normally with the communication thread. In the embodiment of the present application, normal communication means that the image processing process receives the message sent by the App within a preset time. For example, within 1 second, the image processing process receives the memory acquisition success message sent by the App, or receives the return success message within 1 second.

[0138] Communication anomaly refers to abnormal communication between the App and the communication thread. In the embodiment of the present application, communication anomaly refers to the image processing process not receiving a message sent by the App within a preset time. Exemplarily, within 1 second, the image processing process did not receive any message sent by the App. In the embodiment of the present application, communication anomaly refers to the App not responding in time, or abnormal communication between the App and the image processing process.

[0139] If the image processing process does not receive the message sent by the App within the preset time, it indicates that the communication is abnormal; otherwise, it indicates that the communication is normal.

[0140] S33: The image processing process deletes the virtual screen object.

[0141] The virtual screen object refers to threads and other contents related to virtual screen synthesis that exist in the image processing process, such as the GI synthesis part on the virtual screen in the image synthesis thread, the communication thread, etc.

[0142] When communication is abnormal, cross-process communication blocks the communication thread. At this time, the GI synthesis part on the virtual screen cannot be executed normally. In order to realize the normal execution of GI synthesis of the electronic device display interface, the virtual screen object is deleted, such as deleting the GI synthesis part on the virtual screen in the image synthesis thread, and / or the communication thread, etc.

[0143] S34: The image processing process sends the synthesized GI to the App for display on the virtual screen.

[0144] When the communication is normal, the image processing process sends the synthesized GI to the App for display on the virtual screen of the application process.

[0145] Optionally, when the application process is a screen recording process, the synthesized GI is sent to the application process, so that the application process encodes the synthesized GI into a video format for storage.

[0146] Optionally, when the application process is a screen projection process, the synthesized GI is sent to the application process, so that the application process sends the synthesized GI to the screen projection device for display.

[0147] A method for processing a virtual screen is provided in an embodiment of the present application. When the application process does not respond in time, or the communication with the image processing process is abnormal, the task thread of the image processing process is prevented from being blocked, so that the impact of the abnormality is controlled in the communication thread, and other task threads are not affected. Since the communication thread is only used for cross-process communication with the application process, it is not used for GI synthesis of the display interface of the electronic device. Therefore, the blocking of the communication thread will not affect the GI synthesis of the display interface of the electronic device, and the GI synthesis of the display interface of the electronic device is related to the freezing of the display interface. When the GI of the display interface of the electronic device is synthesized normally, the display interface will not appear in a frozen screen state. When the image processing process uses the communication part of the image synthesis thread to communicate with the application process, when the application process does not respond in time, or the communication with the image processing process is abnormal, the communication part of the image processing process is blocked, and other threads of the image processing process are blocked, resulting in the GI of the display interface of the electronic device and the GI on the virtual screen cannot be synthesized. In summary, the present application uses the communication thread in the image processing process to communicate with the application process, effectively reducing the impact of a single application process abnormality on the underlying image processing process, to a certain extent, solves the problem of freezing of the display interface of the electronic device, and improves the stability of the device and the user experience.

[0148] See also Figure 8 , which is an interactive diagram of a method for virtual screen synthesis provided in an embodiment of the present application. The method is illustrated by taking the image processing process as the SurfaceFlinger process as an example. The SurfaceFlinger process node involved in the method is used to implement the SurfaceFlinger process. Among them, the SurfaceFlinger process includes an image synthesis thread, a communication thread, and a termination thread. Among them, the image synthesis thread is used to implement the GI synthesis of the electronic device display interface and the GI synthesis on the virtual screen.

[0149] When the APP creates a virtual screen, it creates a communication thread in the SurfaceFlinger process, creates a virtual screen object, and then performs the following operations.

[0150] In an embodiment of the present application, the communication thread is used to implement communication between the SurfaceFlinger process and the App, and includes at least a first sub-thread and a second sub-thread.

[0151] The first sub-thread is used to obtain memory parameters from the App, and the second sub-thread is used to return the memory parameters to the App.

[0152] S41: The image synthesis thread in the SurfaceFlinger process node sends a first event signal to the communication thread.

[0153] The first event signal refers to a signal that triggers the communication thread to specify memory from the App for caching the synthesized GI. In an embodiment of the present application, the first event signal refers to a signal obtained when the SurfaceFlinger process starts to perform a frame synthesis, which is used to trigger the communication thread to determine the specified memory from the App, that is, the target Buffer. For example, the first event signal is a signal that triggers the communication thread to call DequeueBuffer().

[0154] In order to make the target buffer specified by the communication thread meet the memory size required for virtual screen synthesis, the first event signal can preset the width, height and pixel format of the target buffer. The App obtains the corresponding target buffer according to the preset width, height and pixel format.

[0155] Since the communication thread includes multiple sub-threads, in order to accurately call the first sub-thread of the communication thread, the first event signal may carry a first tag.

[0156] The first tag is a pre-set identifier, which can be a letter, a number, or other identifier. The first tag corresponds to the first sub-thread. When the first event signal carries the first tag, the communication thread calls the first sub-thread to determine the target Buffer from the App. For example, the first tag is DequeueBuffer, which is used to trigger the communication thread to call the first sub-thread DequeueBuffer() to determine the target Buffer from the application App.

[0157] S42: The communication thread sends a memory acquisition message to the App.

[0158] After receiving the first event signal sent by the image synthesis thread, the communication thread sends a memory acquisition message to the App, wherein the memory acquisition message is used to specify a target buffer from the Buffer Queue of the App.

[0159] The memory acquisition message includes at least the App identity and the memory acquisition instruction.

[0160] The App identity is used to indicate a unique corresponding App. Since the software program layer of the electronic device includes many Apps, the communication thread can find the required App from multiple Apps based on the App identity.

[0161] The memory acquisition instruction is used to trigger the App to determine the target Buffer from the Buffer Queue. Optionally, the memory acquisition instruction may be a call to DequeueBuffer() to cause the Buffer Queue of the App to release the target Buffer.

[0162] DequeueBuffer() specifies the order of memory acquisition, and prioritizes the target buffer from the slot bound to the memory. This can improve the efficiency of determination.

[0163] In order to make the target buffer determined by the communication thread meet the memory requirements for virtual screen synthesis, the memory acquisition instruction can also carry the width, height and pixel format specified by the first event signal. The App can determine the target buffer from the Buffer Queue according to the width, height and pixel format specified by the first event signal.

[0164] Optionally, the communication thread can send a memory acquisition message to the App through the Binder mechanism. Compared with other transmission mechanisms, the Binder mechanism only needs to copy once, which is faster, safer and easier to use.

[0165] S43: The App obtains the target Buffer.

[0166] The App determines the target Buffer from the Buffer Queue according to the memory acquisition instruction in the received memory acquisition message.

[0167] When the memory acquisition instruction is to call DequeueBuffer(), a slot is obtained from mFreeBuffers first, and the Bufferslot state corresponding to the slot is changed from FREE to DEQUEUED. Since the slot in mFreeBuffers is already connected to the Buffer, there is no need to reallocate the Buffer to the slot, which improves the acquisition speed.

[0168] In addition, if mFreeBuffers is empty, a slot is obtained from mFreeSlots, and a Buffer is allocated to it, and the Bufferslot state is changed from FREE to DEQUEUED.

[0169] The above mFreeSlots is a slot set whose BufferSlot structure is FREE and no Buffer is bound to it. mFreeBuffers is a slot set whose BufferSlot structure is FREE and there is a Buffer bound to it.

[0170] When the memory acquisition instruction carries the width, height, and pixel format specified by the first event signal, a target Buffer corresponding to the requirement is determined from the BufferQueue.

[0171] S44: The App sends the memory parameters corresponding to the target Buffer to the communication thread.

[0172] The memory parameters are used to specify the target buffer. That is, the memory parameters correspond to the buffers in the buffer queue one by one, and different buffers correspond to different memory parameters.

[0173] In this embodiment of the present application, the App sends the memory parameters corresponding to the target Buffer to the communication thread, so that the communication thread synchronizes the memory parameters to the image synthesis thread.

[0174] Optionally, when the memory acquisition instruction is DequeueBuffer(), the App assigns the acquired slot as an output parameter to the communication thread, where the slot is bound to a specified Buffer.

[0175] Optionally, if the target buffer bound to the slot is reallocated, the communication thread receives the return value sent by the APP in the BUFFER_NEEDS_REALLOCATION state. Otherwise, it returns the NO_ERROR state. In this way, the communication thread can determine the subsequent processing method based on the return value.

[0176] Among them, if the return value is BUFFER_NEEDS_REALLOCATION status, the communication thread needs to send a new pointer acquisition message to the App, such as sending RequestBuffer, to obtain the target Buffer pointer from the App, and then perform subsequent operations.

[0177] If the return value is No_ERROR, the communication thread directly performs subsequent operations.

[0178] Optionally, the App sends messages to the communication thread through the Binder mechanism, including memory acquisition messages, memory parameters, and / or acquisition of new pointer messages.

[0179] S45: The communication thread sends the first result signal to SurfaceFlinger.

[0180] In the embodiment of the present application, the first result signal refers to the result signal triggered after the communication thread receives the memory parameters sent by the App, which is used to indicate that the SurfaceFlinger process can perform the synthesis operation on the target Buffer specified by the BufferQueue. The first result signal includes the memory parameters.

[0181] Optionally, the first result signal may include the width, height, and pixel format of the target Buffer, so that the image synthesis thread determines whether the synthesis can be completed according to the width, height, and pixel format of the Buffer in the first result signal. It should be understood that when the width, height, and pixel format of the Buffer do not meet the requirements, such as the width is lower than the width set by the first event signal, the height is lower than the width set by the first event signal, or the pixel format is inconsistent with the pixel format set by the first event signal, the SurfaceFlinger process cannot synthesize the synthesis result on the target Buffer.

[0182] Furthermore, in order to make the virtual screen of SurfaceFlinger be synthesized on the specified target Buffer, the first result signal includes a Buffer pointer. The Buffer pointer uniquely determines the target Buffer. At this time, after the SurfaceFlinger process finds the target Buffer according to the Buffer pointer, it synthesizes the GI on the virtual screen on the target Buffer.

[0183] S46: The image synthesis thread performs GI synthesis on the target Buffer.

[0184] After receiving the first result signal sent by the communication thread, the image synthesis thread performs layer synthesis and synthesizes the synthesis result on the specified target Buffer.

[0185] The image synthesis thread includes hardware compositor (HWC) synthesis and / or GPU synthesis.

[0186] HWC is a dedicated image processing device used for layer synthesis and display. HWC can be an independent device or integrated into a system-level chip. The layer synthesis method that synthesizes the display data of the layer through HWC is called HWC synthesis.

[0187] GPU is a general-purpose image processing device. In addition to being used for layer synthesis, GPU is also used to complete other graphics processing tasks. The layer synthesis method that uses GPU to synthesize the display data of the layer is called GPU synthesis.

[0188] During actual use, the SurfaceFlinger process sends the layer parameters of the rendered layer to the HWC, and the HWC marks the rendered layer according to its own processing capabilities. When the number of rendered layers is greater than the maximum number of layers that the HWC can synthesize, some are marked as GPU synthesis and some are marked as HWC synthesis. The electronic device first calls the GPU to synthesize the layers marked as GPU synthesis to obtain the GPU synthesis layer, and then synthesizes the GPU synthesis layer and the layer marked as HWC synthesis to obtain the HWC synthesis layer, and synthesizes the HWC synthesis layer on the target Buffer to complete a frame of synthesis processing.

[0189] Optionally, when the first result signal includes the width, height, and pixel format of the target Buffer, SurfaceFlinger may first determine whether the acquired Buffer meets the synthesis requirements based on the first result signal. If the synthesis requirements are met, SurfaceFlinger performs synthesis. If the synthesis requirements are not met, SurfaceFlinger re-executes steps S41 to S46 until the synthesis requirements are met, and then executes S47.

[0190] S47: The image synthesis thread sends a second event signal to the communication thread.

[0191] The second event signal is an event signal for returning the memory parameters to the App. In an embodiment of the present application, the signal obtained after the SurfaceFlinger process completes a frame synthesis is used to trigger the communication thread to return the memory parameters to the App. Exemplarily, the second event signal is a call to QueueBuffer().

[0192] Since the communication thread includes multiple sub-threads, in order to accurately call the second sub-thread of the communication thread, the second event signal may carry a second tag.

[0193] The second tag is a pre-set identifier, which can be a letter, a number, or other identifier. The second tag corresponds to the second sub-thread. When the second event signal carries the second tag, the communication thread calls the second sub-thread to return the memory parameters to the App. For example, the second tag is QueueBuffer, which is used to trigger the communication thread to call the second sub-thread QueueBuffer() to return the memory parameters to the Buffer Queue of the App.

[0194] S48: The communication thread sends a return message to the App.

[0195] The return message includes the App identity, the return instruction and the second buffer.

[0196] The App identity is used to indicate a unique corresponding App. Since the software program layer of the electronic device includes many Apps, the communication thread can find the required App from multiple Apps based on the App identity.

[0197] The return instruction is used to trigger the App to perform subsequent operations.

[0198] Optionally, the second child thread is QueueBuffer(), and the memory can be returned to the App through QueueBuffer(). The specific implementation of QueueBuffer() is:

[0199] QueueBuffer() changes the Bufferslot state corresponding to the slot from DEQUEUED to QUEUED according to the slot parameter corresponding to the Buffer, generates a Buffer event object according to the Bufferslot information, and adds it to the BufferQueue.

[0200] Optionally, the communication thread sends a return message to the App through the Binder mechanism.

[0201] S49: The App sends a return success message to the communication thread.

[0202] When the App receives the memory parameters corresponding to the target Buffer returned by the communication thread, it sends a return success message to the communication thread.

[0203] The return success message is used to indicate that the memory parameters corresponding to the target Buffer are returned successfully. It can be letters, numbers, or other marks that can identify the successful return. For example, the return success message is Queue Buffer success.

[0204] Optionally, the App sends a return success instruction to the communication thread through the Binder mechanism

[0205] S410: The communication thread sends the second result signal to the image synthesis thread.

[0206] In the embodiment of the present application, the second result signal refers to the result signal triggered after the communication thread receives the return success message sent by the App, which is used to notify the image synthesis thread that the memory parameters of the target Buffer are returned successfully.

[0207] S411: Determine whether the communication thread is terminated. If so, execute S41. If not, execute S412.

[0208] S412: After the App initiates the destruction of the virtual screen, the App sends a destroy virtual screen instruction to the image synthesis thread of the SurfaceFlinger process node.

[0209] The destroy virtual screen instruction is used to cause the image synthesis thread process in SurfaceFlinger to destroy the virtual screen object, and SurfaceFlinger does not perform the virtual screen synthesis operation.

[0210] Optionally, when the App terminates the operation, the App initiates the destruction of the virtual screen. The App termination operation may be closing the App. For example, the App automatically terminates when the running time reaches a preset time, clicks to close the space, or automatically closes the App through language, gestures, etc.

[0211] Optionally, the system service process detects that the application process has failed, that is, closes the App, sends a close instruction to the application process, and closes the application process.

[0212] S413: The image synthesis thread sends a termination instruction to the communication thread.

[0213] After receiving the instruction to destroy the virtual screen object sent by the App, the image synthesis thread sends a termination instruction to the communication thread. The termination instruction is used to cause the image synthesis thread to execute the destruction of the virtual screen object, that is, terminate the communication thread and delete the virtual screen object. At this time, the communication thread no longer occupies the memory of the electronic device and will not affect the next virtual screen synthesis.

[0214] S414: The communication thread sends an instruction to destroy the virtual screen object to the image synthesis thread.

[0215] S415: The image synthesis thread destroys the virtual screen object.

[0216] When the App does not respond to SurfaceFlinger's communication request in time, or a communication anomaly occurs, execute Fig. 9 The interaction method shown.

[0217] See also Fig. 9 , which is an interactive diagram of a virtual screen synthesis failure provided in an embodiment of the present application.

[0218] The SurfaceFlinger process synthesizes the first frame.

[0219] S51: The image synthesis thread sends a first event signal to the communication thread.

[0220] S52: The communication thread sends memory acquisition information to the App.

[0221] The process is the same as steps S31 to S32 and will not be described again here.

[0222] S53: If the image synthesis thread does not receive the first result signal sent by the communication thread within the preset time, the communication thread is terminated.

[0223] In an embodiment of the present application, because the App did not respond to the communication request sent by the communication thread in a timely manner, or the communication with the communication thread was abnormal, etc., the image synthesis thread did not receive the first result signal sent by the communication thread within the preset time. At this time, it is determined that the current virtual screen is invalid, and the graphics synthesis thread creates a termination thread to terminate the communication thread, that is, the communication thread is no longer called.

[0224] The preset time length is a time length set by those skilled in the art based on experience, such as 1 second. The image synthesis thread does not receive the first result signal sent by the communication thread within 1 second.

[0225] SurfaceFlinger directly synthesizes the second frame, which is the next frame of the first frame.

[0226] At this time, although the virtual screen has failed, the image synthesis thread still performs the GI synthesis operation of the electronic device display interface. In this way, the display interface of the electronic device can be prevented from being still due to the blocking of the image synthesis thread, which provides the user's visual experience to a certain extent.

[0227] S54: SurfaceFlinger determines whether the communication thread is terminated. If terminated, execute S55.

[0228] In an embodiment of the present application, SurfaceFlinger monitors the running state of the communication thread. When it is detected that the communication thread is not running, for example, when it is detected that the running state of the communication thread is FALSE, the communication thread is terminated.

[0229] If the communication thread is detected to be in the running state, the communication thread has not terminated. SurfaceFlinger will continue to synthesize the second frame. When the second frame is synthesized, it will synthesize the third frame. When synthesizing the third frame, it will continue to determine whether the communication thread has terminated. If it has not terminated, it will continue to synthesize. When synthesizing the fourth frame, it will continue to determine whether the communication thread has terminated. This cycle will continue until the communication thread terminates.

[0230] In the embodiment of the present application, the termination condition of the communication thread is: the system server detects that the App has expired and sends a termination process instruction to the App. The App receives the termination process instruction and sends a memory acquisition instruction failure instruction to the communication thread. The communication thread receives the memory acquisition instruction failure instruction and the communication thread terminates.

[0231] S55: The SurfaceFlinger process deletes the virtual screen object.

[0232] The virtual screen synthesis failure provided in the embodiment of the present application is when the App fails to respond to the communication request sent by the communication thread in time, or the communication with the communication thread is abnormal, etc. Although the virtual screen has failed, SurfaceFlinger still performs the synthesis operation, so that the display interface of the electronic device can be prevented from being static due to the blocking of SurfaceFlinger, which provides the user's visual experience to a certain extent.

[0233] The technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, etc., various media that can store program codes.

[0234] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing a virtual screen, characterized in that: The method comprises: When the image processing process performs the GI synthesis of the image interface of the first frame, the communication thread in the image processing process is used to communicate with the application process to obtain a communication result, wherein the communication result includes normal communication and abnormal communication, and the normal communication means that the image processing process receives a message sent by the application process within a preset time; When the communication result is that the communication is abnormal, the virtual screen object in the image processing process is deleted; when the communication result is that the communication is normal, the synthesized GI is sent to the virtual screen created by the application process.

2. The method according to claim 1, characterized in that: The utilizing the communication thread in the image processing process to communicate with the application process to obtain a communication result includes: When the image synthesis thread in the image processing process sends a first event signal to the communication thread, the communication thread is used to send a memory acquisition message to the application process, wherein the memory acquisition message is used to determine a memory, and the memory is used to cache the synthesized GI; When the communication thread receives the memory parameter sent by the application process within the preset time and sends a first result signal to the image synthesis thread, it is determined that the communication result is that the communication is normal, and the memory parameter is used to specify the memory; When the communication thread does not receive the memory parameter sent by the application process within the preset time, it is determined that the communication result is a communication abnormality.

3. The method according to claim 2, characterized in that: When the first event signal is an event signal marked as DequeueBuffer, the memory acquisition message is DequeueBuffer().

4. The method according to claim 2 or 3, characterized in that: The memory acquisition message also includes memory size and pixel format.

5. The method according to any one of claims 2 to 4, characterized in that: When the communication result is that the communication is normal, sending the synthesized GI to the application process includes: When the communication result is that the communication is normal, the synthesized GI is cached in the memory, and a second event signal is sent to the communication thread; In response to the communication thread receiving the second event signal, using the communication thread to send a return message to the application process; When the communication thread receives the return success message sent by the application process and sends the second result signal to the image synthesis thread, it is determined to cache the synthesized GI into the memory.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: After the image processing process synthesizes the GI and it is determined that the communication thread is terminated, the virtual screen object is destroyed.

7. The method according to claim 5 or 6, characterized in that: The second event signal is an event signal marked as QueueBuffer, and the return message is QueueBuffer().

8. The method according to any one of claims 2 to 7, characterized in that: When the communication result is the communication abnormality, deleting the virtual screen object in the image processing process includes: When the communication is abnormal, a termination thread is constructed and GI synthesis of the second frame is entered, wherein the termination thread is used to terminate the communication thread, and the second frame is the next frame of the first frame; Determine whether the communication thread is terminated; when it is determined that the communication thread is not terminated, enter the GI synthesis of the third frame, and determine again whether the communication thread is terminated, the third frame is the next frame of the second frame, and the cycle is repeated until it is determined that the communication thread is terminated, and the virtual screen object is deleted.

9. The method according to claim 8, characterized in that: The determining that the communication thread is terminated includes: When the system service process detects that the application process has failed and sends a close instruction to the application process to close the application process, the communication thread receives the failure instruction of the memory acquisition message sent by the application process to determine that the communication thread is terminated.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: When the application process creates the virtual screen, the communication thread is created in the image processing process, and the virtual screen object is created to maintain the corresponding relationship between the virtual screen and the communication thread.

11. The method according to claim 10, characterized in that: The destroying of the virtual screen comprises: The communication thread is terminated, and the virtual screen object is destroyed.

12. The method according to any one of claims 1 to 11, characterized in that: After sending the synthesized GI to the application process, the method further includes: When the application process is a screen recording process, sending the synthesized GI to the application process, so that the application process encodes the synthesized GI into a video format for storage; When the application process is a screen projection process, the synthesized GI is sent to the application process, so that the application process sends the synthesized GI to the screen projection device for display.

13. The method according to any one of claims 1 to 12, characterized in that: The utilizing the communication thread in the image processing process to communicate with the application process comprises: The communication thread is utilized to communicate with the application process based on the Binder mechanism.

14. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is coupled to the processor; The memory stores a program, and when the program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 13.

Images