A virtual machine inter-based screen projection method and electronic device

Abstract

Embodiments of the present application provide a screen projection method based on inter-virtual machines and an electronic device, and relate to the technical field of communication. The method comprises: a sending front-end thread of a sending virtual machine receiving a data transmission instruction triggered by a user, wherein the data transmission instruction is used to instruct transmission of specified image data to a receiving virtual machine. Then the sending front-end thread triggers a sending back-end thread of the sending virtual machine to call a GPU to synthesize the specified image data in response to the data transmission instruction. Then the sending front-end thread sends a write-back instruction to the sending back-end thread, triggering the sending back-end thread to call the GPU to write back the specified image data to a preset shared display memory. Then the sending back-end thread sends a frame synchronization signal to the receiving virtual machine through a vsock, so that the receiving virtual machine reads the specified image data from the shared display memory. Thus, the communication efficiency between systems in the vehicle machine is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a screen projection method based on inter-virtual machines and an electronic device. BACKGROUND

[0002] An intelligent cockpit system of an in-vehicle infotainment (IVI) generally includes multiple different systems, such as an Android system, an Ubuntu system, and an Automotive Grade Linux (AGL) system. The systems running in the IVI are isolated from each other and can independently display internal images of the systems.

[0003] In some scenarios, there is a demand for image transmission between different systems. For example, in a map navigation scenario, the Android system needs to project images of a navigation application to an instrument display screen controlled by the AGL system for display. In the existing image transmission method, the Android system encodes images through a hardware encoder, sends the encoded images to the AGL system through a network, and the AGL system decodes the received images through a hardware decoder and then displays the images.

[0004] This method relies on hardware encoders and decoders, making the data transmission method require high hardware. Moreover, the encoded images of the Android system need to be copied from a cache to a network card and then sent through the network, and then the AGL system needs to obtain the received images from the network card, decode the images, and copy the decoded images to the cache, so that the data transmission process needs to undergo multiple data copying, resulting in slow transmission speed. Moreover, network performance not only affects the transmission speed but also can cause data loss, resulting in a loss of transmission performance. Therefore, the communication efficiency between the internal systems of the IVI is low. SUMMARY

[0005] Embodiments of the present application aim to provide a screen projection method based on inter-virtual machines and an electronic device to improve the communication efficiency between systems in an IVI. The specific technical solutions are as follows:

[0006] In a first aspect, a screen projection method based on inter-virtual machines is provided, which is applied to a sending-end virtual machine, the sending-end virtual machine and a receiving-end virtual machine are deployed in a same physical machine, the sending-end virtual machine includes a sending front-end thread and a sending back-end thread; the method includes:

[0007] The sending front-end thread receives a data transmission instruction triggered by a user, the data transmission instruction is used to instruct to transmit specified image data to the receiving-end virtual machine;

[0008] The sending front-end thread triggers the sending back-end thread to call a graphics processing unit (GPU) to synthesize the specified image data in response to the data transmission instruction;

[0009] The sending front-end thread sends a write-back instruction to the sending back-end thread, triggering the sending back-end thread to call the GPU to write back the specified image data into a preset shared display memory;

[0010] The sending back-end thread sends a frame synchronization signal to the receiving end virtual machine through a virtual machine intercommunication socket (vsock) to enable the receiving end virtual machine to read the specified image data from the shared display memory.

[0011] Optionally, before the sending back-end thread calls the GPU to synthesize the specified image data, the method further comprises:

[0012] The sending back-end thread sets a resource tag for a target GPU resource used to synthesize the specified image data;

[0013] The sending back-end thread calls the GPU to write back the specified image data into a preset shared display memory, comprising:

[0014] The sending back-end thread finds the target GPU resource marked by the resource tag, and calls the target GPU resource to write back the specified image data into the shared display memory.

[0015] Optionally, before the sending front-end thread receives a data transmission instruction triggered by a user, the method further comprises:

[0016] After the sending end virtual machine is initialized, the sending back-end thread maps the shared display memory and opens a vsock on the host side.

[0017] Optionally, the shared display memory comprises a plurality of storage areas; and the calling the target GPU resource to write back the specified image data into the shared display memory comprises:

[0018] The sending back-end thread selects a storage area from the shared display memory, sets a state of the selected storage area to a busy state, and the busy state indicates that the storage area is being used;

[0019] The sending back-end thread calls the target GPU resource to write back the specified image data into the selected storage area, updates a state of the selected storage area to a filled state, and sets a fill time stamp of the selected storage area to a current time, and the filled state indicates that the storage area stores data.

[0020] Optionally, the sending back-end thread selects a storage area from the shared display memory, comprising:

[0021] If there are idle storage areas in the shared video memory, the sending backend thread selects an idle storage area from the shared video memory, where idle status indicates that no data is stored in the storage area; or,

[0022] If there is no free storage area in the shared video memory, the sending backend thread selects the storage area that is filled with the earliest timestamp.

[0023] Optionally, before triggering the sending backend thread to call the GPU to synthesize the specified image data, the method further includes:

[0024] The sending front-end thread obtains the display frame rate of the receiving virtual machine;

[0025] Triggering the sending backend thread to call the GPU to synthesize the specified image data includes:

[0026] The sending front-end thread triggers the sending back-end thread to call the target GPU resources to synthesize the specified image data according to the acquired display frame rate.

[0027] A second aspect of this application provides a screen mirroring method based on inter-virtual machine interaction, applied to a receiving virtual machine, wherein the receiving virtual machine and the sending virtual machine are deployed on the same physical machine, and the receiving virtual machine includes a receiving front-end thread; the method includes:

[0028] The receiving front-end thread receives the frame synchronization signal sent by the sending virtual machine through the virtual machine inter-virtual machine communication socket vsock; the frame synchronization signal is the signal sent by the sending back-end thread to the receiving virtual machine after the sending front-end thread responds to the user-triggered data transmission command, triggers the sending back-end thread to call the graphics processor GPU to synthesize specified image data, and calls the GPU to write the specified image data back to the preset shared video memory.

[0029] The receiving front-end thread responds to the frame synchronization signal by reading the specified image data from the shared video memory.

[0030] Optionally, the shared video memory includes multiple storage areas; the receiving front-end thread, in response to the frame synchronization signal, reads the specified image data from the shared video memory, including:

[0031] The receiving front-end thread selects the storage area with the earliest filled timestamp and sets the status of the selected storage area to busy; the filled state indicates that the storage area stores data, and the busy state indicates that the storage area is being used.

[0032] The receiving front-end thread reads the specified image data from the selected storage area and sets the status of the selected storage area to an idle state; the idle state indicates that no data is stored in the storage area.

[0033] Optionally, the receiving virtual machine further includes a receiving backend thread; before the receiving frontend thread receives the frame synchronization signal sent by the sending virtual machine via vsock, the method further includes:

[0034] After the virtual machine at the receiving end is initialized, the receiving backend thread creates the shared video memory;

[0035] The receiving front-end thread maps the shared video memory and opens vsock on the client side.

[0036] A third aspect of the embodiments of this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0037] Memory, used to store computer programs;

[0038] When the processor executes a program stored in memory, it implements the screen projection method steps described above based on virtual machines.

[0039] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described screen mirroring methods based on virtual machines.

[0040] A fifth aspect of this application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the screen mirroring method based on virtual machines described in any of the preceding embodiments.

[0041] Beneficial effects of the embodiments in this application:

[0042] In the screen projection method and electronic device based on virtual machines provided in this application embodiment, the systems that need to communicate are set up in different virtual machines. The sending virtual machine can respond to the data transmission command triggered by the user through the sending front-end thread, triggering the sending back-end thread of the sending virtual machine to call the GPU to synthesize specified image data, and then write back the synthesized image data to the specified storage address through the GPU write-back interface. Therefore, in this application embodiment, the GPU data write-back interface is used to set a preset shared video memory as the storage location for image data write-back, so that the sending front-end thread can trigger the sending back-end thread to call the GPU to write back the specified image data to the shared video memory, so that the receiving virtual machine can directly read the specified image data from the shared video memory. This process does not require data copying, and since it does not go through network transmission, it does not require encoding and decoding, and is not affected by network performance. Therefore, this application embodiment can improve the communication efficiency between various systems in the vehicle system.

[0043] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0045] Figure 1 A flowchart illustrating a screen mirroring method between virtual machines provided in this application embodiment;

[0046] Figure 2 A flowchart illustrating another screen mirroring method between virtual machines provided in this application embodiment;

[0047] Figure 3 A flowchart illustrating another screen mirroring method between virtual machines provided in this application embodiment;

[0048] Figure 4 A flowchart illustrating another screen mirroring method between virtual machines provided in this application embodiment;

[0049] Figure 5 A flowchart illustrating another screen mirroring method between virtual machines provided in this application embodiment;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0052] To improve communication efficiency between systems in an in-vehicle infotainment system, this application provides a screen mirroring method based on virtual machines. The sending and receiving virtual machines involved in this method are both deployed on the same physical machine, which can be the in-vehicle infotainment system. The physical machine runs a host operating system, such as Ubuntu. Both the sending and receiving virtual machines are equivalent to processes (QEMU) running on the host operating system. The sending virtual machine runs a sending virtual operating system, and the receiving virtual machine runs a receiving virtual operating system; for example, the sending virtual machine runs Android, and the receiving virtual machine runs Automotive Grade Linux (AGL).

[0053] This method is applied to the sending virtual machine, which consists of two threads: a sending front-end thread and a sending back-end thread. The sending back-end thread is the main thread that handles virtual machine resource requests and is used to access physical resources in the physical machine, such as graphics processing unit (GPU) resources. The sending front-end thread is an internal thread of the virtual machine, namely the operating system (OS) thread. Figure 1 As shown, the method includes the following steps:

[0054] S101, Send the front-end thread to receive the data transmission command triggered by the user.

[0055] Optionally, users can trigger data transmission commands by clicking buttons on the in-vehicle display screen or by issuing voice commands; this application embodiment does not specifically limit this.

[0056] The data transmission command is used to instruct the transmission of specified image data to the receiving virtual machine. For example, if the sending virtual machine runs Android and the receiving virtual machine runs AGL, the passenger can click the projection button on the in-vehicle display screen in front of their seat to project the image displayed on the in-vehicle display screen onto the instrument panel display in front of the driver. Conversely, if the sending virtual machine runs AGL and the receiving virtual machine runs Android, the driver can issue a voice command to project the image displayed on the instrument panel display in front of the passenger's seat onto the in-vehicle display screen.

[0057] S102. The front-end thread responds to the data transmission command and triggers the back-end thread to call the GPU to synthesize the specified image data.

[0058] In this embodiment, the sending front-end thread can create a virtual display device in response to a data transmission command. The virtual display device captures main screen data and application data. The application data is the data to be displayed in the user-specified application running in the sending virtual machine, and the main screen data is the data to be displayed by the system running in the sending virtual machine. For example, when a data transmission command instructs the navigation application running in the sending virtual machine to be projected onto the receiving virtual machine, the virtual display device can capture the navigation display data of the navigation application and the top display bar of the Android system running in the sending virtual machine, including time, signal status, and the Android system's back and close buttons. The sending front-end thread allocates video memory to the virtual display device, which is used to temporarily store the GPU's composite results. The sending front-end thread uses a semi-virtualized graphics processor (virtio GPU) driver to send a resource allocation command to the sending back-end thread. The sending back-end thread responds to the resource allocation command, allocates target GPU resources, and then uses the target GPU resources to composite the captured data into specified image data.

[0059] S103. The sending front-end thread sends a write-back instruction to the sending back-end thread, triggering the sending back-end thread to call the GPU to write back the specified image data to the preset shared video memory.

[0060] After the specified image data is synthesized, the sending front-end thread can notify the virtual screen image consumer to consume this frame of specified image data. For example, it can notify a preset consumption service to access the specified image data, thereby triggering the data write-back capability, i.e., sending a write-back instruction to the sending back-end thread. In response to the write-back instruction, the sending back-end thread calls the GPU to write the synthesized specified image data back to a preset shared video memory. This shared video memory can be Intel Virtual Machine Shared Memory (ivShareMemory). ivShareMemory is a shared memory in kernel-based virtual machine (KVM) virtualization schemes that can be accessed between virtual machines and between a virtual machine and the host. In this embodiment, it is used as shared video memory between the sending virtual machine and the receiving virtual machine.

[0061] S104. The sending backend thread sends a frame synchronization signal to the receiving virtual machine through the Virtual Machine Communication Interface socket (vsock) so that the receiving virtual machine can read the specified image data from the shared video memory.

[0062] Among them, virtio's vsock can be used for communication between virtual machines and hosts. In this embodiment, vsock is used to implement communication between the sending backend thread of the sending virtual machine and the receiving frontend thread of the receiving virtual machine.

[0063] In the screen mirroring method between virtual machines provided in this application embodiment, the systems requiring communication are set up in different virtual machines. The sending virtual machine can respond to the data transmission command triggered by the user through the sending front-end thread, triggering the sending back-end thread of the sending virtual machine to call the GPU to synthesize specified image data, and then write back the synthesized image data to a specified storage address through the GPU write-back interface. Therefore, in this application embodiment, the GPU data write-back interface is used, and a preset shared video memory is used as the storage location for image data write-back. This allows the sending front-end thread to trigger the sending back-end thread to call the GPU to write back the specified image data to the shared video memory, so that the receiving virtual machine can directly read the specified image data from the shared video memory. This process does not require data copying, and since it does not involve network transmission, it does not require encoding or decoding and is not affected by network performance. Therefore, this application embodiment can improve the communication efficiency between various systems in the vehicle system.

[0064] Since the specified image data in this embodiment of the application is not transmitted over the network, but synchronized through shared video memory, zero frame loss and zero latency can be achieved, enabling efficient screen projection between vehicle systems.

[0065] In one embodiment of this application, before the backend thread in S102 calls the GPU to synthesize the specified image data, the backend thread may also set a resource tag for the target GPU resource used to synthesize the specified image data. The resource tag is used to mark that the target GPU resource is a GPU resource used to synthesize the specified image data.

[0066] The above-mentioned method of sending the backend thread to call the GPU to write back the specified image data to the preset shared video memory can be implemented as follows: the backend thread searches for the target GPU resource marked by the resource marker, calls the target GPU resource to write back the specified image data to the shared video memory.

[0067] This application embodiment sets a resource tag for the target GPU resource, so that when writing back specified image data, the target GPU resource marked by the resource tag can be found, thereby calling the target GPU resource to write back the specified image data to the shared video memory. In other words, this application embodiment distinguishes the target GPU resource from other GPU resources, thereby achieving the writing back of specified image data to the shared video memory without affecting the use of virtual display devices in other scenarios.

[0068] In one embodiment of this application, when the backend thread in step S102 calls the GPU to synthesize specified image data, it can execute the process at a preset frame rate. For example, the preset frame rate is 30 frames per second. The frame rate on which the sending virtual machine synthesizes the specified image data can be different from the frame rate of the receiving virtual machine's display device. After the receiving virtual machine obtains the specified image data, it can display it at the required frame rate.

[0069] Alternatively, in another embodiment of this application, before the sending backend thread triggers the GPU to synthesize the specified image data in S102 above, the sending frontend thread can also obtain the display frame rate of the receiving virtual machine. For example, the sending frontend thread requests the display frame rate of the receiving virtual machine from the receiving virtual machine via the network, or the sending frontend thread can also read the pre-recorded display frame rate of the receiving virtual machine. This application embodiment does not limit the specific method of obtaining the display frame rate of the receiving virtual machine. Afterwards, the sending frontend thread can configure the frame rate of the virtual display device to be the obtained display frame rate of the receiving virtual machine.

[0070] Based on this, the above-mentioned method of sending backend thread to call GPU to synthesize specified image data in S102 can be implemented as follows: the sending frontend thread triggers the sending backend thread to call target GPU resources to synthesize specified image data according to the obtained display frame rate.

[0071] In this embodiment, the virtual display device created by the sending front-end thread can call the sending back-end thread to synthesize specified image data using the target GPU resources, according to the display frame rate of the receiving virtual machine.

[0072] The embodiments of this application can synthesize specified image data according to the display frame rate of the receiving virtual machine and transmit it to the receiving virtual machine, thereby achieving more accurate synchronization between the sending and receiving ends.

[0073] In one embodiment of this application, before the sending front-end thread receives the data transmission command triggered by the user, the host can also initialize the sending virtual machine. That is, the host can start the sending virtual machine according to its startup parameters. These startup parameters include support for vsock and ivShareMemory features, i.e., vsock configuration parameters and ivShareMemory configuration parameters. Afterward, the sending virtual machine creates an operating system thread and uses it as the sending front-end thread. Once the sending front-end thread starts, the sending virtual machine simulates the main thread and uses it as the sending back-end thread.

[0074] After the sending virtual machine is initialized, the sending backend thread maps shared video memory and opens vsock on the host side, that is, it enables vsock host. vsock host can send signals through vsock, such as sending frame synchronization signals to the receiving virtual machine through vsock.

[0075] In this embodiment, the sending backend thread maps the address of the shared video memory to its process space; that is, it converts the logical address of the shared video memory into a directly addressable physical address, thereby enabling subsequent access to the shared video memory. Simultaneously, the sending backend thread opens a vsock on the host side, allowing it to send frame synchronization signals to the receiving virtual machine using the vsock.

[0076] In one embodiment of this application, the shared video memory includes multiple storage regions. See also Figure 2 The method described in S103 above, where the backend thread calls the target GPU resource to write the specified image data back to the shared video memory, can be implemented as follows:

[0077] S1031. The backend thread selects a storage region from the shared video memory and sets the status of the selected storage region to busy. The busy state indicates that the storage region is currently in use.

[0078] In this embodiment of the disclosure, if there is a free storage area in the shared video memory, the sending backend thread can select a free storage area from the shared video memory, where the free state indicates that no data is stored in the storage area; or, if there is no free storage area in the shared video memory, the sending backend thread can select the storage area with the earliest filled timestamp, where the filled state indicates that data is stored in the storage area, and the filled timestamp indicates the time when the storage area was filled with data.

[0079] The bufferstate manages the state of each memory region in shared memory. The sending backend thread requests a free memory region from the bufferstate. If a free memory region exists, the bufferstate returns its memory index. Each memory region has a corresponding memory index, which includes a base address and an offset. The address of the memory region can be calculated using the base address and offset, allowing data to be written to that address. If no free memory region exists, the bufferstate notifies the sending backend thread that there are no free memory regions. The sending backend thread then requests a filled memory region from the bufferstate, which returns the memory index of the earliest filled memory region. Finally, the sending backend thread notifies the bufferstate to set the state of that memory region to busy.

[0080] Alternatively, the backend thread requests a writable storage area from bufferstate. Bufferstate searches for a free storage area; if it exists, it returns the memory index of that free storage area. If it does not exist, it returns a filled storage area with the earliest fill timestamp and updates its state to busy.

[0081] S1032. The backend thread calls the target GPU resource to write the specified image data back to the selected storage area, updates the status of the selected storage area to the filled state, and sets the filling timestamp of the selected storage area to the current time.

[0082] If the selected storage area already contains data, the target GPU resource can overwrite the original stored data with the current specified image data during write-back, so that the selected storage area stores the current specified image data.

[0083] After the backend thread calls the target GPU resource to write the specified image data back to the selected storage area, it sends the memory index of the selected storage area and the current time to bufferstate, causing bufferstate to switch the state of the selected storage area from busy to filled and set its filling timestamp to the current time.

[0084] In this embodiment, a multi-caching mechanism is set up in the shared video memory, so that the reading and writing of the shared video memory by the receiving virtual machine and the sending virtual machine do not affect each other.

[0085] Based on the same inventive concept, this application also provides a screen mirroring method between virtual machines. The receiving virtual machine and the sending virtual machine are deployed on the same physical machine, which can be an in-vehicle infotainment system. A host operating system runs on the physical machine, such as Ubuntu. Both the sending and receiving virtual machines are equivalent to processes running on the host operating system. The sending virtual machine runs a sending virtual operating system, and the receiving virtual machine runs a receiving virtual operating system; for example, the sending virtual machine runs Android, and the receiving virtual machine runs AGL.

[0086] This method is applied to the receiving virtual machine, which includes two threads: a receiving front-end thread and a receiving back-end thread. The receiving back-end thread is the main thread that handles virtual machine resource requests, used to access physical resources in the physical machine, such as GPU resources; the receiving front-end thread is an internal thread of the virtual machine, i.e., an OS thread. Figure 3 As shown, the method includes the following steps:

[0087] S301, The receiving front-end thread receives the frame synchronization signal sent by the sending virtual machine through vsock.

[0088] The frame synchronization signal is a signal sent by the sending backend thread to the receiving virtual machine in response to a user-triggered data transmission command. This triggers the sending backend thread to call the GPU to synthesize the specified image data and then call the GPU to write the specified image data back to the preset shared video memory.

[0089] S302: Receive the frame synchronization signal from the front-end thread and read the specified image data from the shared video memory.

[0090] In the embodiments of this disclosure, after the receiving front-end thread reads the specified image data, it can directly display it on the screen or perform other processing, such as combining the specified image data with the main screen data of the receiving virtual machine and displaying the combined data. This application does not specifically limit the processing of the specified image data by the receiving end.

[0091] The screen mirroring method based on virtual machines provided in this application embodiment requires the communicating systems to be set up in different virtual machines. The sending virtual machine can respond to the data transmission command triggered by the user through the sending front-end thread, triggering the sending back-end thread of the sending virtual machine to call the GPU to synthesize specified image data, and then write back the synthesized image data to a specified storage address through the GPU write-back interface. Therefore, this application embodiment utilizes the GPU data write-back interface, using a preset shared video memory as the storage location for image data write-back, so that the sending front-end thread can trigger the sending back-end thread to call the GPU to write back the specified image data to the shared video memory, allowing the receiving virtual machine to directly read the specified image data from the shared video memory. This process does not require data copying, and since it does not involve network transmission, it does not require encoding or decoding, and is not affected by network performance. Therefore, this application embodiment can improve the communication efficiency between various systems in the vehicle system.

[0092] Since the specified image data in this embodiment of the application is not transmitted over the network, but synchronized through shared video memory, zero frame loss and zero latency can be achieved, enabling efficient screen projection between vehicle systems.

[0093] In one embodiment of this application, the receiving virtual machine further includes a receiving backend thread. Before receiving the frame synchronization signal sent by the sending virtual machine via vsock, the host can also initialize the receiving virtual machine. That is, the host can start the receiving virtual machine according to the startup parameters of the receiving virtual machine. These startup parameters include vsock feature support and ivShareMemory feature support, i.e., vsock configuration parameters and ivShareMemory configuration parameters. Afterwards, the receiving virtual machine creates an operating system thread and uses the operating system thread as the receiving frontend thread. After the receiving frontend thread starts, the receiving virtual machine simulates the main thread and uses the main thread as the receiving backend thread.

[0094] After the receiving virtual machine is initialized, the receiving backend thread creates shared video memory, the receiving frontend thread maps the shared video memory, and opens vsock on the client side, that is, enables vsock guest. vsock guest can receive signals through vsock, such as receiving frame synchronization signals sent by the sending virtual machine through vsock.

[0095] In this embodiment, the receiving front-end thread maps the address of the shared video memory to its process space; that is, it converts the logical address of the shared video memory into a directly addressable physical address, thereby enabling subsequent access to the shared video memory. Simultaneously, the receiving front-end thread opens a vsock on the client side, allowing it to use the vsock to receive frame synchronization signals sent by the sending back-end thread of the sending virtual machine.

[0096] In one embodiment of this application, the shared video memory includes multiple storage regions. The frame synchronization signal sent by the sending backend thread to the receiving frontend thread may carry the memory index of the storage region where the specified image data is located, so that the receiving frontend thread can calculate the address of the storage region based on the memory index and read the specified image data from the address.

[0097] Or see Figure 4 The above-mentioned method for receiving specified image data from shared video memory in response to a frame synchronization signal by the front-end thread includes the following steps:

[0098] S3021. The receiving front-end thread selects the storage area with the earliest filled timestamp and sets the status of the selected storage area to busy. Here, "filled" indicates that data is stored in the storage area, and "busy" indicates that the storage area is being used.

[0099] BufferState manages the state of each storage region in shared memory. The receiving front-end thread can communicate with BufferState via vsock. The receiving front-end thread requests the storage region with the earliest filled timestamp from BufferState, and BufferState returns the memory index of that storage region. This memory index includes a base address and an offset. The address of the storage region can be calculated using the base address and offset, allowing data to be read from that address.

[0100] Subsequently, the receiving front-end thread sends a notification message to bufferstate, which includes the memory index of the storage area, so that bufferstate updates the state of the storage area to busy. Alternatively, bufferstate automatically updates the state of the storage area to busy when it returns the memory index of the storage area with the earliest filled timestamp.

[0101] S3022: The receiving front-end thread reads the specified image data from the selected storage area and sets the status of the selected storage area to idle. Idle status indicates that no data is stored in the storage area.

[0102] The receiving front-end thread reads the specified image data from the selected storage area. After reading, the storage area is empty, so the receiving front-end thread sends a notification message to bufferstate, which includes the memory index of the storage area, so that bufferstate sets the state of the storage area to free.

[0103] In this embodiment, a multi-caching mechanism is set up in the shared video memory, so that the reading and writing of the shared video memory by the receiving virtual machine and the sending virtual machine do not affect each other.

[0104] The descriptions of related steps in the screen projection method based on virtual machines applied to the sending virtual machine and the screen projection method based on virtual machines applied to the receiving virtual machine provided in this application embodiment can be referred to each other.

[0105] See Figure 5 The following describes the overall process of the screen mirroring method based on virtual machines provided in the embodiments of this application, combined with application scenarios:

[0106] The host operating system running on the vehicle's infotainment system is Ubuntu. The host operating system starts virtual machine 1 according to the configuration parameters of virtual machine 1, which include support for vsock and ivShareMemory features.

[0107] Next, virtual machine 1 creates the OS1 thread and uses it as the receiving front-end thread. Assuming virtual machine 1 is running AGL, OS1 is an internal thread of the guest operating system running on the host machine, which can be represented as AGL guest OS. After the OS1 thread starts, virtual machine 1 simulates main thread 1 and uses it as the receiving back-end thread.

[0108] After virtual machine 1 is initialized, main thread 1 opens vsock on the host side and creates shared video memory, i.e., ivShareMemory, for example, the logical address of the shared video memory is dev / shm / xxx. OS1 thread maps the shared video memory, thereby converting the logical address of the shared video memory into a directly addressable physical address, so that the shared video memory can be accessed later, and opens vsock on the guest side.

[0109] The host operating system starts virtual machine 2 according to the configuration parameters of virtual machine 2, which include support for vsock and ivShareMemory features.

[0110] Next, virtual machine 2 creates the OS2 thread and uses it as the front-end thread for sending. Assuming virtual machine 2 is running Android, OS2 is a thread within the guest operating system running on the host machine, which can be represented as Android guest OS. After the OS2 thread starts, virtual machine 2 simulates main thread 2 and uses it as the back-end thread for sending.

[0111] After virtual machine 2 is initialized, main thread 2 maps shared video memory, thereby converting the logical address of shared video memory into a directly addressable physical address, so that shared video memory can be accessed later, and vsock is opened on the host side.

[0112] Virtual machine 1 and virtual machine 2 can be started when needed. For example, virtual machine 1 is an AGL system and starts immediately after the host operating system starts, while virtual machine 2 is an Android system and starts when a passenger opens the display screen located between the driver and passenger seats.

[0113] When OS2 receives a data transfer command triggered by the user, it creates a virtual display device. This virtual display device is used to synthesize the captured main screen data and application data using an image compositing system. The virtual display device captures the main screen data and application data and triggers video memory allocation before compositing them. The allocated video memory is used to store the composite result of the target GPU resource. OS2 uses the virtio GPU driver to send a resource allocation command to main thread 2. Main thread 2 responds to the resource allocation command, allocates the target GPU resource, sets a resource flag for the target GPU resource, and then calls the target GPU resource to composite the captured data into the specified image data. Afterward, it notifies OS2 that the compositing is complete. OS2 submits the specified image data, that is, notifies the consuming service to access the specified image data, thereby triggering the data write-back capability, i.e., sending a write-back command to main thread 2. This causes main thread 2 to search for the target GPU resource marked by the resource flag, select a storage area in shared memory with a free state from the bufferstate, and if not found, select the storage area with the earliest filled timestamp. Main thread 2 sets the state of the selected storage region in bufferstate to busy, calls the target GPU resource to write the specified image data back to the selected storage region, and then calls the frame queue (queueFrame) interface to set the state of that storage region in bufferstate to filled, setting the filling timestamp to the current time. Afterwards, main thread 2 sends a frame synchronization signal to OS1 through vsock to notify OS1 to refresh the image.

[0114] OS1 receives the frame synchronization signal via vsock and, in response, selects the earliest filled memory region in the shared video memory from the bufferstate that is in the filled state, and sets the state of that memory region in the bufferstate to busy. It then reads the specified image data from the selected memory region, displays the read image data, and sets the state of that memory region in the bufferstate to free.

[0115] If a shared memory is set up between the sending and receiving systems, the sending system needs to use its CPU to copy data from its own memory to the shared memory, and then the receiving system needs to use its CPU to copy data from the shared memory to its own memory. Since this data transmission process requires multiple data copies, and the amount of data to be transmitted is very large, it will generate significant CPU overhead, potentially causing system lag and preventing the transmission performance from meeting the requirements of high frame rate transmission.

[0116] The embodiments disclosed herein utilize data write-back capability, using the GPU to write image data to shared video memory. The receiving system can then directly read and display the data from the shared video memory. This eliminates the need for CPU-based data copying, thus avoiding CPU overhead and system lag issues caused by excessive CPU consumption, while also meeting high frame rate transmission requirements.

[0117] This invention also provides an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0118] Memory 603 is used to store computer programs;

[0119] When the processor 601 executes the program stored in the memory 603, it implements the method steps in the above method embodiments.

[0120] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0121] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0122] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0123] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0124] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described screen projection methods based on virtual machines.

[0125] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the screen mirroring methods based on virtual machines in the above embodiments.

[0126] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0127] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0128] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the electronic device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A screen mirroring method based on virtual machines, characterized in that, The method is applied to a sending virtual machine, wherein the sending virtual machine and the receiving virtual machine are deployed on the same physical machine, and the sending virtual machine includes a sending front-end thread and a sending back-end thread, wherein the sending front-end thread is a guest operating system thread within the sending virtual machine, and the sending back-end thread is the main thread of the sending virtual machine; the method includes: The sending front-end thread receives a data transmission instruction triggered by the user, and the data transmission instruction is used to instruct the transmission of specified image data to the receiving virtual machine; The sending front-end thread responds to the data transmission instruction by creating a virtual display device, which captures main screen data and application data. The sending front-end thread also triggers the sending back-end thread to call the graphics processor (GPU) to synthesize the captured data into the specified image data. The main screen data is the data that the operating system running on the sending virtual machine needs to display, and the application data is the data that the user-specified application running on the sending virtual machine needs to display. The sending front-end thread sends a write-back instruction to the sending back-end thread, triggering the sending back-end thread to call the GPU to write back the specified image data to the preset shared video memory; The sending backend thread sends a frame synchronization signal to the receiving virtual machine through the virtual machine inter-virtual machine communication socket vsock, so that the receiving virtual machine reads the specified image data from the shared video memory.

2. The method according to claim 1, characterized in that, Before the sending backend thread invokes the GPU to synthesize the specified image data, the method further includes: The sending backend thread sets resource tags for the target GPU resources used to synthesize the specified image data; The sending backend thread calls the GPU to write the specified image data back to the preset shared video memory, including: The backend thread searches for the target GPU resource marked by the resource tag, and calls the target GPU resource to write the specified image data back to the shared video memory.

3. The method according to claim 2, characterized in that, Before the sending front-end thread receives the data transmission instruction triggered by the user, the method further includes: After the virtual machine on the sending end is initialized, the sending backend thread maps the shared video memory and opens vsock on the host side.

4. The method according to claim 2, characterized in that, The shared video memory includes multiple storage areas; the step of calling the target GPU resources to write the specified image data back to the shared video memory includes: The sending backend thread selects a storage area from the shared video memory and sets the status of the selected storage area to busy state, which indicates that the storage area is being used. The sending backend thread calls the target GPU resource to write the specified image data back to the selected storage area, updates the status of the selected storage area to the filled state, and sets the filling timestamp of the selected storage area to the current time. The filled state indicates that data is stored in the storage area.

5. The method according to claim 4, characterized in that, The sending backend thread selects a storage area from the shared video memory, including: If there are idle storage areas in the shared video memory, the sending backend thread selects an idle storage area from the shared video memory, where idle status indicates that no data is stored in the storage area; or, If there is no free storage area in the shared video memory, the sending backend thread selects the storage area that is filled with the earliest timestamp.

6. The method according to claim 2, characterized in that, Before triggering the sending backend thread to call the GPU to synthesize the specified image data, the method further includes: The sending front-end thread obtains the display frame rate of the receiving virtual machine; Triggering the sending backend thread to call the GPU to synthesize the specified image data includes: The sending front-end thread triggers the sending back-end thread to call the target GPU resources to synthesize the specified image data according to the acquired display frame rate.

7. A screen mirroring method based on virtual machines, characterized in that, The method is applied to a receiving virtual machine, wherein the receiving virtual machine and the sending virtual machine are deployed on the same physical machine, and the receiving virtual machine includes a receiving front-end thread, which is a guest operating system thread within the receiving virtual machine; the method includes: The receiving front-end thread receives the frame synchronization signal sent by the sending virtual machine through the virtual machine inter-virtual machine communication socket vsock. The frame synchronization signal is: after the sending front-end thread of the sending virtual machine responds to the data transmission command triggered by the user, creates a virtual display device so that the virtual display device captures the main screen data and application data, and after the sending front-end thread triggers the sending back-end thread to call the graphics processor (GPU) to synthesize the captured data into specified image data, and calls the GPU to write the specified image data back to the preset shared video memory, the sending back-end thread sends the signal to the receiving virtual machine. The main screen data is the data that the operating system running on the sending virtual machine needs to display, and the application data is the data that the user-specified application running on the sending virtual machine needs to display. The receiving front-end thread responds to the frame synchronization signal by reading the specified image data from the shared video memory.

8. The method according to claim 7, characterized in that, The shared video memory includes multiple storage areas; The receiving front-end thread, in response to the frame synchronization signal, reads the specified image data from the shared video memory, including: The receiving front-end thread selects the storage area with the earliest filled timestamp and sets the status of the selected storage area to busy; the filled state indicates that the storage area stores data, and the busy state indicates that the storage area is being used. The receiving front-end thread reads the specified image data from the selected storage area and sets the status of the selected storage area to an idle state; the idle state indicates that no data is stored in the storage area.

9. The method according to claim 7 or 8, characterized in that, The receiving virtual machine further includes a receiving backend thread; before the receiving frontend thread receives the frame synchronization signal sent by the sending virtual machine via vsock, the method further includes: After the virtual machine at the receiving end is initialized, the receiving backend thread creates the shared video memory; The receiving front-end thread maps the shared video memory and opens vsock on the client side.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method according to any one of claims 1-6 or 7-9.

Images