Screen projection methods and related devices

Abstract

This application discloses a screen mirroring method and related apparatus. The method includes: a first electronic device receiving a first input operation, the first input operation triggering the first electronic device to mirror its screen to a second electronic device; based on the first input operation, the RDMA network card of the first electronic device sending a first request to the RDMA network card of the second electronic device, the first request including first screen mirroring information and the address of a second memory space of the second electronic device, the first request instructing the first screen mirroring information to be written to the second memory space; based on the first request, the RDMA network card of the second electronic device writing the first screen mirroring information to the second memory space; and the second electronic device outputting the first screen mirroring information based on the data stored in the second memory space. This effectively reduces screen mirroring latency, improves screen mirroring smoothness, and enhances the user experience.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to screen projection methods and related devices. Background Technology

[0002] Screen mirroring technology allows the content displayed on a smart terminal or media files (photos, music, and videos) to be projected onto one or more other devices. Screen mirroring technology covers scenarios such as mobile offices, smart cars, and smart home screens. The multi-screen interaction and synergy among multiple smart terminals is a crucial element in establishing a comprehensive ecosystem.

[0003] Currently, wireless screen mirroring requires both devices to encode and decode the content, resulting in significant latency. Poor network conditions or large network fluctuations can also cause screen mirroring lag, leading to a poor user experience. Summary of the Invention

[0004] This application provides a screen mirroring method and related devices, which can effectively reduce screen mirroring latency, improve screen mirroring smoothness, and enhance the user experience.

[0005] In a first aspect, this application provides a screen mirroring method applied to a screen mirroring system, the screen mirroring system including a first electronic device and a second electronic device. The method includes: the first electronic device receiving a first input operation, the first input operation being used to trigger the first electronic device to mirror the screen to the second electronic device; based on the first input operation, the Remote Direct Memory Access (RDMA) network card of the first electronic device sending a first request to the RDMA network card of the second electronic device, the first request including first screen mirroring information and the address of the second memory space of the second electronic device, the first request being used to instruct the first screen mirroring information to be written to the second memory space; based on the first request, the RDMA network card of the second electronic device writing the first screen mirroring information to the second memory space; and the second electronic device outputting the first screen mirroring information based on the data stored in the second memory space.

[0006] Implementing the embodiments of this application, using an RDMA network card, data transmission can be performed directly in user mode. There is no need to copy data to the network software stack or perform multiple context switches between kernel and user modes. The first electronic device can directly write the screen projection information stored in its own memory into the second memory space of the second electronic device. The RDMA network card can provide microsecond-level access latency and high throughput. Thus, after the first electronic device generates screen projection information, the second electronic device can acquire and output the screen projection information in a very short time, effectively reducing screen projection latency and improving screen projection smoothness. Furthermore, using the RDMA network card, the second electronic device can access the screen projection information in the specific memory of the first electronic device without intervention from the first electronic device's central processing unit (CPU), effectively reducing the CPU load on screen projection.

[0007] In one implementation, the first projection information includes image data of the first projection screen; the second electronic device outputs the first projection information based on data stored in the second memory space, including: the display screen of the second electronic device displays the first projection screen based on data stored in the second memory space. By implementing the embodiments of this application, after the first electronic device generates the projection screen, the second electronic device can acquire and display the projection screen in a very short time, effectively reducing projection latency and improving projection smoothness.

[0008] In one implementation, the first projection information includes audio data of the first projection audio; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: the audio output device of the second electronic device outputs the first projection audio based on the data stored in the second memory space. By implementing the embodiments of this application, after the first electronic device generates the projection audio, the second electronic device can acquire and display the projection audio in a very short time, effectively reducing projection latency and improving projection smoothness.

[0009] In one implementation, before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, the method further includes: after the first electronic device detects that it has synthesized and stored the first projection information in the first memory space, the RDMA network card of the first electronic device generates the first request based on the first projection information in the first memory space. Implementing this embodiment combines existing projection information caching mechanisms, such as caching the synthesized projection screen in a display buffer; upon detecting that the projection information is cached in the display buffer, the second electronic device is triggered to promptly write the projection information to the first electronic device, effectively reducing projection latency.

[0010] In one implementation, the first projection information includes image data of the first projection screen; the clocks of the first electronic device and the second electronic device are synchronized. The method further includes: in a first refresh cycle, writing the first projection screen into the first memory space of the first electronic device; in the next refresh cycle, the display screen of the first electronic device displays the first projection screen based on the data stored in the first memory space; the RDMA network card of the second electronic device writes the first projection information into the second memory space, including: within the first refresh cycle, the RDMA network card of the second electronic device writes the first projection information into the second memory space; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: in the next refresh cycle, the display screen of the second electronic device displays the first projection screen based on the data stored in the second memory space. By implementing the embodiments of this application, when the clocks of the first electronic device and the second electronic device are synchronized, after the first electronic device generates projection information, the second electronic device can acquire the projection screen in a very short time before the next refresh cycle arrives and cache it in the display buffer; thus, it can be ensured that the two electronic devices can simultaneously display the projection screen based on the data in the display buffer in the next refresh cycle, effectively improving the user experience.

[0011] In one implementation, after the first electronic device receives the first input operation, and before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, the method further includes: the second electronic device sending a first RDMA message to the first electronic device, the first RDMA message including the address of the second memory space of the second electronic device, the second memory space being used to store screen projection information from the first electronic device. By implementing this embodiment, the second electronic device informs the first electronic device in advance of the address of the second memory space used to store the screen projection information, so that the first electronic device can subsequently use the RDMA network card to promptly write the screen projection information into the second memory space of the second electronic device.

[0012] In one implementation, before the second electronic device sends the first RDMA message to the first electronic device, the method further includes: the second electronic device registering a second memory space with its RDMA network card and obtaining a first key corresponding to the address of the second memory space; the first RDMA message also includes the first key, and the first request also includes the first key; based on the first request, the RDMA network card of the second electronic device writes the first projection information into the second memory space, including: after the RDMA network card of the second electronic device determines that the first key included in the first request is the key corresponding to the address of the second memory space, it writes the first projection information into the second memory space. By implementing this embodiment, the second electronic device registers a second memory space for storing projection information with the RDMA network card in advance to obtain the key corresponding to the second memory space; this key can ensure the security of subsequent data transmission between the first and second electronic devices.

[0013] In one implementation, before the RDMA network card of the second electronic device sends the first RDMA message to the RDMA network card of the first electronic device, the method further includes: based on the first input operation, the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, and the first RDMA message and the first request are sent through the RDMA connection.

[0014] In one implementation, before the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, the method further includes: in response to a first input operation, the first electronic device sends a screen projection request to the second electronic device through a first communication module; the first electronic device receives a screen projection response sent by the second electronic device through the first communication module; the establishment of an RDMA connection between the RDMA network card of the first electronic device and the RDMA network card of the second electronic device includes: when screen projection is determined based on the screen projection response, the RDMA network card of the second electronic device and the RDMA network card of the first electronic device establish an RDMA connection.

[0015] In one implementation, after the second electronic device outputs the first projection information based on the data stored in the second memory space, the method includes: the second electronic device receiving a second input operation; the second electronic device storing a first projection control event corresponding to the second input operation in a third memory space; the RDMA network card of the second electronic device sending a second request to the RDMA network card of the first electronic device, the second request including the data of the first projection control event and the address of the fourth memory space of the first electronic device, the second request being used to instruct the data of the first projection control event to be written into the fourth memory space; based on the second request, the RDMA network card of the first electronic device writing the data of the first projection control event into the fourth memory space; and based on the first projection control event stored in the fourth memory space, the first electronic device executing a first response event. By implementing the embodiments of this application, using the RDMA network card, data transmission can be performed directly in user mode; there is no need to copy data to the network software stack, nor is it necessary to perform multiple context switches between kernel mode and user mode; the second electronic device can directly write the projection control event stored in the second electronic device into the fourth memory space of the first electronic device. In this way, after the second electronic device generates a screen mirroring control event, the first electronic device can obtain the screen mirroring control event in a very short time, and then determine and execute the corresponding response event in the fourth memory space according to the existing mechanism, effectively reducing the screen mirroring control latency and improving the smoothness of screen mirroring. In addition, by using the RDMA network card, the first electronic device can access the screen mirroring control event of a specific memory of the second electronic device without the need for the central processing unit (CPU) of the second electronic device to intervene, effectively reducing the CPU load of screen mirroring control.

[0016] In one implementation, before the RDMA network card of the second electronic device sends the second request to the RDMA network card of the first electronic device, the method further includes: after the second electronic device detects that the first screen projection control event is stored in the third memory space, the RDMA network card of the second electronic device generates the second request based on the data of the first screen projection control event in the third memory space. Implementing this embodiment combines existing input event caching mechanisms, such as caching the screen projection control event in an event buffer; upon detecting that the event is cached in the event buffer, the first electronic device is triggered to promptly write the screen projection control event to the second electronic device, effectively reducing the screen projection control latency.

[0017] In one implementation, before the RDMA network card of the second electronic device sends a second request to the RDMA network card of the first electronic device, the method further includes: the first electronic device sending a second RDMA message to the second electronic device, the second RDMA message including the address of the fourth memory space of the first electronic device, the fourth memory space being used to store input events of the first electronic device. By implementing the embodiments of this application, the first electronic device informs the second electronic device in advance of the address of the fourth memory space used to store input events, so that the second electronic device can subsequently use the RDMA network card to promptly write the screen projection and control events into the fourth memory space of the first electronic device.

[0018] Secondly, this application provides a screen mirroring method applied to a first electronic device, comprising: the first electronic device receiving a first input operation, the first input operation being used to trigger the first electronic device to mirror the screen to a second electronic device; based on the first input operation, the RDMA network card of the first electronic device sending a first request to the RDMA network card of the second electronic device, the first request including first screen mirroring information and the address of the second memory space of the second electronic device, the first request being used to instruct the first screen mirroring information to be written into the second memory space.

[0019] Implementing the embodiments of this application, using an RDMA network card, data transmission can be performed directly in user mode. There is no need to copy data to the network software stack or perform multiple context switches between kernel and user modes. The first electronic device can directly write the screen projection information stored in its own memory into the second memory space of the second electronic device. The RDMA network card can provide microsecond-level access latency and high throughput. Thus, after the first electronic device generates screen projection information, the second electronic device can acquire and output the screen projection information in a very short time, effectively reducing screen projection latency and improving screen projection smoothness. Furthermore, using the RDMA network card, the second electronic device can access the screen projection information in the specific memory of the first electronic device without intervention from the first electronic device's central processing unit (CPU), effectively reducing the CPU load on screen projection.

[0020] In one implementation, the first projection information includes image data of the first projection screen; the first projection information stored in the second memory space is used to display the first projection screen on the display screen of the second electronic device.

[0021] In one implementation, the first projection information includes audio data of the first projection audio; the first projection information stored in the second memory space is used by the audio output device of the second electronic device to output the first projection audio.

[0022] In one implementation, before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, the method further includes: after the first electronic device detects that the first electronic device has synthesized the first projection information and stored it in the first memory space, the RDMA network card of the first electronic device generates the first request based on the first projection information in the first memory space.

[0023] In one implementation, after the first electronic device receives the first input operation and before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, the method further includes: the first electronic device receiving a first RDMA message sent by the second electronic device, the first RDMA message including the address of the second memory space of the second electronic device, the second memory space being used to store screen projection information from the first electronic device.

[0024] In one implementation, before the RDMA network card of the second electronic device sends the first RDMA message to the RDMA network card of the first electronic device, the method further includes: based on the first input operation, the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, and the first RDMA message and the first request are sent through the RDMA connection.

[0025] In one implementation, before the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, the method further includes: in response to a first input operation, the first electronic device sends a screen projection request to the second electronic device through a first communication module; the first electronic device receives a screen projection response sent by the second electronic device through the first communication module; the establishment of an RDMA connection between the RDMA network card of the first electronic device and the RDMA network card of the second electronic device includes: when screen projection is determined based on the screen projection response, the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection.

[0026] In one implementation, after the RDMA network card of the first electronic device sends a first request to the RDMA network card of the second electronic device, the method includes: the RDMA network card of the first electronic device receiving a second request sent by the RDMA network card of the second electronic device, the second request including data of a first screen projection and control event and the address of a fourth memory space of the first electronic device, the second request being used to instruct the data of the first screen projection and control event to be written into the fourth memory space; based on the second request, the RDMA network card of the first electronic device writing the data of the first screen projection and control event into the fourth memory space; based on the first screen projection and control event stored in the fourth memory space, the first electronic device executing a first response event.

[0027] In one implementation, before the RDMA network card of the first electronic device receives the second request sent by the RDMA network card of the second electronic device, the method further includes: the first electronic device sending a second RDMA message to the second electronic device, the second RDMA message including the address of the fourth memory space of the first electronic device, the fourth memory space being used to store input events.

[0028] Thirdly, this application provides a screen mirroring method, characterized in that it is applied to a second electronic device, comprising: the RDMA network card of the second electronic device receiving a first request sent by the RDMA network card of the first electronic device, the first request including first screen mirroring information and the address of the second memory space of the second electronic device, the first request being used to instruct the first screen mirroring information to be written into the second memory space; based on the first request, the RDMA network card of the second electronic device writing the first screen mirroring information into the second memory space; and the second electronic device outputting the first screen mirroring information based on the data stored in the second memory space.

[0029] Implementing the embodiments of this application, using an RDMA network card, data transmission can be performed directly in user mode. There is no need to copy data to the network software stack or perform multiple context switches between kernel and user modes. The first electronic device can directly write the screen projection information stored in its own memory into the second memory space of the second electronic device. The RDMA network card can provide microsecond-level access latency and high throughput. Thus, after the first electronic device generates screen projection information, the second electronic device can acquire and output the screen projection information in a very short time, effectively reducing screen projection latency and improving screen projection smoothness. Furthermore, using the RDMA network card, the second electronic device can access the screen projection information in the specific memory of the first electronic device without intervention from the first electronic device's central processing unit (CPU), effectively reducing the CPU load on screen projection.

[0030] In one implementation, the first projection information includes image data of the first projection screen; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: the display screen of the second electronic device displays the first projection screen based on the data stored in the second memory space.

[0031] In one implementation, the first projection information includes audio data of the first projection audio; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: the audio output device of the second electronic device outputs the first projection audio based on the data stored in the second memory space.

[0032] In one implementation, before the RDMA network card of the second electronic device receives the first request sent by the RDMA network card of the first electronic device, the method further includes: the second electronic device sending a first RDMA message to the first electronic device, the first RDMA message including the address of the second memory space of the second electronic device, the second memory space being used to store screen projection information from the first electronic device.

[0033] In one implementation, before the second electronic device sends the first RDMA message to the first electronic device, the method further includes: the second electronic device registering the second memory space to its RDMA network card and obtaining the first key corresponding to the address of the second memory space; the first RDMA message also includes the first key, and the first request also includes the first key; based on the first request, the RDMA network card of the second electronic device writes the first projection information into the second memory space, including: after the RDMA network card of the second electronic device determines that the first key included in the first request is the key corresponding to the address of the second memory space, it writes the first projection information into the second memory space.

[0034] In one implementation, before the RDMA network card of the second electronic device sends the first RDMA message to the RDMA network card of the first electronic device, the method further includes: establishing an RDMA connection between the RDMA network card of the second electronic device and the RDMA network card of the first electronic device, and sending the first RDMA message and the first request through the RDMA connection.

[0035] In one implementation, after the second electronic device outputs the first projection information based on the data stored in the second memory space, the method includes: the second electronic device receiving a second input operation; the second electronic device storing the first projection reverse control event corresponding to the second input operation in a third memory space; the RDMA network card of the second electronic device sending a second request to the RDMA network card of the first electronic device, the second request including the data of the first projection reverse control event and the address of the fourth memory space of the first electronic device, the second request being used to instruct the data of the first projection reverse control event to be written into the fourth memory space, the fourth memory space being used to store the input event of the first electronic device.

[0036] In one implementation, before the RDMA network card of the second electronic device sends the second request to the RDMA network card of the first electronic device, the method further includes: after the second electronic device detects that the first screen projection and control event is stored in the third memory space, the RDMA network card of the second electronic device generates the second request based on the data of the first screen projection and control event in the third memory space.

[0037] In one implementation, before the RDMA network card of the second electronic device sends a second request to the RDMA network card of the first electronic device, the method further includes: the second electronic device receiving a second RDMA message sent by the first electronic device, the second RDMA message including the address of the fourth memory space of the first electronic device.

[0038] Fourthly, this application provides an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, including computer instructions. When the one or more processors execute the computer instructions, they cause a communication device to perform the screen projection method in any of the possible implementations of the second aspect described above.

[0039] Fifthly, this application provides another electronic device, including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, including computer instructions, which, when executed by the one or more processors, cause the communication device to perform the screen projection method in any possible implementation of the third aspect described above.

[0040] Sixthly, embodiments of this application provide a computer storage medium including computer instructions, which, when executed on an electronic device, cause a communication device to perform the screen projection method in any of the possible implementations of any of the above aspects.

[0041] Seventhly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to execute the screen projection method in any of the possible implementations of any of the above aspects. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the communication system provided in an embodiment of this application;

[0043] Figure 2A A schematic diagram of an in-vehicle display screen provided in an embodiment of this application;

[0044] Figure 2B A schematic diagram of the vehicle-mounted device provided in the embodiments of this application;

[0045] Figures 2C to 2F A schematic diagram illustrating a screen projection scenario provided in an embodiment of this application;

[0046] Figures 3A to 3C A schematic diagram illustrating a screen projection scenario provided in an embodiment of this application;

[0047] Figure 4A schematic diagram illustrating a screen projection scenario provided in an embodiment of this application;

[0048] Figure 5A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0049] Figure 5B A software structure block diagram of an electronic device provided in an embodiment of this application;

[0050] Figure 6A This is a schematic flowchart of a screen sharing method provided in an embodiment of this application;

[0051] Figure 6B A flowchart illustrating the implementation of screen sharing combined with a software structure, provided for an embodiment of this application;

[0052] Figure 6C This is a schematic flowchart of an audio sharing method provided in an embodiment of this application;

[0053] Figure 7A This is a schematic flowchart of a screen mirroring and reverse control method provided in an embodiment of this application;

[0054] Figure 7B A flowchart illustrating the implementation of screen projection and reverse control combined with a software structure is provided in this application embodiment;

[0055] Figure 8 A schematic diagram of an RDMA software stack provided in an embodiment of this application;

[0056] Figure 9A This is a schematic flowchart of a screen sharing method provided in an embodiment of this application;

[0057] Figure 9B A flowchart illustrating the implementation of screen sharing combined with a software structure, provided for an embodiment of this application;

[0058] Figure 9C A flowchart illustrating an implementation of RDMA provided in this application embodiment;

[0059] Figure 9D A timing diagram of a refresh cycle provided for an embodiment of this application;

[0060] Figure 9E This is a schematic flowchart of an audio sharing method provided in an embodiment of this application;

[0061] Figure 9F A flowchart illustrating an implementation of audio sharing combined with a software structure, provided as an embodiment of this application;

[0062] Figure 10A This is a schematic flowchart of a screen mirroring and reverse control method provided in an embodiment of this application;

[0063] Figure 10B A flowchart illustrating the implementation of screen projection and reverse control combined with a software structure is provided in this application embodiment;

[0064] Figure 10C This is a schematic flowchart of an audio sharing method provided in an embodiment of this application;

[0065] Figure 11 This is a flowchart illustrating a screen projection method provided in an embodiment of this application. Detailed Implementation

[0066] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0067] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0068] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0069] The following describes the communication system 10 provided in the embodiments of this application.

[0070] Figure 1 A schematic diagram of the structure of a communication system 10 provided in an embodiment of this application is shown as an example. Figure 1 As shown, the communication system 10 includes an electronic device 100, and one or more electronic devices, such as electronic device 200, connected to the electronic device 100.

[0071] In this embodiment, electronic device 100 can use screen mirroring technology to send specific content (e.g., the display content of a physical screen, the user interface of a specific application, or media files) to one or more connected electronic devices, such as electronic device 200. Electronic device 200 can then perform interface display and / or audio output based on the data sent by electronic device 100. On the electronic device 100 side, the user can locally control the screen mirroring content from electronic device 100 to electronic device 200; on the electronic device 200 side, the user can also control electronic device 100 in reverse through the aforementioned screen mirroring content. In some embodiments, electronic device 100 is configured with a system-on-chip (SOC) 1, which supports the composition, rendering, and display of the screen image of electronic device 100; electronic device 200 is configured with a SOC 2, which also supports the composition, rendering, and display of the screen image of electronic device 200. Using the screen mirroring method provided in this embodiment, electronic device 100 and electronic device 200 can achieve cross-chip screen viewing and device collaboration.

[0072] In some embodiments, electronic device 100 can be directly connected to electronic device 200 via a local wired connection. In one implementation, electronic device 100 is configured with RDMA network card 1, and electronic device 200 is configured with RDMA network card 2. RDMA network card 1 of electronic device 100 and RDMA network card 2 of electronic device 100 establish a wired connection based on the RDMA network protocol (e.g., Ethernet connection / IB connection). In some embodiments, the electronic devices connect to the RDMA network cards via a peripheral component interconnect express (PCIe) bus. The PCIe protocol is an end-to-end interconnect protocol that provides a solution for high-speed transmission bandwidth.

[0073] In some embodiments, electronic device 100 can be directly connected to electronic device 200 via a near-field communication (NFC) connection. For example, electronic devices 100 and 200 may have one or more NFC communication modules, such as a near-field communication (NFC) module, a wireless fidelity (WiFi) module, an ultra-wideband (UWB) module, a Bluetooth module, or a ZigBee module. Taking electronic device 100 as an example, electronic device 100 can use a NFC module (e.g., an NFC module) to emit signals to detect and scan nearby electronic devices, enabling electronic device 100 to discover nearby electronic devices (e.g., electronic device 200) via a near-field communication protocol, establish a wireless communication connection with the nearby electronic device, and transmit data to the nearby electronic device. For example, electronic device 100 and electronic device 200 can be connected via WiFi-direct.

[0074] In some embodiments, electronic devices 100 and 200 can be connected to a local area network (LAN) via electronic device 300 using a wired or WiFi connection. For example, electronic device 300 can be a third-party device such as a router, gateway, or smart device controller. In some embodiments, electronic devices 100 and 200 can also be indirectly connected via at least one electronic device 400 in a wide area network (WAN). For example, electronic device 400 can be a hardware server or a cloud server embedded in a virtualization environment. It is understood that through electronic devices 300 and / or 400, electronic device 100 and electronic device 200 can indirectly establish wireless communication connections and transmit data.

[0075] It is understood that the structure shown in this embodiment does not constitute a specific limitation on the communication system 10. In other embodiments of this application, the communication system 10 may include more or fewer devices than those shown in the figure.

[0076] This application does not specifically limit the types of electronic devices 100 and 200. For example, the electronic devices 100 and 200 involved in this application embodiment can be mobile phones, wearable devices (e.g., smart bracelets, smartwatches), tablets, laptops, handheld computers, in-vehicle devices, smart TVs, ultra-mobile personal computers (UMPCs), cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc. In some embodiments, electronic device 200 can also be a projection device such as a projector.

[0077] In one exemplary scenario, electronic device 100 and electronic device 200 belong to the same device system (e.g., in-vehicle device). For example, electronic device 100 and electronic device 200 are both in-vehicle devices of a smart car. With the development of smart cars, the number of in-vehicle displays is increasing, including multiple screens such as instrument panels, AR-HUDs, central control screens, passenger-side screens, rear-seat screens (e.g., left and right rear-seat screens), and window screens. For example, Figure 2A The image shows multiple screens in a smart car. It's difficult for a single SoC's GPU computing power to handle the compositing and rendering of all screen images, leading some manufacturers to introduce two or more SoCs working together for processing.

[0078] For example, such as Figure 2B As shown, taking two SOCs (i.e., SOC1 and SOC2) as an example, the aforementioned in-vehicle equipment may include electronic device 100 and electronic device 200. Electronic device 100 includes SOC1 and an in-vehicle display screen (e.g., a passenger-side screen) controlled by SOC1, and electronic device 200 includes SOC2 and an in-vehicle display screen (e.g., a rear-seat screen) controlled by SOC2. Users can project the screen content of the display screen controlled by SOC1 onto one or more displays screens controlled by SOC2.

[0079] For example, consider projecting the passenger-side screen controlled by SOC1 onto the rear-seat screen controlled by SOC2. Figures 2C to 2F This demonstrates an interface implementation for screen mirroring between in-vehicle displays.

[0080] like Figure 2C As shown, the passenger-side screen plays video 1 on the user interface 11 of APP1; after detecting the user's input operation 1, it displays on the passenger-side screen. Figure 2DThe screen mirroring selection interface 12 shown is used to select the screen mirroring content, target device, and / or target screen. In one implementation, such as... Figure 2C As shown, the user interface 11 displays a projection icon 101, and the input operation 1 includes a click operation on the projection icon 101. This embodiment does not specifically limit the input operation 1; it can also be applied to other in-vehicle displays, and may include other touch operations, voice operations, or gesture operations.

[0081] like Figure 2D As shown, the screen casting selection interface 12 includes a list of screens that can be cast to in this vehicle. This screen list may include options for any display screen other than the passenger-side screen, such as option 102 for the right rear screen. Users can select one or more display screens from the screen list for casting.

[0082] like Figure 2D As shown, the screen casting selection interface 12 may also include a list of devices that can be cast to the screen. The device list includes options for other detected devices that can be cast to the screen, such as the option 103 for mobile phones. Users can select one or more devices from the screen list to cast to the screen.

[0083] like Figure 2D As shown, the screen casting selection interface 12 may also include a list of content that can be cast. The list includes options 104 corresponding to the display screen (i.e., the main screen image) of the passenger screen's main screen (i.e., the physical screen), and may also include options corresponding to some applications installed on the passenger screen (e.g., option 105 for APP2). If the user selects option 104 for the main screen image, the current main screen image of the passenger screen will be cast to the target device and / or the target screen. If the user selects option 105 for APP2, the SOC1 will run APP2 on the virtual display screen and cast the display content of the virtual display screen to the target device and / or the target screen. That is, the cast content and the main screen image of the passenger screen can be different, and it will not affect the normal display of the passenger screen.

[0084] like Figure 2D and Figure 2E As shown, the screen mirroring selection interface 12 may also include a confirmation control 106. After the user selects the option 102 for the right rear screen and the option 104 for the main screen, the electronic device 100 detects that the user clicks the confirmation control 106. In response to the user's click operation, the electronic device 100 sends screen mirroring data to the electronic device 200. Based on the above screen mirroring data, the electronic device 200 displays the screen mirroring image on the right rear screen, which is the main screen image of the passenger side screen. During screen mirroring, the right rear screen may only display the screen mirroring image, or it may display the screen mirroring image as a small window overlaid on the original display interface; no specific limitation is made here.

[0085] It is understandable that when the main screen display for the passenger-side control changes, the screen displayed on the right rear-seat screen also changes accordingly; through this screen display, the user can also control the passenger-side screen from the right rear-seat screen. For example, such as... Figure 2F As shown, the system detects a pause operation performed by the user on the projected screen displayed on the right rear screen (e.g., a click operation on the video playback area of ​​video 1 in user interface 11); in response to the above pause operation, the electronic device 100 corresponding to the passenger screen controls the video 1 to pause, and both the passenger screen and the right rear screen display a pause interface 13, which indicates that video 1 is currently paused.

[0086] In another exemplary scenario, electronic device 100 is a mobile phone, and electronic device 200 is a large-screen device (such as an in-vehicle device). The mobile phone projects its user interface onto the large-screen device, and the mobile phone's user interface displayed on the large-screen device can be used to control the mobile phone in reverse.

[0087] For example, Figure 3A As shown, the mobile phone detects the user's screen mirroring operation, which triggers screen mirroring to the vehicle's central control screen. In response to this operation, the mobile phone sends a screen mirroring request to the vehicle's central control screen. Based on this request, the vehicle's central control screen displays a screen mirroring prompt, which prompts the user to confirm whether to allow screen mirroring. The prompt includes a confirmation control 201 and a cancellation control 202. The confirmation control 201 confirms screen mirroring, and the cancellation control 202 rejects it. (Reference) Figures 2C to 2E Regarding the relevant descriptions, users can also select the target device, target screen, and / or content to be cast on their mobile phones, which will not be elaborated here. No specific restrictions are placed on the user's actions to trigger screen casting.

[0088] like Figure 3A and Figure 3B As shown, after the vehicle-mounted device detects that the user clicks the confirmation control 201, it displays the screen to be projected based on the screen projection data sent by the mobile phone. The screen projection data includes the user interface 11 currently displayed on the mobile phone. In some embodiments, no user confirmation is required, i.e., no screen projection prompt box needs to be displayed; the vehicle-mounted device can directly display the screen projection data sent by the mobile phone.

[0089] like Figure 3B and Figure 3C As shown, the user interface 11 includes an icon 203 for the gallery application; after the in-vehicle device detects that the user clicks on the icon 203 in the projection screen, the mobile phone displays the user interface 23 of the gallery application, and the in-vehicle device also displays the user interface 23 of the gallery application in the projection screen. That is, the user controls the mobile phone to open the gallery application from the in-vehicle device side.

[0090] In another exemplary scenario, the screen projection content sent by electronic device 100 to electronic device 200 has undergone immersive processing. Immersive processing refers to performing corresponding image processing on the screen projection image according to the screen projection effect selected by the user (such as cropping, selection, color changing, adding filters, etc.). No specific limitations are made here regarding the aforementioned screen projection effect and the corresponding image processing method.

[0091] For example, Figure 4 This illustrates a projection scenario with immersive processing. For example... Figure 4 As shown, electronic device 100 projects its screen onto electronic devices 200 and 500. The projected images displayed on electronic devices 500 and 200 are images of the wallpaper on electronic device 100 after cropping, color changing, and transparency modification. After projection, the combined screens of electronic devices 200, 100, and 500 can create a specific visual effect.

[0092] The structure of the electronic device involved in the embodiments of this application will be described by way of example below.

[0093] Figure 5A A schematic diagram of the structure of electronic device 100 is shown. The structure of electronic device 200 can be referred to the relevant description of electronic device 100. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0094] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0096] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0097] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0098] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0099] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0100] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0101] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0102] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0103] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0104] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0105] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0106] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0107] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0108] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0109] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0110] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0111] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0112] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0113] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, frequency modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0114] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0115] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0116] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0117] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0118] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0119] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0120] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0121] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0122] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0123] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0124] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and double data rate synchronous dynamic random access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM). Non-volatile memory can include disk storage devices and flash memory.

[0125] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0126] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0127] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0128] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0129] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0130] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0131] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0132] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0133] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C.

[0134] The 170D headphone jack is used to connect wired headphones.

[0135] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0136] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B.

[0137] The 180C barometric pressure sensor is used to measure barometric pressure.

[0138] The magnetic sensor 180D includes a Hall sensor.

[0139] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0140] Distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser.

[0141] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode.

[0142] The 180L ambient light sensor is used to detect ambient light intensity.

[0143] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the characteristics of the collected fingerprints to unlock the device.

[0144] The 180J temperature sensor is used to detect temperature.

[0145] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0146] The bone conduction sensor 180M can acquire vibration signals.

[0147] Button 190 includes the power button, volume buttons, etc. Button 190 can be a mechanical button or a touch button.

[0148] Motor 191 can generate vibration alerts.

[0149] Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, etc.

[0150] The SIM card interface 195 is used to connect the SIM card.

[0151] The software structure of the electronic device 100 and electronic device 200 involved in the embodiments of this application will be described below.

[0152] In this embodiment, the electronic device may run iOS, Android, Microsoft, or other software systems; no specific limitation is made to the software system here. The software system of the electronic device may adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. The following uses electronic device 100 and a layered Android system as an example to illustrate the software structure of the electronic device. The software structure of electronic device 200 can be referred to the relevant description of electronic device 100.

[0153] Figure 5B This is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0154] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: Application Layer, Application Framework Layer, Hardware Abstraction Layer (HAL), Native Layer, and Kernel Layer. In one implementation, layers above the Kernel belong to the user space of the software system, while the Kernel itself belongs to the kernel space.

[0155] The application layer can include a series of application packages. For example... Figure 5B As shown, the application package may include screen mirroring applications, video applications, and may also include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, and SMS.

[0156] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0157] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0158] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0159] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0160] The Native layer consists of local services and libraries. From a performance perspective, applications (APPs) need to use C / C++ implementations in the Native layer. The Native layer then provides these implementations via JNI to the Java layer, which handles basic application functionality, to extend computationally intensive features.

[0161] The kernel layer is the layer between hardware and software. It provides low-level drivers for various hardware components, such as WiFi drivers, Bluetooth drivers, mobile communication drivers, display drivers, audio drivers, keyboard drivers, power drivers, and sensor drivers (e.g., touch sensor drivers). It is also responsible for file management, process management, network protocol stacks, system security, and memory management.

[0162] The HAL layer is the interface layer between the kernel and hardware circuits. Its purpose is to abstract the hardware and provide a virtual hardware platform and services for the operating system. It transforms the calls from the upper layer into direct access to and control of the hardware, and its specific activity is to create hardware-related drivers.

[0163] In some embodiments, such as Figure 5B As shown, the application framework layer may include WindowManagerService (WMS), the Native layer includes Surfaceflinger, and the HAL layer includes the hardware compositor (hwcomposer, HWC).

[0164] WMS manages window programs, controlling the entire lifecycle, hierarchy, and attributes of window displays. For example, WMS handles window startup, addition, and deletion; obtains the screen size; determines if a status bar is present; locks the screen; and captures screenshots. SurfaceFlinger composites multiple rendered layers from WMS and then sends the composited image to the display device for display. After collecting all buffers for visible layers, SurfaceFlinger queries HWC for compositing instructions. Layer compositing methods include GPU compositing and HWC compositing; GPU compositing uses OpenGL to composite layers, while HWC compositing uses the display controller. HWC compositing, performed via hardware, reduces the compositing load on the GPU.

[0165] For example, when an application (e.g., APP1) launches an Activity, it calls WMS to create and register an application window corresponding to that Activity. When APP1 displays this window in the foreground, WMS instructs SurfaceFlinger to create a surface (layer) for that window. Based on the graphics drawing instructions from APP1, WMS calls the CPU / GPU to draw and render each layer, and caches them into a buffer queue. SurfaceFlinger receives the layer data from the buffers of all visible layers and then performs compositing. For example, the layers displayed on the screen typically include the status bar at the top of the screen, the system bar at the bottom of the screen, and the UI parts of the app (i.e., the application interface, icons, and / or widgets), and may also include the background wallpaper.

[0166] In some embodiments, such as Figure 5B As shown, the application framework layer may include mediacodec, the native layer includes a video encoder and a video decoder, and the HAL layer includes videocodec. The native layer's video encoder calls videocodec to encode the synthesized image and obtain the encoded video stream; correspondingly, the video decoder calls videocodec to decode the encoded video stream. MediaCodec is a class in the application framework layer used for audio and video encoding and decoding, including encoder / decoder components. It implements encoding and decoding functions by accessing the underlying codec.

[0167] In some embodiments, such as Figure 5B As shown, the application framework layer may include an audio recording module (Audiorecoder), the Native layer includes an Audiooflinger, an audio encoder, and an audio decoder, and the HAL layer includes an Audio HAL (Audio HAL). The audio recording module is used to record audio files output by the application (e.g., APP1) and obtain the recorded audio. The Audiooflinger in the Native layer is used to synthesize the audio sent from the application layer; the audio encoder is used to call Audiocodec to encode the synthesized audio; and correspondingly, the audio decoder is used to call Audiocodec to decode the encoded audio.

[0168] In some embodiments, such as Figure 5BAs shown, the Native layer includes the input subsystem, and the HAL layer includes I / O device nodes (e.g., / dev / input / eventX). Android offers various input devices, such as screens, keyboards, mice, game controllers, joysticks, etc. When an input device is available, the Linux kernel creates a corresponding I / O device node in / dev / input; different input devices correspond to different I / O device nodes. When an input operation is performed through an input device (e.g., the screen), the driver for that input device (e.g., the touch sensor driver) processes the corresponding input information (e.g., hardware interrupts) into raw input events (including touch coordinates, touch timestamps, etc.), and stores them in the corresponding I / O device node. The Input subsystem can retrieve the raw input events from the I / O device node; these events are then encapsulated and submitted to the target window for processing.

[0169] In some embodiments, such as Figure 5B As shown, the kernel layer may also include a driver that drives the communication module 1 to work; for example, the communication module 1 includes one or more of the following: a short-range wireless communication module (e.g., a WiFi communication module, a Bluetooth communication module, or a UWB communication module), a mobile communication module, and a wired communication module. Correspondingly, the kernel layer may include one or more of the following: a WiFi driver, a Bluetooth driver, a UWB driver, a mobile communication driver, etc.

[0170] The following section provides a detailed description of the two screen projection implementation schemes provided in this application, based on the aforementioned software structure.

[0171] In this embodiment, the screen projection scheme includes two parts: screen sharing and screen projection control. The specific implementations of screen sharing and screen projection control in the two schemes are described below. For ease of description, in subsequent embodiments, electronic device 100 can be referred to as the source device for screen projection, and electronic device 200 as the target device for screen projection.

[0172] Option 1: Screen mirroring solution based on encoding and decoding

[0173] For example, Figure 6A This paper presents a screen sharing method and process for this solution. The following section combines... Figure 6B The screen sharing implementation process shown is described below. The method flow of screen sharing includes steps S101 to S109.

[0174] S101, Electronic device 100 displays the user interface 1 of APP1.

[0175] In some embodiments, before the electronic device 100 projects its screen onto the electronic device 200, the electronic device 100 runs and displays the user interface 1 of the APP 1. Step S101 includes: the APP 1 sends a drawing instruction to the WMS; the WMS draws and renders the display content of each layer of the user interface 1 of the APP 1 based on the drawing instruction, wherein the drawing instruction can indicate the view corresponding to each layer of the user interface, and the attributes of each view (e.g., size, position, transparency); the WMS instructs the surfaceflinger to composite each layer; the surfaceflinger calls the GPU compositing and HWC compositing to obtain the composited user interface 1; the HWC caches the composited user interface 1 in a preset buffer; the display driver sends the user interface 1 to the physical display screen of the electronic device 100 based on the data in the buffer.

[0176] In this embodiment of the application, APP1 can be a third-party application (e.g., a video application) or a system application (e.g., a desktop application), and no specific limitation is made here.

[0177] S102. After receiving the user's screen projection operation, the electronic device 100 stores the screen projection trigger event corresponding to the screen projection operation in its IO device node (i.e., / dev / input / eventX).

[0178] In some embodiments, a user triggers screen projection by performing a touch operation on the display screen. Step S102 includes: (1) after the touch sensor in the physical display screen detects the screen projection operation, it sends the hardware awareness information corresponding to the screen projection operation to the touch sensor driver; (2) the touch sensor driver processes the hardware awareness information into a raw input event (e.g., the raw input event includes touch coordinates, touch timestamps, etc.) and stores it in the IO device node corresponding to the display screen (e.g., / dev / input / eventX). For ease of description, the above raw input event can be referred to as the screen projection trigger event.

[0179] For example, see Figure 2C As described in the example, the screen mirroring operation may include a click operation on the confirmation control 106 displayed on the screen. It is not limited to touch operations on the screen; the screen mirroring operation involved in this application embodiment may also be an input operation performed through other input devices (such as a mouse or remote control), or it may be other forms of input operation such as voice operation or gesture operation, which are not specifically limited here.

[0180] S103, the screen mirroring trigger event in the IO device node of electronic device 100 is distributed to the screen mirroring application.

[0181] In some embodiments, step S103 includes: (3) the input subsystem reads the original input event (i.e., the screen casting trigger event) from the IO device node; (4) the input subsystem distributes the screen casting trigger event to the screen casting application. In one implementation, after the IMS detects the screen casting trigger event output by the input subsystem, it calls the WMS to determine the target window corresponding to the touch event based on the layout of each window in the main screen. The target application corresponding to the screen casting trigger event is the screen casting application to which the target window belongs.

[0182] S104, The screen casting application instructs to draw and synthesize the screen casting screen 1 to the electronic device 200, and the synthesized screen casting screen 1 is cached in the display buffer 1.

[0183] In some embodiments, the aforementioned screen mirroring trigger event indicates that the target device for screen mirroring is electronic device 200. Electronic device 100 creates a virtual display screen 1 corresponding to electronic device 200, and virtual display screen 1 is used to run the screen mirrored onto electronic device 200. When the user selects multiple target devices for screen mirroring, a virtual display screen is created for each target device. In some embodiments, the screen mirroring content is the main screen of electronic device 100, and it is not necessary to create a virtual display screen. The main screen of electronic device 100 can also be mirrored onto electronic device 200, for example, using screen mirroring technology.

[0184] In some embodiments, step S104 includes: (5) the screen mirroring application of electronic device 100 establishes a service connection with the screen mirroring application of electronic device 100; (6) the screen mirroring application instructs WMS to create a virtual display screen 1 corresponding to electronic device 200; (7) WMS instructs surfaceflinger to create virtual display screen 1; (8) surfaceflinger calls GPU synthesis and HWC synthesis to synthesize the screen mirroring image 1 corresponding to virtual display screen 1. HWC stores the synthesized screen mirroring image 1 in a preset buffer (i.e., display buffer 1). For ease of description, the buffer in the memory space of electronic device 100 used to store the screen mirroring image of electronic device 200 (e.g., the image to be displayed on virtual display screen 1) can be simply referred to as the display buffer of electronic device 100 (e.g., display buffer 1). In some embodiments, the display image of electronic device 100 is the screen mirroring image projected onto electronic device 200, and the above-mentioned display buffer 1 is used to cache the image to be displayed of electronic device 100; electronic device 100 displays the image based on the image data in the display buffer 1 in the next refresh cycle.

[0185] In some embodiments, reference Figures 2C to 2EAccording to the relevant description, a user-triggered screen mirroring event can indicate one or more of the target device, target screen, and screen mirroring content. The electronic device 100 creates a corresponding virtual display screen based on the target device selected by the user, and determines the display content (i.e., the screen mirroring image) of the virtual display screen based on the screen mirroring content selected by the user.

[0186] In some embodiments, when the screen projection content selected by the user is the main screen of the electronic device 200, and the main screen of the electronic device 200 includes the user interface 1 of the APP 1, the screen projection screen 1 includes the user interface 1; when the screen projection content selected by the user is not the main screen of the electronic device 200, the screen projection screen 1 and the main screen of the electronic device 200 may be different.

[0187] S105, electronic device 100 encodes the screen projection screen 1 and obtains screen projection data 1.

[0188] In some embodiments, step S105 includes: (8) the video encoder obtains the image data of the projection screen 1 in the display buffer 1; (9) the video encoder calls videocode to encode the projection screen and obtain the encoded video stream; (10) videocodec sends the encoded video stream to mediacodec; (11) mediacodec sends the encoded video stream to the projection application, and the projection application encrypts and packages the data to obtain projection data 1.

[0189] S106. After encapsulating the projection data 1 using the communication protocol, the electronic device 200 sends the encapsulated projection message 1 through the communication module 1.

[0190] In some embodiments, the communication module 1 can be a short-range communication module (e.g., a WiFi communication module, a Bluetooth communication module), a mobile communication module, or other wireless communication modules. Taking the WiFi communication module as an example, step S105 includes: (12) the screen-casting application sends screen-casting data 1 to the TCP / IP protocol stack; (13) the TCP / IP protocol stack encapsulates the screen-casting data 1 into a screen-casting message 1 using the TCP / IP protocol and then sends the screen-casting message 1 to the WiFi driver; (14) the WiFi driver sends the screen-casting message 1 to the WiFi communication module; (15) the WiFi communication module sends the screen-casting message 1 to the electronic device 200. It can be understood that the screen-casting message 1 includes the screen-casting data 1.

[0191] Not limited to the TCP / IP protocol, the embodiments of this application can also send screen projection messages through other communication protocols, which are not specifically limited here.

[0192] S107. After decapsulating the screen projection message 1, the screen projection application of the electronic device 200 obtains the screen projection data 1.

[0193] In some embodiments, taking the WiFi communication module as an example, step S106 includes: (16) the WiFi driver obtains the screen projection message 1 through the WiFi communication module; (17) the WiFi driver sends the screen projection message 1 to the TCP / IP protocol stack; (18) the TCP / IP protocol stack decapsulates the screen projection data 1 using the TCP / IP protocol and sends the decapsulated screen projection data 1 to the screen projection application.

[0194] S108, electronic device 200 decodes the projection data 1 and obtains the decoded projection screen 1.

[0195] In some embodiments, step S108 includes: (19) the screen casting application concatenates and decrypts the screen casting data 1 and sends it to Mediacodec; (20) Mediacodec decodes the data sent by the screen casting application and sends it to the video decoder; (21) the video decoder calls Videocodec to decode the data sent by Mediacodec and obtains the image data of the screen casting screen 1; (22) Videocodec sends the image data of the screen casting screen 1 to the screen casting application.

[0196] S109, the electronic device 200 stores the projected image 1 into the display buffer 2, and the physical display screen displays the projected image 1 according to the image data of the projected image 1 cached in the display buffer 2.

[0197] In some embodiments, step S109 includes: (23) the projection application instructs the WMS to display projection screen 1 in the projection window; (24) the WMS instructs the surfaceflinger to perform layer compositing; (25) the surfaceflinger calls the GPU compositing and HWC compositing; (26) the HWC stores the composited projection screen 1 in the display buffer 2; (27) the display driver sends the projection screen 1 to the physical display screen of the electronic device 200 based on the image data in the display buffer 2. In some embodiments, the buffer in the memory space of the electronic device 200 used to store the projection screen from the electronic device 100 can be simply referred to as the display buffer (e.g., display buffer 2) of the electronic device 200, and the physical display screen of the electronic device 200 displays the projection screen 1 based on the data in the display buffer 2. In some embodiments, the display buffer 2 is used to cache the latest composited image output by the SF / HWC of the electronic device 200, that is, the image to be displayed on the physical display screen of the electronic device 200 in the next refresh cycle.

[0198] In some embodiments, the electronic device 200 can simultaneously display the projected image and the interfaces of other applications installed on the electronic device 200; SurfaceFlinger collects the buffers of all visible layers for image compositing, and the buffers of visible layers include the display buffer 2 corresponding to the projected image 1. In this case, the composited display image includes the projected image 1 and the interfaces of other applications; the display driver sends the above display image to the physical display screen of the electronic device 200.

[0199] In some embodiments, the screen projection sent by electronic device 100 to electronic device 200 is playing audio and video. During the projection process, electronic device 100 also sends audio corresponding to the screen projection to electronic device 200. For ease of description, this audio can be referred to as projection audio. See also Figure 6C The screen projection solution provided in this application embodiment also includes audio sharing, and the audio sharing method includes steps S110 to S115.

[0200] S110, the screen mirroring application of electronic device 100 is instructed to synthesize screen mirroring audio; the audio data of the synthesized screen mirroring audio 1 is stored in audio buffer 1.

[0201] In some embodiments, (28) the audio recording module records the audio file sent by the upper-layer application (e.g., APP1), stores the recorded audio in a buffer, and notifies Audiooflinger; (29) Audiooflinger synthesizes the projection audio 1 based on the audio data of one or more buffers; Audio HAL obtains the synthesized projection audio 1 and stores it in audio buffer1. For ease of description, the buffer in the memory space of electronic device 100 used to store the projection audio of electronic device 200 (e.g., the audio to be output from virtual display screen 1) can be simply referred to as the audio buffer of electronic device 100 (e.g., audio buffer1). In some embodiments, the output audio of electronic device 100 is the projection audio sent to electronic device 200, and the aforementioned audio buffer1 is used to buffer the audio to be output from electronic device 100.

[0202] In some embodiments, similar to the display buffer, the audio buffer of the electronic device 200 may also include multiple buffers, with audio buffer 2 being one of these multiple buffers. For example, the most recently acquired screen-casting audio (e.g., screen-casting audio 1) can be buffered in a polling manner into one of the multiple buffers according to the address order of the buffers. The audio driver sends the most recently buffered screen-casting audio to the audio output device.

[0203] S111, Electronic device 100 encodes the screen-casting audio 1 in audio buffer 1 and obtains screen-casting data 2.

[0204] In some embodiments, (30) the audio encoder obtains the screen-casting audio 1 in the audio buffer 1; (31) the audio codec is called to encode the screen-casting audio 1; and (32) the encoded screen-casting data 2 is sent to the screen-casting application through the media codec.

[0205] S112. After encapsulating the projection data 2 using the communication protocol, the electronic device 100 sends the encapsulated projection message 2 through the communication module 1.

[0206] S113. After the electronic device 200 decapsulates the screen projection message 2, the screen projection application of the electronic device 200 obtains the screen projection data 2.

[0207] For details on the implementation of steps S112 and S113, please refer to the descriptions of steps S106 and S107, which will not be repeated here.

[0208] S114, Electronic device 200 decodes the screen projection data 2 and obtains the screen projection audio 1.

[0209] In some embodiments, on the electronic device 200 side, (33) the screen casting application sends the screen casting data 2 to the audio decoder of the Native layer through Mediacodec; (34) the audio decoder calls Audiocodec to decode the data sent by Mediacodec and obtain the screen casting audio 1; (35) Audiocodec then returns the screen casting audio 1 to the screen casting application.

[0210] S115. Buffer the screen-casting audio 1 into audio buffer 2. The audio output device of electronic device 200 outputs the screen-casting audio 1 according to the audio data stored in audio buffer 2.

[0211] In some embodiments, on the electronic device 200 side, (36) the audio recording module records audio based on the audio data of the screen-casting audio 1 sent by the upper layer application (e.g., the screen-casting application), stores the recorded screen-casting audio 1 in the buffer, and notifies Audiooflinger; (37) Audiooflinger synthesizes screen-casting audio 1 based on the audio data in the buffer; (38) Audio HAL obtains the synthesized screen-casting audio 1 and stores it in audio buffer 2.

[0212] For ease of description, the buffer in the memory space of electronic device 200 used to store the projected audio from electronic device 100 can be simply referred to as the audio buffer (e.g., audio buffer 2) of electronic device 200. The audio output device (e.g., speaker) of electronic device 200 outputs projected audio 1 based on the data in audio buffer 2. In some embodiments, audio buffer 2 is used to store the latest synthesized audio output by Audio HAL, that is, the audio to be output by the audio output device of electronic device 200.

[0213] In some embodiments, (39) the audio driver of the electronic device 200 sends the audio data in the audio buffer 2 to the audio output device; (40) the audio output device outputs the projected audio 1.

[0214] In some application scenarios, even if the screen projection sent by electronic device 100 to electronic device 200 is playing audio or video, electronic device 100 does not need to send the corresponding audio to electronic device 200 during the projection process. For example, in one application scenario, electronic device 100 and electronic device 200 are very close, and the user can hear the audio corresponding to the projection screen through electronic device 100 without electronic device 200 playing the audio. For instance, electronic device 100 and electronic device 200 are in-vehicle devices in the same vehicle.

[0215] Combination Figure 6A and Figure 6B It is understood that during the screen sharing phase of this solution, the electronic device 100 needs to encode the projected content (i.e., each frame of projected video and / or each frame of projected audio). After the encoded projected data is sent to the target device via the network, it also needs to be decoded. The overall process of the solution involves multiple user-mode and kernel-mode switches (e.g., ...). Figure 6B The steps (9), (11), (13), (17), (21), (22), and (26) shown represent the data copying between the encoder / decoder and the CPU (e.g., Figure 6B The steps (9) and (21) shown in the figure, and the encoding / decoding process (e.g.) Figure 6B The steps (9)-(11) and (19)-(22) shown, the message encapsulation / decapsulation process (e.g.) Figure 6B The steps (12), (13), (17), and (18) shown lead to high CPU / GPU load, high power consumption during screen mirroring, and large screen mirroring latency. This can result in the screen mirroring images displayed on the source and target devices being out of sync, affecting the user's screen mirroring experience. In addition, the encoding / decoding process of the screen mirroring image involves data compression, and lossy compression can lead to a decrease in image quality.

[0216] For example, Figure 7AThis paper presents a method and process for screen mirroring and reverse control based on this solution. The following section combines... Figure 7B The implementation process of screen mirroring reverse control is shown below. The method flow for screen mirroring reverse control is described, and the screen mirroring reverse control operation is applied to the user interface of APP1 in the screen mirroring screen. Specifically:

[0217] S201. After receiving the user's screen mirroring and control operation, the electronic device 200 stores the screen mirroring and control operation corresponding to the screen mirroring and control operation in the event buffer2.

[0218] In some embodiments, the screen mirroring reverse control operation includes a user's touch operation on the screen mirroring screen displayed on the electronic device 200. Step S201 includes: (1) after the touch sensor in the physical display screen detects the screen mirroring reverse control operation, it sends the hardware perception information corresponding to the screen mirroring reverse control operation to the touch sensor driver; (2) the touch sensor driver processes the hardware perception information into a raw input event (e.g., the raw input event includes touch coordinates, touch timestamps, etc.) and sends it to the IO device node corresponding to the display screen (e.g., / dev / input / eventX), and the IO device node stores the event in event buffer2.

[0219] For ease of description, the above-mentioned original input events can be referred to as screen mirroring and control events; the buffer in the memory space of electronic device 200 used to store screen mirroring and control events of electronic device 200 can be simply referred to as the event buffer of electronic device 200 (e.g., event buffer2).

[0220] For example, see Figure 2F As described in the example, screen mirroring control operations may include click operations performed on the video playback area within the mirrored screen. For example, see [link to example]. Figure 3B and Figure 3C As described in the example, the screen mirroring control operation may include a click operation on the icon 203 of the gallery application in the screen mirroring screen. It is not limited to touch operations on the display screen; the screen mirroring control operation involved in the embodiments of this application may also be an input operation performed through other input devices (such as a mouse or remote control), or it may be other forms of input operation such as voice operation or gesture operation, which are not specifically limited here.

[0221] S202, The aforementioned screen mirroring control event in the IO device node of electronic device 200 is distributed to the screen mirroring application.

[0222] In some embodiments, step S202 includes: (3) The input subsystem reads the original input event (i.e., the screen mirroring control event) from the event buffer2 stored in the IO device node. (4) After the IMS detects the original input event output by the input subsystem, it calls the WMS to determine the target window corresponding to the touch event according to the layout of each window in the main screen. The target application corresponding to the original input event is the application corresponding to the target window (i.e., the screen mirroring application). (5) The WMS distributes the original input event to the screen mirroring application.

[0223] S203, After the screen projection application of electronic device 200 encapsulates the screen projection reverse control event using the communication protocol, it sends the encapsulated screen projection message 3 to electronic device 100 through communication module 1.

[0224] In some embodiments, taking the WiFi communication module 1 as an example, step S105 includes: (6) the screen-casting application sends an encrypted screen-casting reverse control event to the TCP / IP protocol stack; (7) the TCP / IP protocol stack encapsulates the screen-casting reverse control event using the TCP / IP protocol and sends the screen-casting message 2 corresponding to the screen-casting reverse control event to the WiFi driver; (8) the WiFi driver triggers the WiFi communication module to send the screen-casting message 1 to the electronic device 200; (9) the WiFi communication module sends the screen-casting message 1 to the electronic device 200. It can be understood that the screen-casting message 2 includes the above-mentioned screen-casting reverse control event.

[0225] S204. After the electronic device 100 decapsulates the screen projection message 3, the screen projection application obtains the screen projection reverse control event.

[0226] In some embodiments, taking the WiFi communication module as an example, step S106 includes: (10) the WiFi driver obtains the screen projection message 2 from the WiFi communication module; (11) the WiFi driver sends the screen projection message 2 to the TCP / IP protocol stack; (12) the TCP / IP protocol stack decapsulates the screen projection message 2 using the TCP / IP protocol and sends the decapsulated screen projection reverse control event to the screen projection application.

[0227] S205, the screen mirroring application of electronic device 100 distributes screen mirroring control events to APP1 through the input subsystem.

[0228] In some embodiments, step S205 includes: (13) the screen mirroring application decrypts the screen mirroring reverse control event and sends the decrypted screen mirroring reverse control event to the input subsystem; (14) the user's screen mirroring reverse control operation is applied to APP1, and the input subsystem distributes the screen mirroring reverse control event to APP1 (e.g., a video application) running in the foreground of the virtual display screen 1 corresponding to the electronic device 200. In one implementation, after the IMS detects the screen mirroring reverse control event (i.e., touch event) output by the above-mentioned input subsystem, it calls the WMS to determine the target window corresponding to the touch event in the virtual display screen according to the layout of each window in the virtual display screen, and the target application corresponding to the above-mentioned screen mirroring reverse control event is APP1 corresponding to the target window.

[0229] S206, APP1 of electronic device 100 determines the response event corresponding to the screen projection reverse control event. The response event indicates the display content of the screen projection screen 2 after the response. Based on the response event, the screen projection screen 2 after the response is drawn and synthesized. The synthesized screen projection screen 2 is cached in display buffer 3.

[0230] In some embodiments, step S206 includes: (15) APP1 determines a response event based on the screen mirroring control event, the response event indicating the screen mirroring screen 2 after the response; APP1 sends a drawing instruction to WMS, the drawing instruction being used to instruct the drawing of the user interface 2 after the response of APP1; (16) WMS instructs surfaceflinger to composite each layer; (17) surfaceflinger calls GPU compositing and HWC compositing to obtain the composited screen mirroring screen 2, the screen mirroring screen 2 including the user interface 2. HWC stores the composited screen mirroring screen 2 into the display buffer 3.

[0231] In some embodiments, display buffer 3 and display buffer 1 of electronic device 100 are the same buffer. In some embodiments, the display buffer of electronic device 100 has multiple buffers (e.g., display buffer 1 and display buffer 3), and electronic device 100 stores the latest synthesized screen projection image in a polling manner into one of the above multiple buffers.

[0232] In some embodiments, the main screen of the physical display of the electronic device 100 includes the projection screen 2; the above method further includes: sending the projection screen 2 to the physical display of the electronic device 100. In some implementations, (18) HWC sends the composite projection screen 2 to the display driver, that is, the display driver displays the image data of the projection screen 2 in the buffer 3; (19) the display driver sends the projection screen 2 to the physical display of the electronic device 100.

[0233] For example, see Figure 2FThe example described, screen mirroring screen 1 includes Figure 2F The user interface 11 of the video application shown includes a video playback area. The screen mirroring control operation can include a click operation (i.e., a pause operation) on the video playback area of ​​the screen mirroring screen 1. The screen mirroring screen 2 includes a pause interface 13 in response to the aforementioned pause operation. For example, see [link to example]. Figure 3B and Figure 3C The example described, screen mirroring screen 1 includes Figure 3B The main interface 21 of the system application shown includes an icon 203 for the gallery application. The screen mirroring control operation can include a click operation on the icon 203 in the screen mirroring screen. The screen mirroring screen 2 includes a user interface 23 of the gallery application that is determined in response to the click operation.

[0234] S207, Electronic device 100 encodes the screen projection image 2 and obtains the screen projection data 3.

[0235] In some embodiments, step S207 includes Figure 7A Steps (20) to (23) in the software flow shown are described in detail below. For specific details, please refer to the description of step S105, which will not be repeated here.

[0236] S208, after the electronic device 100 encapsulates the projection data 3 using the communication protocol, it sends the encapsulated projection message 4 through the communication module 1.

[0237] In some embodiments, step S208 includes Figure 7A Steps (24) to (27) in the software flow shown are described in detail below. For specific details, please refer to the description of step S106, which will not be repeated here.

[0238] S209, after the electronic device 200 decapsulates the screen projection message 4, the screen projection application obtains the screen projection data 3.

[0239] In some embodiments, step S209 includes Figure 7A Steps (28) to (30) in the software flow shown are described in detail below. For specific details, please refer to the description of step S107, which will not be repeated here.

[0240] S210 and electronic device 200 decode the projection data 3 and obtain the decoded projection screen 2.

[0241] In some embodiments, step S210 includes Figure 7A Steps (31) to (34) in the software flow shown are described in detail below. For specific details, please refer to the description of step S108, which will not be repeated here.

[0242] S211, the electronic device 200 stores the projected image 3 into the display buffer 4, and the physical display screen displays the projected image 2 based on the image of the projected image 2 cached in the display buffer 4.

[0243] In some embodiments, in the electronic device 200, the display buffer 4 storing the projection screen 2 and the display buffer 2 storing the projection screen 1 are the same buffer.

[0244] In some embodiments, the display buffer of the electronic device 100 has multiple buffers (e.g., display buffer2 and display buffer4), and the electronic device 100 stores the latest synthesized projection screen in one of the multiple buffers in a polling manner. For example, the display buffer of the electronic device 200 includes buffer a, buffer b, and buffer c; according to the address order of the buffers, the HWC polls and caches the latest synthesized projection screen in one of the multiple buffers in a polling manner; when the Vsync signal of the next refresh cycle arrives, the display driver sends the latest cached projection screen to the physical display screen. For example, the HWC synthesizes projection screens a, b, and c sequentially, and projection screens a, b, and c are cached in buffer a, buffer b, and buffer c sequentially; when the HWC synthesizes projection screen d, projection screen d is polled and cached in buffer a, overwriting the original projection screen a.

[0245] In some embodiments, step S211 includes Figure 7A Steps (35) to (39) in the software flow shown are described in detail below. For specific details, please refer to the description of step S109, which will not be repeated here.

[0246] In some embodiments, the screen projection 2 sent by electronic device 100 to electronic device 200 is used for playing audio and video, and electronic device 100 also sends the audio corresponding to the screen projection 2 to electronic device 200. For details, please refer to... Figure 6C The relevant descriptions will not be repeated here.

[0247] Combination Figure 7A and Figure 7B It can be seen that during the screen mirroring and control phase of this scheme, when the user's screen mirroring and control event is sent to the target application (e.g., APP1) on the source device, both devices involved in the screen mirroring process are involved in the kernel's handling of the screen mirroring and control event (e.g., ...). Figure 7B The steps (3) to (7) on the target device side and (11) to (13) on the source device side are shown, as well as multiple user mode and kernel mode switches (e.g. Figure 7BThe steps (2), (7), and (11) shown lead to a significant response delay for the source device in responding to the screen mirroring control event. Combined with the processing delay for screen sharing of screen mirroring screen 2 after responding to the screen mirroring control event, the overall screen mirroring control delay is even greater.

[0248] Option 2: RDMA-based screen mirroring solution

[0249] In Option 2, electronic devices 100 and 200 support RDMA technology. RDMA technology allows user-mode applications to directly access (e.g., write to or read from) remote memory, bypassing the operating system and CPU, and directly reading and writing data (e.g., data required for screen mirroring) from the memory of one host to the memory of another host.

[0250] For example, Figure 8 An RDMA software stack is illustrated, which includes a user-mode software stack and a kernel-mode software stack. For example... Figure 8 As shown, the user-space software stack includes the application layer and the user-space lib library for RDMA. The user-space lib library includes Libibumad, Libirdmacm, Libibcm, and Libibverbs. The kernel-space software stack includes the kernel-space lib library, the RDMA subsystem core module (Ib-core), and the RDMA network card driver. The kernel-space lib library includes ib_umad, ib_ucm, rdma_ucm, and ib_uverbs.

[0251] The main functions of each lib library are to register and provide character / miscellaneous devices, perform subnet management, connection management, and provide interfaces. The kernel-mode lib libraries interface with the user-mode lib libraries through the device's file operation interface (file_operations), converting the RDMA operations of the user-mode lib libraries into RDMA operations of the kernel device (such as the ib_device device) (such as read, write, release, etc.).

[0252] Specifically, by calling the "ibv_post_send()" interface of the libibverbs library to issue I / O commands, the transmit and receive queues of the RDMA network card can be directly manipulated in user space. This implementation avoids system calls (i.e., switching between user space and kernel space, and copying memory data), thus saving CPU resources to the greatest extent and improving I / O efficiency.

[0253] There are three hardware implementations of RDMA technology using RDMA network protocols: Infiniband (IB), RDMA over converged Ethernet (RoCE) (including RoCEv1 and RoCEv2), and Internet Wide Area RDMA protocol (iWARP). Infiniband is a network specifically designed for RDMA, guaranteeing reliable transmission at the hardware level. RoCE and iWARP are both Ethernet-based RDMA technologies, supporting corresponding verb interfaces; RoCE also guarantees reliable transmission at the hardware level.

[0254] An RDMA network interface card (NIC) is a network interface card that supports the RDMA network protocol. In the IB / RoCE protocol specification, an RDMA NIC can be referred to as a Host Channel Adapter (HCA). In the iWARP protocol specification, an RDMA NIC can be referred to as an RNIC, which stands for Network Interface Card (NIC) with an RDMA engine.

[0255] RDMA network cards embed the RDMA network protocol onto the network card itself. The network layer and transport layer are implemented within RDMA. RDMA network cards directly encapsulate application data (e.g., TCP / IP protocol encapsulation) and send it, as well as parse received packets and send the parsed data directly to the application layer. In some embodiments, the bus interface of the RDMA network card adopts an interface with the high-speed serial computer extended bus standard (PCIe).

[0256] In some embodiments, during the screen sharing phase of Scheme 2, the electronic device 200 can directly write the image data of the projected screen in the display buffer 1 of the electronic device 100 into the display buffer 2 of the electronic device 200 using the RDMA network card 2, and the physical display screen of the electronic device 200 displays the projected screen based on the image data in the display buffer 2.

[0257] For example, Figure 9A This paper presents a screen sharing method and process for this solution. The following section combines... Figure 9B The software structure shown describes the method flow for screen sharing, which includes steps S301 to S310.

[0258] like Figure 9B This illustrates the software architecture of another electronic device, such as... Figure 9BAs shown, electronic devices 100 and 200 are equipped with RDMA network cards; the kernel layer includes the RDMA network card driver corresponding to the RDMA network card.

[0259] S301, Electronic device 100 displays the user interface 1 of APP1.

[0260] S302. After receiving the user's screen projection operation, the electronic device 100 stores the screen projection trigger event corresponding to the screen projection operation in its IO device node (i.e., / dev / input / eventX).

[0261] S303, the screen mirroring trigger event in the IO device node of electronic device 100 is distributed to the screen mirroring application.

[0262] For the specific implementation of steps S301 to S303, please refer to the relevant descriptions of steps S101 to S103 above, which will not be repeated here.

[0263] S304. An RDMA connection is established between the RDMA network card of electronic device 100 and the RDMA network card of electronic device 200.

[0264] In some embodiments, after determining that screen casting is allowed based on the aforementioned screen casting trigger event, the screen casting application triggers the RDMA network card of electronic device 100 to establish an RDMA connection (i.e., a channel connection) with electronic device 200. In some embodiments, based on the aforementioned screen casting trigger event, the screen casting application sends a screen casting service connection request to electronic device 200 through communication module 1 (e.g., a WiFi communication module); the screen casting application of electronic device 200 sends a screen casting service connection response to electronic device 100; after determining that electronic device 200 agrees to screen casting based on the aforementioned screen casting service connection response, the screen casting application of electronic device 200 triggers the RDMA network card of electronic device 100 to establish an RDMA connection with electronic device 200.

[0265] Channel connections are used for RDMA message transmission. (See reference) Figure 9C Each Channel connection maintains a pair of queues (QPs) at both ends of the network interface card (NIC). Each QP consists of a send queue (SQ) and a receive queue (RQ). The queues manage work queue elements (WQEs) corresponding to various types of RDMA work requests (WRs). The WQE (i.e., the task description of the WR) waits for the RDMA NIC to schedule and parse it. The RDMA NIC retrieves data from the buffer pointed to by the WQE and sends it to the other end of the Channel connection.

[0266] S305, Electronic device 200 sends RDMA message 1 to electronic device 100, RDMA message 1 carries the virtual address of display buffer 2.

[0267] In some embodiments, after establishing an RDMA connection, RDMA network card 2 sends the virtual address of display buffer 2 to RDMA network card 1. This means that RDMA network card 1 is informed in advance of the target storage address of the projected screen in electronic device 200, so that RDMA network card 1 can directly write the image data of the projected screen to that target storage address.

[0268] In some embodiments, RDMA message 1 further includes key 1. After establishing an RDMA connection, electronic device 200 registers display buffer 2 with RDMA network card 2 and obtains key 1 (Remote key) corresponding to display buffer 2. RDMA network card 2 records the correspondence between the address of display buffer 2 and key 1. After registration, RDMA network card 2 sends RDMA message 1 to RDMA network card 1. RDMA message 1 includes the virtual address of display buffer 2 and key 1. Sending key 1 to RDMA network card 1 of electronic device 100 grants the network card permission to remotely operate display buffer 2. In some embodiments, RDMA message 1 may also include the memory length of display buffer 2, which indicates the maximum amount of data that can be written to display buffer 2.

[0269] In some embodiments, reference Figure 9B The electronic device 200 registers the display buffer2 to the RDMA network card 2, including: (1) the upper-layer application of the electronic device 200 (e.g., a screen mirroring application) instructs the HWC to send an instruction to the RDMA network card driver; (2) based on the instruction, the RDMA network card driver instructs the RDMA network card 2 to register the address of the display buffer2. In one implementation, the screen mirroring application of the electronic device 200 instructs the HWC to send an instruction to the RDMA network card driver, including: the screen mirroring application of the electronic device 200 sends an instruction to the surfaceflinger through the WMS, and based on the instruction, the surfaceflinger instructs the HWC to register the address of the display buffer2 to the RDMA network card 2; the HWC sends an instruction to the RDMA network card driver. Step S305 includes: (3) the RDMA network card 2 sends an RDMA message 1 to the RDMA network card 1.

[0270] In some embodiments, the display buffer of the electronic device 200 includes multiple buffers (e.g., buffer a, buffer b, and buffer c), and display buffer 2 is one of these multiple buffers. The RDMA message 1 includes the virtual address of each of the multiple buffers. In subsequent embodiments, the RDMA network card 1 of the electronic device 100 may also, according to the address order of the buffers, poll and write the latest synthesized projection screen to one of the multiple buffers of the electronic device 200 via an RDMA write request. In some embodiments, the RDMA message 1 may also include the memory size of each buffer.

[0271] S306, The screen projection application instruction of electronic device 100 is to draw and synthesize the screen projection screen 1 to electronic device 200.

[0272] For the specific implementation of step S305, please refer to the relevant description of step S104 above, which will not be repeated here.

[0273] S307. After the electronic device 100 determines that the projection screen 1 has been synthesized, it submits a work request WR1 to the RDMA network card 1. The work request WR1 is used to instruct the image data of the projection screen 1 to be written into the display buffer 2 of the electronic device 200.

[0274] In some embodiments, after HWC determines that the projection screen 1 has been successfully composited (e.g., it determines that the image data of the projection screen 1 has been written to display buffer 1), it submits a work request WR1 to RDMA network card 1. For example, see... Figure 9B Step S307 includes: (4) After HWC determines that the projection screen 1 is completed, it submits a work request WR1 to the RDMA network card driver. The work request WR1 includes the virtual address of the display buffer 1 that stores the projection screen 1; (5) The RDMA network card driver sends the work request WR1 to the RDMA network card 1.

[0275] In some embodiments, after an RDMA connection is established, the upper-layer application of the electronic device 200 (e.g., a screen mirroring application) instructs the HWC to enable switch 1. After enabling switch 1 is enabled, when it is determined that a new screen mirroring image has been successfully composited, the HWC submits a work request WR1 to the RDMA network card 1. In some embodiments, the upper-layer application of the electronic device 200 instructing the HWC to enable switch 1 includes: the screen mirroring application of the electronic device 200 sending a command to surfaceflinger via WMS, and based on this command, surfaceflinger instructing the HWC to enable switch 1.

[0276] In some embodiments, see Figure 9CThe RDMA implementation process shown includes step S306, which includes: (1) When the HWC of the electronic device 100 determines that the projection screen 1 is to be composited, it submits WR1 to the RDMA network card 1. The RDMA network card 1 generates the task WQE corresponding to WR (e.g., based on the virtual address of the display buffer 2 carried by the above WR1 and RDMA message 1) Figure 9C The WQE1 shown is added to the SQ queue. WQE1 may include the RDMA operation type (i.e., write operation), the local data address (i.e., the virtual address of the display buffer1 storing the projection screen 1), the target storage address (i.e., the virtual address of the display buffer2 of the electronic device 200), and the key 1 corresponding to the display buffer2.

[0277] S308. Based on WR1, electronic device 100 sends RDMA write request 1 to electronic device 200. RDMA write request 1 is used to instruct the data of the projection screen 1 to be written to the display buffer 2 of electronic device 200. RDMA write request 1 carries the virtual address of display buffer 2 and the image data of projection screen 1 in display buffer 1.

[0278] In some embodiments, the RDMA write request 1 also carries a key 1 corresponding to the display buffer 2, which is used to verify that the electronic device 100 has the authority to operate the display buffer 2.

[0279] In some embodiments, see Figure 9B Step S308 includes: (6) Based on the address of display buffer1 indicated by WR1, RDMA network card 1 instructs RDMA network card driver to read the data of display buffer1; (7) RDMA network card 1 obtains the data of display buffer1; (8) Based on the data of display buffer1, RDMA network card 1 assembles RDMA write request 1 and sends RDMA write request 1 to RDMA network card 2.

[0280] In this embodiment of the application, the electronic device 100 can write the screen projection 1 in the display buffer 1 of the electronic device 100 to the display buffer 2 of the electronic device 200 through RDMA one-sided operation (write operation) or RDMA two-sided operation.

[0281] In some embodiments, see Figure 9CThe RDMA implementation process shown includes step S308 as follows: (2) RDMA network card 1 retrieves WQE1 (i.e., the task corresponding to WR1) from SQ and parses the information WQE1; according to the virtual address of display buffer1 in WQE1, it converts to obtain the physical address; (3) according to the physical address, it reads the image data of the projection screen 1 from the memory of electronic device 100 and assembles RDMA write request 1; (4) RDMA network card 1 sends RDMA write request 1 to RDMA network card 2 through the physical link (e.g., IB link / Ethernet link) connected to RDMA.

[0282] S309. Based on RDMA write request 1, the RDMA network card 2 of electronic device 200 writes the data of display buffer 1 to display buffer 2.

[0283] In some embodiments, see Figure 9B Step S308 includes: (9) After parsing RDMA write request 1, RDMA network card 2 sends an instruction to RDMA network card driver; (10) Based on the instruction, RDMA network card driver writes the data of display buffer 1 into display buffer 2 in local memory.

[0284] In some embodiments, see Figure 9C The RDMA implementation process shown includes step S309: (5) RDMA network card 2 parses RDMA write request 1 and converts the target storage address (i.e., the virtual address of display buffer 2) in RDMA write request 1 into the physical address of local memory; (6) writes the target data (i.e., the data of display buffer 1) in RDMA write request 1 into the physical address.

[0285] In some embodiments, see Figure 9C The RDMA implementation process shown includes the following steps after step S308: (9) RDMA network card 2 replies with an acknowledgment message (ACK) to RDMA network card 1. (7) RDMA network card 1 generates a complete queue element (CQE) (e.g., CQE) based on the acknowledgment message and places it in the CQ. (8) The upper layer application determines that the task is completed based on CQE1.

[0286] S310 and electronic device 200 display the projected screen 1 based on the image data in display buffer 2.

[0287] The specific implementation of step S310 can be found in the description of step S109 above, and will not be repeated here.

[0288] Option 2 utilizes RDMA for screen mirroring data transmission, providing microsecond-level access latency and approximately 100Gbps throughput per connection. For an example, see... Figure 9D In the refresh cycle following vsync1, electronic device 100 renders and buffers each layer in the buffer queue. In the refresh cycle following the next vsync (vsync2), electronic device 100 synthesizes the projected image (e.g., projected image 1) based on the rendered layers. After storing the image data of the projected image in display buffer1, the data in display buffer1 can be written to display buffer2 of electronic device 200 in a very short time. Thus, with synchronized clocks on both devices, electronic device 100 and electronic device 200 can display the same projected image in the refresh cycle following another vsync (vsync3). This scheme supports higher refresh rates (e.g., 60fps, 90fps, or 120fps).

[0289] In some embodiments, the screen projection sent by electronic device 100 to electronic device 200 is playing audio and video. During the projection process, electronic device 100 also sends the audio corresponding to the screen projection to electronic device 200. For example... Figure 9E As shown, the screen mirroring solution also includes audio sharing, and the audio sharing method includes steps S311 to S316. The following section will combine... Figure 9F The software structure shown provides a detailed explanation of steps S311 to S316.

[0290] S311, Electronic device 200 sends RDMA message 2 to electronic device 100, RDMA message 2 carrying the virtual address of audio buffer 2.

[0291] In some embodiments, after establishing an RDMA connection, RDMA network card 2 sends the virtual address of the display audio buffer 2 to RDMA network card 1. This means that RDMA network card 1 is informed in advance of the target storage address of the projected audio in electronic device 200, so that RDMA network card 1 can directly write the audio data of the projected audio to that target storage address.

[0292] In some embodiments, RDMA message 2 further includes key 2. After establishing an RDMA connection, electronic device 200 registers audio buffer 2 with RDMA network card 2 and obtains key 2 (Remote key) corresponding to audio buffer 2. RDMA network card 2 records the correspondence between the address of audio buffer 2 and key 2. After registration, RDMA network card 2 sends RDMA message 2 to RDMA network card 1. RDMA message 2 includes the virtual address of audio buffer 2 and key 2. Sending key 2 to RDMA network card 1 of electronic device 100 grants the network card permission to remotely operate audio buffer 2. In some embodiments, RDMA message 2 may also include the memory length of audio buffer 2, which indicates the maximum amount of data that can be written to audio buffer 2.

[0293] In some embodiments, reference Figure 9F The electronic device 200 registers audio buffer 2 to RDMA network card 2, including: (1) the upper-layer application of the electronic device 200 (e.g., a screen mirroring application) instructs the Audio HAL to send a command to the RDMA network card driver; (2) based on the command, the RDMA network card driver instructs the RDMA network card 2 to register the address of audio buffer 2. In one implementation, the screen mirroring application of the electronic device 200 instructs the Audio HAL to register audio buffer 2 to RDMA network card 2 via Audiooflinger. Step S311 includes: (3) RDMA network card 2 sends RDMA message 2 to RDMA network card 1.

[0294] In some embodiments, similar to the display buffer, the audio buffer of electronic device 200 may also include multiple buffers, with audio buffer 2 being one of these multiple buffers. The RDMA message 2 includes the virtual address of each of the multiple buffers. In subsequent embodiments, the RDMA network card 1 of electronic device 100 may also, according to the address order of the buffers, poll and write the latest synthesized projection audio to one of the multiple buffers of electronic device 200 via an RDMA write request. In some embodiments, the RDMA message 2 may also include the memory size of each buffer.

[0295] In some embodiments, the RDMA message also includes a buffer type (e.g., audio buffer or display buffer). This allows the RDMA network card 1 of the electronic device 100 to recognize that the virtual address in RDMA message 2 is the address of the audio buffer, and the virtual address in RDMA message 1 is the address of the display buffer. For example, the buffer type included in RDMA message 2 indicates an audio buffer.

[0296] S312, the screen mirroring application of electronic device 100 is instructed to synthesize screen mirroring audio; the audio data of the synthesized screen mirroring audio 1 is stored in audio buffer 1.

[0297] For a detailed implementation of step S312, please refer to [link / reference]. Figure 6C The relevant description of step S110 will not be repeated here.

[0298] S313. After determining that the projection audio 1 has been synthesized, the electronic device 100 submits a work request WR2 to the RDMA network card 1. The work request WR2 is used to instruct the audio data of the projection audio 1 to be written into the audio buffer 2 of the electronic device 200.

[0299] In some embodiments, after the Audio HAL determines that the projected audio 1 has been synthesized (e.g., it determines that the audio data of the projected audio 1 has been written to the display buffer 1), it submits a work request WR2 to the RDMA network card 1. See, for an example... Figure 9F Step S313 includes: (4) After the Audio HAL determines that the screen-casting audio 1 has been synthesized, it submits a work request WR2 to the RDMA network card driver. The work request WR2 includes the virtual address of the audio buffer 1 that stores the screen-casting audio 1; (5) The RDMA network card driver sends the work request WR2 to the RDMA network card 1.

[0300] In some embodiments, after an RDMA connection is established, the upper-layer application of the electronic device 200 (e.g., a screen mirroring application) instructs the Audio HAL to enable switch 2. After enabling switch 2 is enabled, when screen mirroring audio synthesis is detected to be complete, the Audio HAL submits a work request to the RDMA network card 1. In some embodiments, the upper-layer application of the electronic device 200 instructing the Audio HAL to enable switch 2 includes: the screen mirroring application of the electronic device 200 instructing the Audio HAL to enable switch 2 via Audiooflinger.

[0301] In some embodiments, RDMA network card 1 generates WQE2 based on WR2 and adds WQE2 to the SQ queue. WQE2 may include the RDMA operation type (i.e., write operation), local data address (i.e., the virtual address of audio buffer 1 storing screen-projected audio 1), target storage address (i.e., the virtual address of audio buffer 2 of electronic device 200), and key 2 corresponding to audio buffer 2.

[0302] S314. Based on WR2, electronic device 100 sends RDMA write request 2 to electronic device 200. RDMA write request 2 is used to instruct the data of screen-casting audio 1 to be written to audio buffer 2 of electronic device 200. RDMA write request 1 carries the virtual address of audio buffer 2 and the audio data of screen-casting audio 1 in audio buffer 1.

[0303] In some embodiments, the RDMA write request 1 also carries a key 2 corresponding to the audio buffer 2, which is used to verify that the electronic device 100 has the authority to operate the audio buffer 2; the key 2 corresponding to the audio buffer 2.

[0304] In some embodiments, see Figure 9F Step S314 includes: RDMA network card 1 retrieves WQE2 (i.e., the task corresponding to WR2) from SQ and parses information WQE2; according to the virtual address of audio buffer1 in WQE, it converts to obtain the physical address; (6) RDMA network card 1 instructs RDMA network card driver to read the data of audio buffer1 at the physical address; (7) RDMA network card 1 obtains the data of audio buffer1; (8) based on the data of audio buffer1, RDMA network card 1 assembles RDMA write request 2 and sends RDMA write request 1 to RDMA network card 2.

[0305] In some embodiments, the work request WR also includes a data type (e.g., audio or image). This allows the RDMA network card 2 to distinguish between a work request WR2 for projected audio and a work request WR1 for projected video, thereby generating an RDMA write request 2 for projected audio or an RDMA write request 1 for projected video. For example, WR2 indicates that the data type is audio. Based on this data type, the RDMA network card 2 obtains the virtual address of the audio buffer 1 of the electronic device 100; then, it assembles the RDMA write request 2 based on this address. WR1 indicates that the data type is image. Based on this data type, the RDMA network card 2 obtains the virtual address of the display buffer 1 of the electronic device 100; then, it assembles the RDMA write request 1 based on this address.

[0306] S315, Based on RDMA write request 2, the RDMA network card 2 of electronic device 200 writes the data of audio buffer 1 to audio buffer 2.

[0307] In some embodiments, see Figure 9FStep S308 includes: (9) parsing RDMA write request 2, RDMA network card 2 sending an instruction to RDMA network card driver; (10) based on the instruction, RDMA network card driver writing the data of audio buffer1 in RDMA write request 2 into audio buffer2 in local memory.

[0308] In some embodiments, step S309 includes: (5) RDMA network card 2 parses RDMA write request 2 and converts the target storage address (i.e., the virtual address of audio buffer 2) in RDMA write request 2 into the physical address of local memory; (6) writes the target data (i.e., the data of audio buffer 1) in RDMA write request 2 into the physical address.

[0309] S316, the audio output device of electronic device 200 outputs screen-projected audio 1 based on the audio data buffered in audio buffer 2.

[0310] For a detailed implementation of step S312, please refer to [link / reference]. Figure 6C The relevant description of step S115 will not be repeated here.

[0311] In some embodiments, reference Figure 9B and Figure 9F The description states that RDMA network card 2 directly writes the screen projection from electronic device 100 to local display buffer 2, and directly writes the screen projection audio from electronic device 100 to local audio buffer 2.

[0312] In some embodiments, if the electronic device 100 projects its screen onto the electronic device 200, and the audio is also sent to the electronic device 200 in a synchronized manner, then it is necessary to ensure that the video and audio of the projected screen are synchronized (for example, projected screen 1 corresponds to projected audio 1, and projected screen 2 corresponds to projected audio 2).

[0313] In some embodiments, see Figure 9F The kernel layer of the electronic device 200 includes a synchronization module, which is used to control the synchronization of the screen projection screen and the screen projection audio. After the RDMA network card obtains the screen projection screen (e.g., screen projection screen 1 in step S308) through RDMA write request 1 and obtains the screen projection audio (e.g., screen projection audio in step S314) through RDMA write request 2, (11) the RDMA network card sends the above-mentioned screen projection screen 1 and screen projection audio 1 to the synchronization module through the RDMA network card driver; screen projection screen 1 and screen projection audio 1 can be sent at the same time or separately, and no specific limitation is made here. After the synchronization module performs image-audio synchronization on the received screen projection screen and screen projection audio, in the same refresh cycle, (12) screen projection screen 1 is written to display buffer 2, and (13) screen projection audio 1 is written to audio buffer 2.

[0314] For how the synchronization module synchronizes the projected screen and audio, existing technologies can be referenced. In one implementation, RDMA write request 1 also carries the timestamp of projected screen 1, and RDMA write request 2 also carries the timestamp of projected audio 1. The RDMA network card also sends the timestamps of the projected screen and audio to the synchronization module. When the synchronization module determines that the received timestamps of projected screen 1 and projected audio 1 are consistent, it determines that projected screen 1 and projected audio 1 are synchronized; otherwise, they are not synchronized. In case of asynchrony, an error can be reported to RDMA network card 1 through RDMA network card 2.

[0315] In some embodiments, during the screen projection and control phase of Scheme 2, the RDMA network card 3 of electronic device 100 directly writes the screen projection and control event in event buffer 2 of electronic device 200 into event buffer 1 of electronic device 100, and the upper-layer application of electronic device 100 responds to the event based on the input event in event buffer 1.

[0316] For example, Figure 10A This paper presents a method and process for screen mirroring and reverse control based on this solution. The following section combines... Figure 10B The implementation process of screen mirroring reverse control is shown below, and the method flow of this screen mirroring reverse control is introduced. Among them:

[0317] S401, Electronic device 100 sends RDMA message 3 to electronic device 200. RDMA message 3 carries the virtual address of event buffer 1.

[0318] For ease of description, the buffer in the memory space of electronic device 100 used to store raw input events can be simply referred to as the event buffer of electronic device 100 (e.g., event buffer). In some embodiments, raw input events of I / O device nodes corresponding to different input devices (e.g., screen, mouse) are stored in different event buffers (e.g., event buffer 2 corresponding to the screen of electronic device 100). RDMA message 3 carries the virtual address of each event buffer of electronic device 100, as well as the identifier of the corresponding input device.

[0319] In some embodiments, after establishing an RDMA connection, RDMA network card 1 sends the virtual address of event buffer 1 to RDMA network card 2. That is, it informs RDMA network card 2 in advance of the target storage address of the original input event in electronic device 100, so that RDMA network card 2 can directly write the screen projection and control event to the target storage address.

[0320] In some embodiments, RDMA message 3 further includes key 3. After establishing an RDMA connection, electronic device 100 registers event buffer1 with RDMA network card 1 and obtains key 3 (Remote key) corresponding to event buffer1. RDMA network card 1 records the correspondence between the address of event buffer1 and key 3. After registration, RDMA network card 1 sends RDMA message 3 to RDMA network card 2. RDMA message 3 includes the virtual address of event buffer1 and key 3. Sending key 3 to RDMA network card 2 of electronic device 200 grants the network card permission to remotely operate event buffer1. In some embodiments, RDMA message 2 may also include the memory length of event buffer1, which indicates the maximum amount of data that can be written to event buffer1.

[0321] In some embodiments, reference Figure 10B The electronic device 100 registers event buffer1 with RDMA network card 1, including: (1) the screen projection application of the electronic device 100 instructs the IO device node (e.g., / dev / input / eventX) to send an instruction to the RDMA network card driver; (2) based on the instruction, the RDMA network card driver instructs RDMA network card 1 to register the address of event buffer1. Step S401 includes: (3) RDMA network card 1 sends RDMA message 3 to RDMA network card 2.

[0322] S402 After receiving the user's screen mirroring and control operation, the IO device node ( / dev / input / eventX) stores the screen mirroring and control operation corresponding to the screen mirroring and control operation in the event buffer2.

[0323] The specific implementation of step S402 can be found in the description of step S201, and will not be repeated here.

[0324] After storing the screen projection and control event 1 in event buffer2, the IO device node of electronic device 200 submits a work request WR3 to RDMA network card 2. The work request WR3 is used to instruct the screen projection and control event 1 to be written into the event buffer1 of electronic device 100.

[0325] In some embodiments, see Figure 10B Step S403 includes: (4) After the IO device node ( / dev / input / eventX) stores the screen projection reverse control event 1 in event buffer2, it submits a work request WR3 to the RDMA network card driver. The work request WR3 packet stores the virtual address of the event buffer2 of the screen projection reverse control event 1; (5) The RDMA network card driver sends the work request WR3 to the RDMA network card 2.

[0326] In some embodiments, after an RDMA connection is established, the upper-layer application of the electronic device 200 (e.g., a screen mirroring application) instructs the IO device node to enable switch 3; after enabling switch 3, when a new screen mirroring control event is stored in event buffer 2, the IO device node submits a work request WR3 to the RDMA network card 2.

[0327] In some embodiments, see Figure 10C The RDMA implementation process shown includes step 403 as follows: (1) After the IO device node stores the screen projection control event 1 in event buffer2, it submits WR3 to the RDMA network card 2. The RDMA network card 2 generates the task WQE3 corresponding to WR3 based on the virtual address of event buffer1 carried by the above WR3 and RDMA message 3, and adds it to the SQ queue. WQE3 may include the RDMA operation type (i.e., write operation), the local data address (i.e., the virtual address of event buffer2), the target storage address (i.e., the virtual address of event buffer2), and the key 3 of event buffer2.

[0328] In some embodiments, the RDMA message 3 carries the virtual address of each event buffer of the electronic device 100 and the identifier of the corresponding input device. The electronic device 200 determines that the event buffer 2 of the local storage screen projection control event 1 corresponds to the input device 1 (e.g., the screen); the electronic device 200 selects the virtual address of the event buffer corresponding to the input device 1 (e.g., the event buffer 1 corresponding to the screen) in the RDMA message 3 as the target storage location.

[0329] S404, Electronic device 200 sends RDMA write request 3 to electronic device 100. RDMA write request 3 is used to instruct the screen mirroring and control event 1 to be written to the event buffer 1 of electronic device 100. RDMA write request 3 carries the virtual address of event buffer 1 and the data of screen mirroring and control event 1 in event buffer 2.

[0330] In some embodiments, the RDMA write request 3 also carries a key 3, which is used to verify that the electronic device 200 has permission to operate the event buffer 1.

[0331] In some embodiments, see Figure 10BStep S404 includes: (6) Based on the address of event buffer2 indicated by WR3, RDMA network card 2 instructs RDMA network card driver to read the data of event buffer2; (7) RDMA network card 2 obtains the data of event buffer2; (8) Based on the data of event buffer2, RDMA network card 2 assembles RDMA write request 3 and sends RDMA write request 3 to RDMA network card 1.

[0332] In some embodiments, see Figure 10C The RDMA implementation process shown includes step S404 as follows: (2) RDMA network card 2 retrieves WQE3 (i.e., the task corresponding to WR3) from SQ and parses information WQE3; according to the virtual address of event buffer2 in WQE3, the physical address is converted; (3) according to the physical address 3, the data of screen projection reverse control event 1 is read from the memory of electronic device 200 and RDMA write request 3 is assembled; (4) RDMA network card 2 sends RDMA write request 3 to RDMA network card 1 through RDMA connection.

[0333] S405, Based on RDMA write request 3, the RDMA network card 1 of electronic device 100 writes the data of event buffer 2 into event buffer 1.

[0334] In some embodiments, see Figure 10B Step S405 includes: (9) After parsing RDMA write request 3, RDMA network card 1 sends an instruction to RDMA network card driver; (10) Based on the instruction, RDMA network card driver writes the data of event buffer2 in RDMA write request 3 into event buffer1 in local memory.

[0335] In some embodiments, see Figure 10C The RDMA implementation process shown includes step S405: (5) RDMA network card 1 parses RDMA write request 3 and converts the target storage address (i.e., the virtual address of event buffer 1) in RDMA write request 3 into the physical address of local memory; (6) writes the target data (i.e., the data in event buffer 2) in RDMA write request 3 into the physical address.

[0336] In some embodiments, see Figure 10C The RDMA implementation process shown includes the following steps after step S405: (9) RDMA network card 1 replies with an acknowledgment message (ACK) to RDMA network card 2. (7) RDMA network card 1 generates a complete queue element (CQE) (e.g., CQE3) based on the acknowledgment message and places it in the CQ. (8) The upper layer application determines that the task is completed based on CQE3.

[0337] S406, the IO device node of electronic device 100 distributes screen projection and control events to APP1 through the input subsystem.

[0338] In some embodiments, see Figure 10B Step S103 includes: (11) The input subsystem reads the screen mirroring control event from event buffer1 through the IO device node. (12) The user's screen mirroring control operation is applied to APP1, and the input subsystem distributes the above screen mirroring control event to APP1. In one implementation, after the IMS detects the screen mirroring control event (i.e., touch event) output by the input subsystem, it calls the WMS to determine the target window corresponding to the touch event. The target application corresponding to the above screen mirroring control event is APP1 corresponding to the target window.

[0339] S407, APP1 of electronic device 100 determines the response event corresponding to the screen projection control event. The response event indicates the display content of the screen projection screen 2 after the response. Based on the response event, the screen projection screen 2 after the response is drawn and synthesized. The synthesized screen projection screen 2 is cached in display buffer 3.

[0340] In some embodiments, the specific implementation of step S407 can be referred to the relevant description of step S206, and will not be repeated here.

[0341] S408 After the electronic device 100 determines that the projection screen 2 has been synthesized, it submits a work request WR4 to the RDMA network card 1. The work request WR4 is used to instruct the image data of the projection screen 1 to be written into the display buffer 4 of the electronic device 200.

[0342] In some embodiments, the specific implementation of step S408 can be referred to the relevant description of step S307, and will not be repeated here.

[0343] S409. Based on WR4, electronic device 100 sends RDMA write request 4 to electronic device 200. RDMA write request 4 is used to instruct the data of the projection screen 2 to be written to the display buffer 3 of electronic device 200. RDMA write request 4 carries the virtual address of display buffer 4 and the image data of projection screen 2 in display buffer 3.

[0344] In some embodiments, the specific implementation of step S409 can be referred to the relevant description of step S308, and will not be repeated here.

[0345] S410, based on RDMA write request 4, the RDMA network card 2 of electronic device 200 writes data from display buffer 3 to display buffer 4.

[0346] In some embodiments, the specific implementation of step S410 can be referred to the relevant description of step S309, which will not be repeated here.

[0347] S411, Electronic device 200 displays the projected screen 2 based on the image data stored in display buffer 4.

[0348] The specific implementation of step S411 can be found in the description of step S109 above, and will not be repeated here.

[0349] In some embodiments, when electronic device 100 sends the screen projection screen 2 to electronic device 200, it also sends audio corresponding to the screen projection screen 2 to electronic device 200. For details, please refer to... Figure 9E The relevant descriptions will not be repeated here.

[0350] The embodiments provided in this application can be adapted to each other. For example, in Scheme 2 and Scheme 1, the same technical concepts (e.g., display buffer, audio buffer, and event buffer) and the same implementation steps can be referenced / combined with each other.

[0351] In summary, the implementation of the second scheme, which combines RDMA technology to achieve screen sharing, audio sharing, and screen mirroring control in the screen mirroring scenario, has the following advantages: (1) This device (e.g., electronic device 100 or electronic device 200) can directly write the data required for screen mirroring (e.g., screen mirroring image, screen mirroring audio, or screen mirroring control event) from its own buffer to the buffer of another device without copying the data to the network software stack. (2) Upper-layer applications can directly perform data transmission in user mode without needing to switch contexts between kernel mode and user mode multiple times. (3) Upper-layer applications of this device can access the memory of another device (e.g., display buffer1 / audio buffer1 of electronic device 100, or event buffer2 of electronic device 200) without CPU intervention from the other device, reducing CPU load. (4) No encoding or decoding is required. (5) No data compression is involved, which avoids the problem of image quality degradation caused by lossy compression. Based on the above advantages, CPU load, screen mirroring power consumption, and screen mirroring latency can be effectively reduced, screen mirroring image quality can be guaranteed, and the user experience can be effectively improved.

[0352] Based on the foregoing embodiments, this application provides a screen mirroring method applied to a screen mirroring system, which includes a first electronic device and a second electronic device. For example, see... Figure 11 The above method includes steps S501 to S503.

[0353] S501, The first electronic device receives a first input operation, which is used to trigger the first electronic device to project the screen to the second electronic device.

[0354] In this embodiment, electronic device 100 may also be referred to as a first electronic device; electronic device 200 may also be referred to as a second electronic device. The first input operation may include the aforementioned screen projection operation.

[0355] S502. Based on the first input operation, the remote direct memory access (RDMA) network card of the first electronic device sends a first request to the RDMA network card of the second electronic device. The first request includes the first projection information and the address of the second memory space of the second electronic device. The first request is used to instruct the first projection information to be written into the second memory space.

[0356] In some embodiments, before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, the method further includes: after the first electronic device detects that the first electronic device synthesizes the first projection information and stores it in the first memory space, the RDMA network card of the first electronic device generates the first request based on the first projection information in the first memory space.

[0357] In some embodiments, after the first electronic device receives the first input operation, and before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, the method further includes: the second electronic device sending a first RDMA message to the first electronic device, the first RDMA message including the address of the second memory space of the second electronic device, the second memory space being used to store screen projection information from the first electronic device.

[0358] In some embodiments, before the RDMA network card of the second electronic device sends the first RDMA message to the RDMA network card of the first electronic device, the method further includes: based on the first input operation, the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, and the first RDMA message and the first request are sent through the RDMA connection.

[0359] In some embodiments, before the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, the method further includes: in response to a first input operation, the first electronic device sends a screen projection request to the second electronic device through a first communication module; the first electronic device receives a screen projection response sent by the second electronic device through the first communication module; the establishment of an RDMA connection between the RDMA network card of the first electronic device and the RDMA network card of the second electronic device includes: when screen projection is determined based on the screen projection response, the RDMA network card of the second electronic device and the RDMA network card of the first electronic device establish an RDMA connection. In this embodiment, the first communication module may include the aforementioned communication module 1.

[0360] In some embodiments, before the second electronic device sends the first RDMA message to the first electronic device, the method further includes: the second electronic device registering the second memory space to the RDMA network card of the second electronic device and obtaining the first key corresponding to the address of the second memory space; the first RDMA message also includes the first key, and the first request also includes the first key; based on the first request, the RDMA network card of the second electronic device writes the first projection information into the second memory space, including: after the RDMA network card of the second electronic device determines that the first key included in the first request is the key corresponding to the address of the second memory space, it writes the first projection information into the second memory space.

[0361] In some embodiments, the first projection information includes image data of the first projection screen; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: the display screen of the second electronic device displays the first projection screen based on the data stored in the second memory space.

[0362] In this embodiment of the application, when the first projection information includes the image data of the first projection screen, the first request may include the aforementioned RDMA write request 2, the first projection screen may include the aforementioned projection screen 1, the first memory space may include the aforementioned display buffer 1, the second memory space may include the aforementioned display buffer 2, the first RDMA message may include the aforementioned RDMA message 1, and the first key may include the aforementioned key 1.

[0363] In some embodiments, the first projection information includes audio data of the first projection audio; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: the audio output device of the second electronic device outputs the first projection audio based on the data stored in the second memory space.

[0364] In this embodiment of the application, when the first screen projection information includes the audio data of the first screen projection audio, the first request may include the aforementioned RDMA write request 2, the first screen projection audio may include the aforementioned screen projection audio 1, the first memory space may include the aforementioned audio buffer 1, the second memory space may include the aforementioned audio buffer 2, the first RDMA message may include the aforementioned RDMA message 2, and the first key may include the aforementioned key 2.

[0365] In some embodiments, the first projection information includes image data of the first projection screen; the clocks of the first electronic device and the second electronic device are synchronized, and the method further includes: in a first refresh cycle, writing the first projection screen into the first memory space of the first electronic device; in the next refresh cycle of the first refresh cycle, the display screen of the first electronic device displays the first projection screen based on the data stored in the first memory space; the RDMA network card of the second electronic device writes the first projection information into the second memory space, including: within the first refresh cycle, the RDMA network card of the second electronic device writes the first projection information into the second memory space; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: in the next refresh cycle of the first refresh cycle, the display screen of the second electronic device displays the first projection screen based on the data stored in the second memory space. In this embodiment, the first refresh cycle may include the aforementioned vsync2, and the second refresh cycle may include the aforementioned vsync3.

[0366] S503. Based on the first request, the RDMA network card of the second electronic device writes the first projection information into the second memory space; the second electronic device outputs the first projection information based on the data stored in the second memory space.

[0367] In some embodiments, after the second electronic device outputs the first screen projection information based on the data stored in the second memory space, the method includes: the second electronic device receiving a second input operation; the second electronic device storing a first screen projection reverse control event corresponding to the second input operation in a third memory space; the RDMA network card of the second electronic device sending a second request to the RDMA network card of the first electronic device, the second request including the data of the first screen projection reverse control event and the address of the fourth memory space of the first electronic device, the second request being used to instruct the data of the first screen projection reverse control event to be written into the fourth memory space; based on the second request, the RDMA network card of the first electronic device writing the data of the first screen projection reverse control event into the fourth memory space; and based on the first screen projection reverse control event stored in the fourth memory space, the first electronic device executing a first response event.

[0368] In some embodiments, before the RDMA network card of the second electronic device sends the second request to the RDMA network card of the first electronic device, the method further includes: after the second electronic device detects that the first screen projection and control event is stored in the third memory space, the RDMA network card of the second electronic device generates the second request based on the data of the first screen projection and control event in the third memory space.

[0369] In some embodiments, before the RDMA network card of the second electronic device sends a second request to the RDMA network card of the first electronic device, the method further includes: the first electronic device sending a second RDMA message to the second electronic device, the second RDMA message including the address of a fourth memory space of the first electronic device, the fourth memory space being used to store input events of the first electronic device.

[0370] In this embodiment of the application, the second input operation may include the aforementioned screen mirroring and control operation; the third memory space may include the aforementioned event buffer2; the first screen mirroring and control event may include the aforementioned screen mirroring and control event 1; the fourth memory space may include the aforementioned event buffer1; the second request may include the aforementioned RDMA write request 3; and the second RDMA message may include the aforementioned RDMA message 3.

[0371] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0372] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0373] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0374] In summary, the above description is merely an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made according to the disclosure of the present invention should be included within the scope of protection of the present invention.

Claims

1. A screen projection method, characterized in that, Applied to a screen mirroring system, the screen mirroring system including a first electronic device and a second electronic device, the method includes: The first electronic device receives a first input operation, which triggers the first electronic device to project its screen onto the second electronic device. Based on the first input operation, the RDMA network card of the first electronic device sends a first request to the RDMA network card of the second electronic device. The first request includes first screen projection information and the address of the second memory space of the second electronic device. The first request is used to instruct the first screen projection information to be written into the second memory space. Based on the first request, the RDMA network card of the second electronic device writes the first screen projection information into the second memory space; The second electronic device outputs the first projection information based on the data stored in the second memory space.

2. The method according to claim 1, characterized in that, The first projection information includes image data of the first projection screen; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: The display screen of the second electronic device displays the first projected image based on the data stored in the second memory space.

3. The method according to claim 1, characterized in that, The first projection information includes the audio data of the first projection audio; The second electronic device outputs first projection information based on the data stored in the second memory space, including: The audio output device of the second electronic device outputs the first screen-projected audio based on the data stored in the second memory space.

4. The method according to claim 1, characterized in that, Before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, it also includes: After the first electronic device detects that the first electronic device synthesizes the first projection information and stores it in the first memory space, the RDMA network card of the first electronic device generates the first request based on the first projection information in the first memory space.

5. The method according to claim 4, characterized in that, The first projection information includes image data of the first projection screen; the first electronic device and the second electronic device are clock-synchronized, and the method further includes: In the first refresh cycle, the first screen projection image is written into the first memory space of the first electronic device; In the next refresh cycle of the first refresh cycle, the display screen of the first electronic device displays the first projection screen based on the data stored in the first memory space; The RDMA network card of the second electronic device writes the first projection information into the second memory space, including: During the first refresh cycle, the RDMA network card of the second electronic device writes the first projection information into the second memory space; The second electronic device outputs first projection information based on the data stored in the second memory space, including: In the next refresh cycle of the first refresh cycle, the display screen of the second electronic device displays the first projection screen based on the data stored in the second memory space.

6. The method according to claim 1, characterized in that, After the first electronic device receives the first input operation, and before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, the method further includes: The second electronic device sends a first RDMA message to the first electronic device. The first RDMA message includes the address of the second memory space of the second electronic device, which is used to store screen projection information from the first electronic device.

7. The method according to claim 6, characterized in that, Before the second electronic device sends the first RDMA message to the first electronic device, it also includes: The second electronic device registers the second memory space with the RDMA network card of the second electronic device and obtains the first key corresponding to the address of the second memory space; the first RDMA message also includes the first key, and the first request also includes the first key; Based on the first request, the RDMA network card of the second electronic device writes the first screen projection information into the second memory space, including: After the RDMA network card of the second electronic device determines that the first key included in the first request is the key corresponding to the address of the second memory space, it writes the first screen projection information into the second memory space.

8. The method according to claim 6, characterized in that, Before the RDMA network card of the second electronic device sends the first RDMA message to the RDMA network card of the first electronic device, it also includes: Based on the first input operation, the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, and the first RDMA message and the first request are sent through the RDMA connection.

9. The method according to claim 8, characterized in that, Before the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, the following steps are also included: In response to the first input operation, the first electronic device sends a screen mirroring request to the second electronic device through the first communication module; The first electronic device receives the screen projection response sent by the second electronic device through the first communication module; The RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, including: When screen projection is determined based on the projection response, the RDMA network card of the second electronic device and the RDMA network card of the first electronic device establish an RDMA connection.

10. The method according to claim 1, characterized in that, After the second electronic device outputs the first projection information based on the data stored in the second memory space, the method includes: The second electronic device receives the second input operation; The second electronic device stores the first screen projection reverse control event corresponding to the second input operation in the third memory space; The RDMA network card of the second electronic device sends a second request to the RDMA network card of the first electronic device. The second request includes the data of the first screen projection and control event and the address of the fourth memory space of the first electronic device. The second request is used to instruct the data of the first screen projection and control event to be written into the fourth memory space. Based on the second request, the RDMA network card of the first electronic device writes the data of the first screen projection and control event into the fourth memory space; Based on the first screen projection and control event stored in the fourth memory space, the first electronic device executes the first response event.

11. The method according to claim 10, characterized in that, Before the RDMA network card of the second electronic device sends the second request to the RDMA network card of the first electronic device, it also includes: After the second electronic device detects that the first screen projection and control event is stored in the third memory space, the RDMA network card of the second electronic device generates the second request based on the data of the first screen projection and control event in the third memory space.

12. The method according to claim 10, characterized in that, Before the RDMA network card of the second electronic device sends a second request to the RDMA network card of the first electronic device, the method further includes: The first electronic device sends a second RDMA message to the second electronic device. The second RDMA message includes the address of the fourth memory space of the first electronic device, which is used to store input events of the first electronic device.

13. A screen projection method, characterized in that, Applied to a first electronic device, including: The first electronic device receives a first input operation, which triggers the first electronic device to project its screen onto the second electronic device. Based on the first input operation, the RDMA network card of the first electronic device sends a first request to the RDMA network card of the second electronic device. The first request includes first projection information and the address of the second memory space of the second electronic device. The first request is used to instruct the first projection information to be written into the second memory space.

14. The method according to claim 13, characterized in that, The first projection information includes image data of the first projection screen; the first projection information stored in the second memory space is used to display the first projection screen on the display screen of the second electronic device.

15. The method according to claim 13, characterized in that, The first screen projection information includes audio data of the first screen projection audio; the first screen projection information stored in the second memory space is used by the audio output device of the second electronic device to output the first screen projection audio.

16. The method according to claim 13, characterized in that, Before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, it also includes: After the first electronic device detects that the first electronic device synthesizes the first projection information and stores it in the first memory space, the RDMA network card of the first electronic device generates the first request based on the first projection information in the first memory space.

17. The method according to claim 13, characterized in that, After the first electronic device receives the first input operation, and before the RDMA network card of the first electronic device sends the first request to the RDMA network card of the second electronic device, the method further includes: The first electronic device receives a first RDMA message sent by the second electronic device. The first RDMA message includes the address of the second memory space of the second electronic device, and the second memory space is used to store screen projection information from the first electronic device.

18. The method according to claim 17, characterized in that, Before the RDMA network card of the second electronic device sends the first RDMA message to the RDMA network card of the first electronic device, it also includes: Based on the first input operation, the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, and the first RDMA message and the first request are sent through the RDMA connection.

19. The method according to claim 18, characterized in that, Before the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, the following steps are also included: In response to the first input operation, the first electronic device sends a screen mirroring request to the second electronic device through the first communication module; The first electronic device receives the screen projection response sent by the second electronic device through the first communication module; The RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection, including: When screen projection is determined based on the projection response, the RDMA network card of the first electronic device and the RDMA network card of the second electronic device establish an RDMA connection.

20. The method according to claim 13, characterized in that, After the RDMA network card of the first electronic device sends a first request to the RDMA network card of the second electronic device, the method includes: The RDMA network card of the first electronic device receives a second request sent by the RDMA network card of the second electronic device. The second request includes data of the first screen projection and control event and the address of the fourth memory space of the first electronic device. The second request is used to instruct the data of the first screen projection and control event to be written into the fourth memory space. Based on the second request, the RDMA network card of the first electronic device writes the data of the first screen projection and control event into the fourth memory space; Based on the first screen projection and control event stored in the fourth memory space, the first electronic device executes the first response event.

21. The method according to claim 20, characterized in that, Before the RDMA network card of the first electronic device receives the second request sent by the RDMA network card of the second electronic device, the method further includes: The first electronic device sends a second RDMA message to the second electronic device. The second RDMA message includes the address of the fourth memory space of the first electronic device, which is used to store input events.

22. A screen projection method, characterized in that, Applied to a second electronic device, including: The RDMA network card of the second electronic device receives a first request sent by the RDMA network card of the first electronic device. The first request includes first screen projection information and the address of the second memory space of the second electronic device. The first request is used to instruct the first screen projection information to be written into the second memory space. Based on the first request, the RDMA network card of the second electronic device writes the first screen projection information into the second memory space; The second electronic device outputs the first projection information based on the data stored in the second memory space.

23. The method according to claim 22, characterized in that, The first projection information includes image data of the first projection screen; the second electronic device outputs the first projection information based on the data stored in the second memory space, including: The display screen of the second electronic device displays the first projected image based on the data stored in the second memory space.

24. The method according to claim 22, characterized in that, The first projection information includes the audio data of the first projection audio; The second electronic device outputs first projection information based on the data stored in the second memory space, including: The audio output device of the second electronic device outputs the first screen-projected audio based on the data stored in the second memory space.

25. The method according to claim 22, characterized in that, Before the RDMA network card of the second electronic device receives the first request sent by the RDMA network card of the first electronic device, the method further includes: The second electronic device sends a first RDMA message to the first electronic device. The first RDMA message includes the address of the second memory space of the second electronic device, which is used to store screen projection information from the first electronic device.

26. The method according to claim 25, characterized in that, Before the second electronic device sends the first RDMA message to the first electronic device, it also includes: The second electronic device registers the second memory space with the RDMA network card of the second electronic device and obtains the first key corresponding to the address of the second memory space; the first RDMA message also includes the first key, and the first request also includes the first key; Based on the first request, the RDMA network card of the second electronic device writes the first screen projection information into the second memory space, including: After the RDMA network card of the second electronic device determines that the first key included in the first request is the key corresponding to the address of the second memory space, it writes the first screen projection information into the second memory space.

27. The method according to claim 25, characterized in that, Before the RDMA network card of the second electronic device sends the first RDMA message to the RDMA network card of the first electronic device, it also includes: The RDMA network card of the second electronic device and the RDMA network card of the first electronic device establish an RDMA connection, and the first RDMA message and the first request are sent through the RDMA connection.

28. The method according to claim 22, characterized in that, After the second electronic device outputs the first projection information based on the data stored in the second memory space, the method includes: The second electronic device receives the second input operation; The second electronic device stores the first screen projection reverse control event corresponding to the second input operation in the third memory space; The RDMA network card of the second electronic device sends a second request to the RDMA network card of the first electronic device. The second request includes the data of the first screen projection and control event and the address of the fourth memory space of the first electronic device. The second request is used to instruct the data of the first screen projection and control event to be written into the fourth memory space, which is used to store the input events of the first electronic device.

29. The method according to claim 28, characterized in that, Before the RDMA network card of the second electronic device sends the second request to the RDMA network card of the first electronic device, it also includes: After the second electronic device detects that the first screen projection and control event is stored in the third memory space, the RDMA network card of the second electronic device generates the second request based on the data of the first screen projection and control event in the third memory space.

30. The method according to claim 28, characterized in that, Before the RDMA network card of the second electronic device sends a second request to the RDMA network card of the first electronic device, the method further includes: The second electronic device receives a second RDMA message sent by the first electronic device, the second RDMA message including the address of the fourth memory space of the first electronic device.

31. An electronic device comprising a memory, one or more processors, and one or more computer programs; wherein, The one or more computer programs are stored in the memory; characterized in that, when the one or more processors execute the one or more computer programs, they cause the electronic device to implement the method as described in any one of claims 13 to 21 or 22 to 30.

32. A computer storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 13 to 30.

Images