Device using method and device

Abstract

The present application provides a device use method and device, in which a screen projection source device runs a first application and projects the interface of the first application to a screen projection destination device so that the screen projection destination device displays the interface of the first application on the screen projection window, and the first application of the screen projection source device uses the multimedia device of the screen projection source device to process multimedia information; the screen projection destination device responds to a first operation on its screen projection window by sending a first operation message to the screen projection source device, and in response to the first operation message, the screen projection source device calls the multimedia device of the screen projection destination device to process multimedia information; in response to a second operation on the screen projection source device, the screen projection source device calls the multimedia device of the screen projection source device to process multimedia information. Through the above method, it is possible to achieve rapid switching of the use of multimedia devices of different devices in the screen projection scenario, so that users can enjoy a more user-friendly experience.

Description

Technical Field

[0001] The present application relates to the technical field of electronic equipment, and in particular to a device use method and device. Background Art

[0002] Mobile phones can play videos, and can also display documents, pictures, application interfaces, or web pages. Since the display screen of mobile phones is small, the content displayed on the mobile phone can be projected to a display device (such as a tablet, computer, or another mobile phone) through screen projection technology. In the screen projection scenario, the operations performed by the user in the projection window on the display device will also be performed on the mobile phone accordingly.

[0003] In the prior art, in the screen projection scenario, when the user's operation on the display device involves the use of audio playback, recording or camera functions, the audio device or camera device of the mobile phone is called instead of the audio device or camera device of the display device. If the user wants to call the device on the display device, it is necessary to manually set it up on the mobile phone to switch. This method of switching between the device of the screen projection destination and the device of the screen projection source is not convenient enough, which affects the user's operating experience. Summary of the invention

[0004] The present application provides a method for using a device, which can enable an application running on a projection source device to intelligently switch the use of different projection device devices in a projection scenario based on the user's operations on different projection devices, thereby improving the user's operating experience when switching between the projection source device and the projection destination device.

[0005] In a first aspect, the present application provides a device use method that can be applied to a screen projection source device, including: the screen projection source device runs a first application, the screen projection source device projects the interface of the first application to the screen projection destination device, so that the screen projection destination device displays a screen projection window on its display screen, and the screen projection window displays the interface of the first application. At this time, the first application uses the first multimedia device of the screen projection source device to process multimedia information, and the screen projection source device and the screen projection destination device are in a screen projection connection state; the screen projection source device receives a first operation message from the screen projection destination device, wherein the first operation message A message is generated by the screen projection destination device in response to a first operation on the screen projection window detected by the screen projection destination device; in response to the first operation message, the first application of the screen projection source device will switch to using the second multimedia device of the screen projection destination device to replace the first multimedia device of the screen projection source device for multimedia information processing; the screen projection source device detects the second operation on the screen projection source device, and in response to the second operation, the first application of the screen projection source device resumes using the first multimedia device of the screen projection source device for multimedia information processing. At this time, the screen projection source device and the screen projection destination device are still in a screen projection connection state.

[0006] Through the above method, in the screen projection scenario, when the user is using a certain application, it is possible to quickly switch to the multimedia device of the device the user is operating to process multimedia information without the user having to manually set up the switch. This makes the user's use between the screen projection source device and the screen projection destination device more continuous and smooth, allowing the user to enjoy a more user-friendly experience.

[0007] In combination with the first aspect, in some implementations of the first aspect, the first multimedia device includes at least one of a first audio device and a first camera device, the second multimedia device includes at least one of a second audio device and a second camera device, and multimedia information processing includes audio information acquisition, audio information playback, video information acquisition, and image information acquisition, wherein the first audio device and the second audio device are used for audio information acquisition, or for audio information playback, or for audio information acquisition and audio information playback, and the first camera device and the second camera device are used for video information acquisition, or for image information acquisition.

[0008] Exemplarily, the first application of the screen projection source device can be a video call application, which can call the first audio device and the first camera device of the screen projection source device itself to collect audio information, play audio information and collect video information; in response to the first operation message generated by the user's first operation on the screen projection window of the screen projection destination device, the video call application of the screen projection source device switches to call the second audio device and the second camera device of the screen projection destination device to collect audio information, play audio information and collect video information; then, in response to the user's second operation on the screen projection source device, the video call application of the screen projection source device resumes calling the first audio device and the first camera device of the screen projection source device to collect audio information, play audio information and collect video information.

[0009] Exemplarily, the first application of the screen projection source device can be an audio and video application, which can call the first audio device of the screen projection source device itself to play audio information; in response to the first operation message generated by the user's first operation on the screen projection window of the screen projection destination device, the audio and video application of the screen projection source device switches to call the second audio device of the screen projection destination device to play audio information; then, in response to the user's second operation on the screen projection source device, the audio and video application of the screen projection source device resumes calling the first audio device of the screen projection source device to play audio information.

[0010] Exemplarily, the first application of the screen projection source device can be a camera application, which can call the first camera device of the screen projection source device itself to collect video information or image information; in response to the first operation message generated by the user's first operation on the screen projection window of the screen projection destination device, the camera application of the screen projection source device switches to call the second camera device of the screen projection destination device to collect video information or image information; then, in response to the user's second operation on the screen projection source device, the camera application of the screen projection source device resumes calling the first camera device of the screen projection source device to collect video information or image information.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the first operation message includes first identification information of the screen projection destination device.

[0012] In combination with the first aspect, in some implementations of the first aspect, the first operation message also includes a first operation input event, and the above-mentioned first operation input event is used to instruct the screen projection source device to perform related operations corresponding to the first operation input event.

[0013] In combination with the first aspect, in certain implementations of the first aspect, in response to the first operation message, the first application of the screen projection source device switches to using the second multimedia device of the screen projection destination device to process multimedia information, including: the screen projection source device determines that the first identification information of the received screen projection destination device is different from the last received identification information saved by the screen projection source device, and generates a first switching instruction; in response to the first switching instruction, the first application of the screen projection source device calls the second multimedia device of the screen projection destination device, such as calling the second audio device of the screen projection destination device, or calling the second camera device of the screen projection destination device, or calling the second audio device and the second camera device of the screen projection destination device at the same time.

[0014] In combination with the first aspect, in certain implementation methods of the first aspect, after the screen projection source device determines that the first identification information received from the screen projection destination device is different from the last received identification information saved by the screen projection source device, the last received identification information is updated to the first identification information of the screen projection destination device.

[0015] In combination with the first aspect, in certain implementations of the first aspect, in response to the first switching instruction, the first application of the screen projection source device calls the second multimedia device of the screen projection destination device, including: the first application of the screen projection source device calls the first multimedia abstract module of the hardware abstraction layer of the screen projection source device according to the first corresponding relationship, wherein the first corresponding relationship is a mapping relationship between the first identification information of the screen projection destination device and the first multimedia abstract module, the first multimedia abstract module includes at least one of the first audio abstract module and the first camera abstract module, the first audio abstract module is used to call the second audio device of the screen projection destination device to collect audio information, or to play audio information, or to collect audio information and play audio information at the same time, and the first camera abstract module is used to call the second camera device of the screen projection destination device to collect video information, or to collect image information.

[0016] Exemplarily, the first application of the screen projection source device can be a video call application, which calls the first audio abstract module and the first camera abstract module of the hardware abstraction layer of the screen projection source device according to the first corresponding relationship. The above-mentioned first audio abstract module is used to call the second audio device of the screen projection destination device to collect audio information and play audio information, and the above-mentioned first camera module is used to call the second camera module of the screen projection destination device to collect video information.

[0017] Exemplarily, the first application of the screen projection source device can also be an audio and video application. The audio and video application calls the first audio abstract module of the hardware abstraction layer of the screen projection source device according to the first corresponding relationship. The above-mentioned first audio abstract module is used to call the second audio device of the screen projection destination device to play audio information.

[0018] Exemplarily, the first application of the screen projection source device can also be a camera application. The camera application calls the first camera abstraction module of the hardware abstraction layer of the screen projection source device according to the first corresponding relationship. The above-mentioned first camera module is used to call the second camera module of the screen projection destination device to collect image information, or to collect video information.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the screen projection source device responds to the second operation, and the first application of the screen projection source device resumes using the first multimedia device of the screen projection source device to process multimedia information, including: in response to the second operation, the screen projection source device generates second identification information; the screen projection source device determines that the second identification information of the screen projection source device is different from the last received identification information saved by the screen projection source device, and generates a second switching instruction; in response to the second switching instruction, the first application of the screen projection source device calls the first multimedia device of the screen projection source device, such as calling the first audio device, or calling the first camera device, or calling the first audio device and the first camera device at the same time.

[0020] In combination with the first aspect, in certain implementation methods of the first aspect, after the screen projection source device determines that the second identification information of the screen projection source device received is different from the last received identification information saved by the screen projection source device, the last received identification information is updated to the second identification information of the screen projection source device.

[0021] In combination with the first aspect, in certain implementations of the first aspect, in response to a second switching instruction, the first application of the screen projection source device calls the first multimedia device of the screen projection source device, including: the first application of the screen projection source device calls the second multimedia abstract module of the hardware abstraction layer of the screen projection source device according to the second correspondence, wherein the second correspondence is a mapping relationship between the second identification information of the screen projection source device and the second multimedia abstract module, the second multimedia abstract module includes at least one of a second audio abstract module and a second camera abstract module, the second audio abstract module is used to call the first audio device of the screen projection source device to collect audio information, or to play audio information, or to collect audio information and play audio information at the same time, and the second camera abstract module is used to call the first camera device of the screen projection source device to collect video information, or to collect image information.

[0022] Exemplarily, the first application of the screen projection source device can be a video call application, which calls the second audio abstract module and the second camera abstract module of the hardware abstraction layer of the screen projection source device according to the second corresponding relationship. The second audio abstract module is used to call the first audio device of the screen projection source device to collect audio information and play audio information, and the second camera module is used to call the first camera module of the screen projection source device to collect video information.

[0023] Exemplarily, the first application of the screen projection source device can also be an audio and video application. The audio and video application calls the second audio abstraction module of the hardware abstraction layer of the screen projection source device according to the second corresponding relationship. The second audio abstraction module is used to call the first audio device of the screen projection source device to play audio information.

[0024] Exemplarily, the first application of the screen projection source device can also be a camera application. The camera application calls the second camera abstraction module of the hardware abstraction layer of the screen projection source device according to the second corresponding relationship. The above-mentioned second camera module is used to call the first camera module of the screen projection source device to collect video information, or to collect image information.

[0025] In combination with the first aspect, in certain implementations of the first aspect, after the screen projection source device projects the interface of the first application to the screen projection destination device, it also includes: the screen projection source device projects the interface of the first application to another screen projection destination device, so that the other screen projection destination device displays another screen projection window on its display screen, and the other screen projection window displays the interface of the first application; before the screen projection source device detects the second operation on the screen projection source device, it also includes: the screen projection source device receives a third operation message from the other screen projection destination device, wherein the third operation message is a message generated by the other screen projection destination device in response to the third operation on the other screen projection window detected by the other screen projection destination device; in response to the third operation message, the first application of the screen projection source device switches to use the third multimedia device of the other screen projection destination device to perform multimedia information processing;

[0026] In combination with the first aspect, in certain implementations of the first aspect, the third multimedia device includes at least one of a third audio device and a third camera device, and the multimedia information processing includes audio information acquisition, audio information playback, video information acquisition, and image information acquisition, wherein the third audio device is used for audio information acquisition, or for audio information playback, or for audio information acquisition and audio information playback, and the third camera device is used for video information acquisition, or for image information acquisition.

[0027] In combination with the first aspect, in certain implementations of the first aspect, the third operation message includes third identification information of another screen projection destination device.

[0028] In combination with the first aspect, in certain implementations of the first aspect, in response to a third operation message, the first application of the screen projection source device switches to using the third multimedia device of another screen projection destination device to perform multimedia information processing, including: the screen projection source device determines that the third identification information received from the other screen projection destination device is different from the last received identification information saved by the screen projection source device, and generates a third switching instruction; in response to the third switching instruction, the first application of the screen projection source device calls the third multimedia device of the other screen projection destination device, such as calling the third audio device, or calling the third camera device, or calling the third audio device and the third camera device at the same time.

[0029] In combination with the first aspect, in certain implementations of the first aspect, after the screen projection source device determines that the third identification information received from another screen projection destination device is different from the last received identification information saved by the screen projection source device, the last received identification information is updated to the third identification information of the other screen projection destination device.

[0030] In combination with the first aspect, in certain implementations of the first aspect, in response to a third switching instruction, the first application of the screen projection source device calls the third multimedia device of another screen projection destination device, including: the first application of the screen projection source device calls the third multimedia abstract module of the hardware abstraction layer of the screen projection source device according to the third correspondence, wherein the third correspondence is a mapping relationship between the third identification information of the other screen projection destination device and the third multimedia abstract module, the third multimedia abstract module includes at least one of a third audio abstract module and a third camera abstract module, the third audio abstract module is used to call the third audio device of the other screen projection destination device to collect audio information, or to play audio information, or to collect audio information and play audio information at the same time, and the third camera abstract module is used to call the third camera device of the other screen projection destination device to collect video information, or to collect image information.

[0031] In combination with the first aspect, in some implementation methods of the first aspect, the first operation on the projection window of the projection destination device includes: touch operation, air gesture operation, mouse click operation, and voice control operation on the projection window.

[0032] In combination with the first aspect, in some implementations of the first aspect, the second operation on the screen projection source device includes: touch operation, air gesture operation, and voice control operation on the screen of the screen projection source device.

[0033] In a second aspect, the present application provides a device using method, which can be applied to a screen projection source device, including: the screen projection source device runs a first application, and the screen projection source device projects the interface of the first application to the screen projection destination device, so that the screen projection destination device displays a projection window on its display screen, and then the projection window displays the interface of the first application; the first application of the screen projection source device uses the first audio device and the first camera device of the screen projection source device; the screen projection source device receives a first operation message from the screen projection destination device, wherein the first operation message is a message generated by the screen projection destination device in response to a first operation on the screen projection window detected by the screen projection destination device; in response to the first operation message, the first application of the screen projection source device switches to use at least one of the second audio device and the second camera device of the screen projection destination device according to the device capability of the screen projection destination device; the screen projection source device detects a second operation on the screen projection source device, and in response to the second operation, the first application of the screen projection source device resumes using the first audio device and the first camera device of the screen projection source device.

[0034] Through the above method, in a screen projection scenario, such as when a user uses a video call application, the device of the projection target device can be partially or fully used to collect audio information, play audio information, and collect video information according to the device capabilities of the projection target device. This allows the application on the projection source device to make adaptive changes when using the above method to call the projection target device with limited device capabilities, thereby optimizing the user experience.

[0035] In a third aspect, the present application provides a device using method, which can be applied to a screen projection source device, including: the screen projection source device runs a first application, and the screen projection source device projects the interface of the first application to the screen projection destination device, so that the screen projection destination device displays a projection window on its display screen, and then the projection window displays the interface of the first application; the first application of the screen projection source device uses the first audio device and the first camera device of the screen projection source device; the screen projection source device receives a first operation message from the screen projection destination device, wherein the first operation message is a message generated by the screen projection destination device in response to a first operation on the screen projection window detected by the screen projection destination device; in response to the first operation message, the first application of the screen projection source device uses at least one of the second audio device and the second camera device of the screen projection destination device according to the setting; the screen projection source device detects a second operation on the screen projection source device, and in response to the second operation, the first application of the screen projection source device resumes using the first audio device and the first camera device of the screen projection source device.

[0036] Through the above method, in a screen projection scenario, such as when a user uses a video call application, the devices of the screen projection destination device can be partially or fully called to collect audio information, play audio information, and collect video information according to the settings, so that the above method can make adaptive changes according to the user's personalized settings, thereby optimizing the user experience.

[0037] In a fourth aspect, the present application provides a device using method, which can be applied to a screen projection source device, including: the screen projection source device runs a first application in the background, the screen projection source device runs a second application, and the screen projection source device projects the interface of the second application to the screen projection destination device, so that the screen projection destination device displays a projection window on its display screen, and then the projection window displays the interface of the second application; the first application of the screen projection source device uses the first audio device and the first camera device of the screen projection source device; the screen projection source device receives a first operation message from the screen projection destination device, wherein the first operation message is a message generated by the screen projection destination device in response to a first operation on the screen projection window detected by the screen projection destination device; in response to the first operation message, the first application of the screen projection source device uses at least one of the second audio device and the second camera device of the screen projection destination device; the screen projection source device detects a second operation on the screen projection source device, and in response to the second operation, the first application of the screen projection source device resumes using the first audio device and the first camera device of the screen projection source device.

[0038] Through the above method, in the screen projection scenario, if the music application of the screen projection source device is playing music in the background, the above method can be used to switch the music played in the background between the screen projection source device and the screen projection destination device. The user does not need to switch the background application to the foreground and then switch the device usage, which saves the user's operating steps and brings a more convenient experience to the user.

[0039] In a fifth aspect, the present application provides a method for using a device, which can be applied to a screen projection destination device, including: the screen projection destination device receives the projection data from the screen projection source device, and in response to the projection data, the screen projection destination device displays a projection window on its display screen, and the projection window displays the interface of a first application running on the screen projection source device, wherein the screen projection source device uses the first multimedia device of the screen projection source device to process multimedia information; the screen projection destination device detects a first operation on the screen projection window; in response to the first operation, the screen projection destination device generates first operation information and sends the first operation information to the screen projection source device, wherein the first operation information can trigger the screen projection source device to switch to using the second multimedia device of the screen projection destination device to process multimedia information.

[0040] Through the above method, the screen projection destination device can detect the first operation from the user, thereby generating a first operation message. Through this message, the application on the screen projection source device can be triggered to use the multimedia device of the screen projection destination device to process multimedia information. The user does not need to operate on the screen projection source device to trigger the switching of device use, which reduces the user's operation steps, greatly simplifies the operation process of device switching, and improves the user experience.

[0041] In combination with the fifth aspect, in some implementations of the fifth aspect, the first multimedia device includes at least one of a first audio device and a first camera device, the second multimedia device includes at least one of a second audio device and a second camera device, and multimedia information processing includes audio information acquisition, audio information playback, video information acquisition, and image information acquisition, wherein the first audio device and the second audio device are used for audio information acquisition, or for audio information playback, or for audio information acquisition and audio information playback, and the first camera device and the second camera device are used for video information acquisition, or for image information acquisition.

[0042] In combination with the fifth aspect, in some implementations of the fifth aspect, the first operation information includes first identification information of the screen projection destination device.

[0043] In combination with the fifth aspect, in some implementation methods of the fifth aspect, the first operation on the projection window of the projection destination device includes: touch operation, air gesture operation, mouse click operation, and voice control operation on the projection window.

[0044] In the sixth aspect, the present application provides a screen projection source device, including a processor, a memory, a transceiver and a display screen; wherein the memory is used to store computer execution instructions; when the screen projection source device is running, the processor executes the computer execution instructions stored in the memory, so that the screen projection source device executes any one of the methods described in the first aspect to the fourth aspect.

[0045] In the seventh aspect, the present application provides a screen projection destination device, including a processor, a memory, a transceiver and a display screen; wherein the memory is used to store computer execution instructions; when the screen projection destination device is running, the processor executes the computer execution instructions stored in the memory, so that the screen projection destination device executes the method described in the fifth aspect.

[0046] In an eighth aspect, the present application provides a screen projection system, comprising the screen projection source device described in the sixth aspect and the screen projection destination device described in the seventh aspect.

[0047] In a ninth aspect, the present application provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on a device, the device executes any one of the methods described in the first to fifth aspects.

[0048] In a tenth aspect, the present application provides a computer program product, which, when executed on a device, enables the device to execute any one of the methods described in the first to fifth aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a scenario of a screen projection system provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the device structure of a screen projection device provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the software architecture of a screen projection device provided in an embodiment of the present application;

[0052] Figure 4 A flowchart of a method for using a device provided in an embodiment of the present application;

[0053] Figure 5 A working principle diagram of a device use method provided in an embodiment of the present application;

[0054] Figures 6A-6D An application schematic diagram of a device use method provided in an embodiment of the present application;

[0055] Figures 7A-7C A working principle diagram of a device use method provided in an embodiment of the present application;

[0056] Figures 8A-8D An application schematic diagram of a device use method provided in an embodiment of the present application;

[0057] Fig. 9 A working principle diagram of a device use method provided in an embodiment of the present application;

[0058] Fig.10 An application schematic diagram of a device use method provided in an embodiment of the present application;

[0059] Fig.11 An application schematic diagram of a device use method provided in an embodiment of the present application;

[0060] Fig.12 A working principle diagram of a device use method provided in an embodiment of the present application;

[0061] Figures 13A-13C A working principle diagram of a device use method provided in an embodiment of the present application;

[0062] Fig.14 A working principle diagram of a device use method provided in an embodiment of the present application;

[0063] Fig.15 A working principle diagram of a device use method provided in an embodiment of the present application;

[0064] Figures 16A-16C A working principle diagram of a device use method provided in an embodiment of the present application;

[0065] Fig.17An application schematic diagram of a device use method provided in an embodiment of the present application;

[0066] Figures 18A-18D A working principle diagram of a device use method provided in an embodiment of the present application;

[0067] Fig.19 A flowchart of another device use method provided in an embodiment of the present application;

[0068] Fig. 20 An application schematic diagram of another device use method provided in an embodiment of the present application;

[0069] Figures 21A-21D A working principle diagram of another device use method provided in an embodiment of the present application;

[0070] Fig. 22 A schematic diagram of a scenario of a method for using a device provided in an embodiment of the present application.

[0071] Fig.23 A schematic diagram of another scenario of a method for using a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. In the embodiments of the present application, unless otherwise specified, "multiple" includes "two".

[0073] Screen projection refers to a device projecting the content of its running application onto the display screen or display medium of multiple other devices. It is a typical way of information synchronization between devices with screens. In the embodiment of the present application, the device that projects the content of its running application can be called a screen projection source device, and the device that receives the projection of the screen projection source device and displays the content of the application running on the screen projection source device can be called a screen projection destination device. Figure 1 As shown, the screen projection source device can project the content of the application it is running onto the screens of multiple devices.

[0074] When casting a screen, the source device can compress and encode the video stream data of the application content running on its device and send it to the destination device. The destination device receives the video stream data of the source device, decodes it, and displays the content of the application running on the source device on the display screen of the destination device. For the sake of convenience, the window on the display screen of the destination device that displays the content of the application running on the source device can be called a casting window.

[0075] When casting the screen, the screen casting source device may be in certain interfaces, in which related applications involving multimedia information processing may be in progress, and these applications will use the related multimedia devices of the screen casting source device accordingly. Among them, the multimedia devices may include audio devices and camera devices, and the above-mentioned multimedia devices can perform multimedia information processing, and the above-mentioned multimedia information processing includes audio information playback, audio information collection, video information collection and image information collection. For example, when the screen casting source device executes a music playback application, the screen casting source device will use its own audio device, such as a speaker, to play audio information; for another example, when the screen casting source device executes a recording application, the screen casting source device will use its own audio device, such as a microphone, to collect audio information; for another example, when the screen casting source device executes a camera application, the screen casting source device will use its own camera device, such as a camera, to collect video information; for another example, when the screen casting source device executes a video call application, the screen casting source device will use its own audio device and camera device, such as a speaker, a microphone and a front camera, to play and collect audio information and collect video information. The target device will only display the corresponding projection window, and the multimedia device of the target device will not be used. If the user operates the projection window on the target device, the application running on the source device will still use its own multimedia device, and the user may prefer to use the multimedia device of the target device.

[0076] The present invention provides a device switching method which can be applied to Figure 1 The scenario shown. The screen projection source device is using its own multimedia device. At this time, the user operates the screen projection window of the screen projection destination device. In response to the operation, the screen projection destination device sends the operation message to the screen projection source device. The screen projection source device receives the operation message from the screen projection destination device and switches to use the multimedia device of the screen projection destination device. When the user turns to operate the screen projection source device, the screen projection source device switches back to using its own multimedia device. In this way, the switching use of the screen projection source device devices and the screen projection destination device devices is realized in the screen projection scenario, thereby improving the user experience of operating between screen projection devices.

[0077] Exemplarily, the screen projection source device and the screen projection destination device can be a mobile phone, a speaker, a tablet computer, a television (also known as a smart TV, a smart screen or a large-screen device), a laptop computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a vehicle-mounted device, a virtual reality device, and other devices with screen display functions. The embodiments of the present application do not impose any restrictions on this.

[0078] For example, a mobile phone is used as the screen projection source device and the screen projection destination device. Figure 2 A schematic diagram of the structure of a mobile phone is shown.

[0079] The mobile phone may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, etc.

[0080] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone. In other embodiments of the present application, the mobile phone may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented as devices, software, or a combination of software and devices.

[0081] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0082] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0083] The wireless communication function of the mobile phone can be realized through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0084] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0085] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied on mobile phones. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the same device as at least some modules of the processor 210.

[0086] The wireless communication module 260 can provide wireless communication solutions for use on mobile phones, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0087] In some embodiments, the mobile phone's antenna 1 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the mobile phone can communicate with the network and other devices through wireless communication technology.

[0088] The mobile phone implements display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0089] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0090] The mobile phone can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0091] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile phone may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0092] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0093] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the mobile phone by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0094] The mobile phone can realize audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the earphone interface 270D, and the application processor.

[0095] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

[0096] The speaker 270A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The mobile phone can listen to music or listen to hands-free calls through the speaker 270A.

[0097] The receiver 270B, also called a "handset", is used to convert audio electrical signals into sound signals. When the mobile phone receives a call or voice message, the voice can be heard by placing the receiver 270B close to the ear.

[0098] Microphone 270C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 270C to input the sound signal into the microphone 270C. The mobile phone can be provided with at least one microphone 270C. In other embodiments, the mobile phone can be provided with two microphones 270C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the mobile phone can also be provided with three, four or more microphones 270C to realize the collection of sound signals, noise reduction, identification of sound sources, and realization of directional recording function, etc.

[0099] The earphone interface 270D is used to connect a wired earphone and can be a USB interface 230 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0100] The sensor module 280 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0101] Of course, the mobile phone may also include a charging management module, a power management module, a battery, buttons, indicators, and one or more SIM card interfaces, etc., and the embodiments of the present application do not impose any restrictions on this.

[0102] The software system of the mobile phone can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the mobile phone. Of course, in other operating systems, as long as the functions implemented by each functional module are similar to the embodiments of the present application.

[0103] Figure 3 It is a software structure block diagram of the mobile phone of the embodiment of the present application.

[0104] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, the HAL (hardware abstraction layer) layer, and the kernel layer.

[0105] The application layer can include a series of application packages.

[0106] like Figure 3 As shown, the application layer can be installed with applications such as calls, memos, browsers, contacts, cameras, galleries, calendars, maps, Bluetooth, music, videos, and short messages.

[0107] It should be noted that these applications installed in the application layer can have audio functions, wherein the audio functions include playback functions and recording functions. For example, a music application can play audio, and a camera application can output a system preset shutter sound when taking a photo. For another example, a video application can output audio corresponding to the video screen while playing a video. For another example, a map application can output navigation voice after turning on the navigation function. For another example, a recording application can record voice. In an embodiment of the present application, an application with an audio function may be referred to as an audio application. These applications installed in the application layer may have a video function. For example, a camera application can take photos or videos. For another example, a video call application will also turn on the camera to capture video of the user when running a video call. In an embodiment of the present application, an application with a video function may be referred to as a video application.

[0108] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0109] like Figure 3 As shown, the application framework layer may include a service framework, which includes modules such as audio service, Camera service, and display service, as well as corresponding access components.

[0110] Among them, for audio applications, the Android system is provided with audio services and audio access components for realizing audio functions for audio applications. In the application framework layer, the above audio services may include an audio manager (AudioManager), an audio information player and an audio processor (not shown in the figure).

[0111] Exemplarily, the audio information player may be a player such as AudioTrack or MediaPlayer. The audio processor may be an audio processor such as AudioFlinger. When the audio application plays audio, the audio information player may be called to input corresponding audio data into the audio information player. For example, the audio application may input original audio data into the audio information player, and the audio information player may parse, decapsulate or decode the original audio data to obtain frames of PCM (pulse code modulation) data. Alternatively, the audio application may also directly output PCM data to the audio information player. Furthermore, the audio information player may send the audio data to the audio processor for audio processing.

[0112] Specifically, the audio processor can perform audio processing on the audio data according to the audio playback strategy output by the audio manager. Exemplarily, the audio manager includes an audio policy module (AudioPolicy). The audio application can call the audio policy module to set the specific audio playback strategy when the audio data is input and output at runtime. For example, the volume, the sampling rate of the audio data, the sound effect setting, the mixing setting, etc. The audio policy module can output the corresponding audio playback strategy to the audio processor in real time during the audio playback process, so that the audio processor can perform mixing, resampling, sound effect setting, etc. on the audio data output by the audio player according to the latest audio playback strategy, and the processed audio data is the audio data to be played.

[0113] The audio service can also receive audio collection instructions from audio applications, such as recording applications, and send them to the audio access component.

[0114] The audio access component can send the above-mentioned audio data to be played to the hardware interface that can realize the audio playback function according to the configuration instructions issued by the audio service. The audio access component can also send the above-mentioned audio acquisition instructions to the hardware interface that can realize the audio acquisition function according to the configuration instructions issued by the audio service. For example, the AudioPolicyImpl in the audio access component can select an audio device according to the configuration instructions issued by the audio service. The AudioFlinger in the audio access can send the audio data or audio acquisition instructions issued by the audio service to the hardware interface corresponding to the selected audio device. AudioFlinger can also receive the audio data collected by the above-mentioned audio device from the hardware interface corresponding to the audio device.

[0115] For camera applications, the Android system is equipped with a Camera service and a Camera access component to implement the camera function for the camera application. When the camera application takes an image or video, the Camera service can be called to generate a call instruction based on the camera application's requirements for image format information. The requirements for image format information may include information such as the resolution, frame rate, and color space of the image or video.

[0116] The Camera access component can send the above call instructions to the hardware interface that can realize the image or video acquisition function according to the configuration instructions issued by the Camera service.

[0117] The service framework also includes a display service and a screen access component. The display service can obtain drawing instructions for the content displayed on the screen, draw and generate corresponding display content. The screen access component can encode the above display content, and send the encoded data to the transmission module to transmit it to other devices to realize the screen projection function between devices. The screen access component also includes a switching identification component, which is used to receive device identification information in operation messages from other devices, and identify the conditions for satisfying the use of device switching. When the conditions for device switching are met, a device switching instruction is generated.

[0118] The application framework layer may also include a distributed device management service, which can obtain and store the identification information of other devices and the multimedia device resource information of other devices when or after the mobile phone and other devices establish a network connection. The distributed device management service can also create a corresponding virtual multimedia device on the mobile phone based on the acquired multimedia device resource information of other devices, and establish a mapping relationship between the above virtual multimedia device and the identification information of the device corresponding to the above virtual multimedia device. The distributed device management service also stores the mapping relationship between the mobile phone's own identification information and the mobile phone's own multimedia device.

[0119] The distributed device management service includes an intelligent switching module, which can respond to the device switching instruction generated by the switching identification component in the screen access component and generate a device switching scheduling instruction for calling the relevant multimedia device according to the above correspondence.

[0120] Still Figure 3 As shown, HAL (hardware abstraction layer) can also be included between the application framework layer and the kernel layer of the Android system. HAL is responsible for interacting with various devices of the mobile phone. On the one hand, HAL hides the implementation details of each device, and on the other hand, it can provide the Android system with an interface for calling each device. HAL provides HALs corresponding to different mobile phone devices, such as Audio HAL, Camera HAL, Wi-Fi HAL, etc.

[0121] Among them, the hardware abstraction layer also includes a virtual HAL. The virtual HAL is a virtual HAL associated with the multimedia devices of other devices established by the distributed device management service in the hardware abstraction layer of the mobile phone when the mobile phone and other devices establish a network connection relationship or after the network connection relationship is established. Unlike the traditional HAL, the virtual HAL does not correspond to the actual device of the mobile phone. The virtual HAL can transmit the received data or instructions to other devices through the established network connection. At the same time, the virtual HAL can also accept data or instructions from other devices. For example, the virtual Audio HAL can send audio device call instructions to other devices, the virtual Audio HAL can also send audio data to be played to other devices, and the virtual Audio HAL can also accept recording data from other devices; the virtual Camera HAL can send camera device call instructions to other devices, and the virtual Camera HAL can also accept pictures or video data collected from other devices. It can be understood that the above-mentioned virtual Audio HAL and virtual Camera HAL can be collectively referred to as virtual multimedia HAL.

[0122] In addition, the application framework layer may also include a window manager, a content provider, a view system, a resource manager, a notification manager, etc., and the embodiments of the present application do not impose any restrictions on this.

[0123] For example, the above-mentioned window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The above-mentioned content provider is used to store and obtain data, and make these data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The above-mentioned view system can be used to build the display interface of the application. Each display interface can be composed of one or more controls. Generally speaking, controls can include interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. The above-mentioned resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The above-mentioned notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, displaying text messages in the status bar, making beep sounds, vibrating, or flashing indicator lights, etc.

[0124] like Figure 3As shown in Figure 1, Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0125] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0126] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0127] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0128] Among them, the surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple commonly used audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0129] The kernel layer is located below the HAL and is a layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, a sensor driver, etc., and the present application embodiment does not impose any restrictions on this.

[0130] Next, refer to Figure 4 , a device use method provided in an embodiment of the present application is specifically introduced. The method can be executed by any electronic device with a display screen and data processing capabilities. Figure 4As shown, the method includes: step 401, the screen projection source device runs the first application and displays the interface of the first application on its screen, the screen projection source device projects the content of the first application to the screen projection destination device, and the screen projection destination device displays a screen projection window on its display screen, and the screen projection window displays the interface of the first application; step 402, the first application running on the screen projection source device uses the first multimedia device of the screen projection source device itself to process multimedia information; step 403, the screen projection destination device generates a first operation message in response to the first operation on the screen projection window and sends it to the screen projection source device; step 404, the screen projection source device receives the first operation message, and in response to the first operation message, the first application of the screen projection source device switches to use the second multimedia device of the screen projection destination device to process multimedia information; step 405, the screen projection source device detects the second operation on the screen projection source device; step 406, in response to the second operation, the first application of the screen projection source device resumes using the first multimedia device of the screen projection source device itself to process multimedia information. It should be understood that the screen projection destination device and the screen projection source device are both electronic devices.

[0131] Next, the method for using the device provided in the embodiment of the present application is introduced with reference to specific examples.

[0132] First, in step 401, the projection source device runs the first application and displays the interface of the first application on its screen. The projection source device projects the content of the first application to the projection destination device. The projection destination device displays a projection window on its display screen, and the projection window displays the interface of the first application.

[0133] Take computers and mobile phones as examples. When a computer and a mobile phone are connected to a network through near-field communication technology (such as Bluetooth, Wi-Fi, peer-to-peer (P2P), ZigBee, etc.), using screen projection technology, the mobile phone can be used as a projection source device to project the application content it is running onto the computer's display screen. At this time, the computer is the projection destination device; or the computer can be used as a projection source device to project the application content it is running onto the mobile phone's display screen. At this time, the mobile phone is the projection destination device. At this time, the connection state between the mobile phone and the computer can be called the projection connection state.

[0134] Specifically, you can combine Figure 5, describing the method by which the screen projection source device projects the application content it is running to the screen projection destination device. The display service of the screen projection source device obtains the surface from the screen access component, and draws the display area content on its screen on the surface; after the drawing is completed, the final display area content data is synthesized on the surface; the synthesized display area content data is sent to the encoder module in the screen access component for encoding to obtain the encoded data; the encoded data is sent to the screen projection destination device through the transmission module; the screen projection destination device decodes the received data to obtain the decoded surface data; the decoded data is sent to the display service of the screen projection destination device, and the display service synthesizes the decoded surface and the surface of the screen projection destination device for display on the display screen of the screen projection destination device. It should be noted that the embodiments of the present application do not limit the method by which the screen projection source device projects the content of the application it is running to the screen projection destination device.

[0135] Take the projection of a mobile phone to a computer as an example. When a mobile phone projects the application content running on the mobile phone to the computer, the computer can synchronously display the application content running on the mobile phone in the projection window. The computer can display the projection window on its entire display interface, that is, the projection window occupies the entire display interface of the computer. The computer can also display the projection window on part of its display interface, that is, the projection window only displays part of the computer's interface content. The computer can also scale the size of the projection window according to the user's operation, and display a projection window of a size different from the mobile phone screen on the computer's display. When projecting the screen, the mobile phone can still display the same content as the projection window on the computer, that is, mirror projection; it can also minimize the content displayed in the projection window on the computer on the mobile phone, that is, transfer projection. When projecting the screen, the mobile phone can also be in the screen-off state, and the projection window on the computer still displays the application content running on the mobile phone.

[0136] It should be understood that the screen projection window can be run in the form of an application on the screen projection destination device. For example, the screen projection destination device responds to the projection of the screen projection source device and creates a screen projection shell application in the screen projection destination application layer. The screen projection shell application runs so that the screen projection destination device displays the screen projection window.

[0137] In some embodiments, the screen projection source device and the screen projection destination device establish a screen projection connection, and the distributed device management service of the screen projection source device can obtain the identification information of the screen projection destination device and the multimedia device resource information and store it. Furthermore, the distributed device management service can also create a corresponding virtual multimedia HAL on the mobile phone based on the multimedia device resource information of the screen projection destination device obtained, and establish a mapping relationship between the above-mentioned virtual multimedia HAL and the identification information of the device corresponding to the above-mentioned virtual multimedia HAL. The above-mentioned virtual multimedia HAL can be used to call the multimedia device on its corresponding screen projection destination device. For example, after the screen projection source device and the screen projection destination device establish a connection relationship, a virtual Audio HAL and a virtual Camera HAL can be established in the HAL layer. The screen projection source device can call the audio device (such as a microphone, a speaker) of the screen projection destination device through the virtual Audio HAL, and the screen projection source device can call the camera device (such as a camera) of the screen projection destination device through the virtual Camera HAL. It should be understood that the screen projection source device can also establish connections with multiple screen projection destination devices. The distributed device management service can create corresponding multiple virtual multimedia HALs for multiple screen projection destination devices, and save the mapping relationship between multiple virtual multimedia HALs and multiple screen projection destination device identification information.

[0138] In some embodiments, when the first application is running on the screen projection source device, the screen projection source device can display an interface corresponding to the first application on the screen. Correspondingly, the screen projection window of the screen projection destination device will also display the interface corresponding to the first application.

[0139] In one implementation, Fig. 6AAs shown, the first application running may be a video call application, and the relevant interface of the video call application is displayed on the screen of the screen projection source device 602 and in the screen projection window 610 of the screen projection destination device 601. In this interface, the screen projection window 610 of the screen projection destination device 601 may include a system status bar 641, video display content 642, video call control button 643, etc. The system status bar 641 may include a battery level indicator, a time indicator (e.g., 8:00 shown in the figure), and a signal strength indicator of user wireless communication (e.g., 5G). It should be noted that, in some embodiments, the content displayed in the status bar is not limited to this, for example, the status bar may also display a Bluetooth indicator, an alarm clock indicator, etc. It should be understood that this embodiment does not limit the arrangement position of the content displayed in the status bar. For example, in some embodiments, the position of the time indicator may be displayed in the center. The video call display content 642 may include real-time images of both parties in the video call, and the above-mentioned real-time images may include real-time images of the call end and real-time images of the call end. The video call control button 643 may include a call connect / end button, a front and rear camera switch button, a call setting button, etc. In some embodiments, the video call interface may include more or fewer elements, and this application does not make specific limitations.

[0140] In another implementation, Figure 6B As shown, the first application running can also be an audio and video application, and the relevant interface of the audio and video application is displayed on the screen of the screen projection source device 602 and in the screen projection window 610 of the screen projection destination device 601. In this interface, the screen projection window 610 of the screen projection destination device 601 may include a system status bar 621, audio and video playback content 622, audio and video playback control options 623, audio and video playback function options 624, etc. Among them, the description of the system status bar 621 is detailed in Fig. 6A The relevant description in the description will not be repeated here. The audio and video playback content 622 can display the audio and video content currently being played. The audio and video playback options 623 include buttons for controlling the playback of audio and video, such as the play button, pause button, next song button, previous song button, etc. The audio and video playback function options 624 may include buttons for changing the currently playing content (such as selecting an album or a song), as well as some additional function buttons or display content (such as comments). In some embodiments, the audio and video playback interface may include more or fewer elements, and this application does not make specific restrictions.

[0141] In another implementation, Figure 6CAs shown, the first application running can also be a camera application, and the relevant interface of the camera application is displayed on the screen of the screen projection source device 602 and in the screen projection window 610 of the screen projection destination device 601. In this interface, the screen projection window 610 of the screen projection destination device 601 may include a photo image display content 631, a photo button 632, a photo option 633, etc. Among them, the photo image display content 631 displays the image acquired by the camera through the camera in real time, and the above image can be an image acquired by the front camera, the rear camera or other cameras. The photo button 632 may include a photo shutter button, a photo preview button, and a front and rear camera switch button. The photo option 633 may include the type setting of the current photo (for example, aperture, night scene, portrait, photo, video, professional, etc.), and the photo option 633 may also include specific options for setting the photo, such as exposure setting, flash setting, AI (Artificial Intelligence) photo auxiliary setting, filter setting, etc. In some embodiments, the photo interface may include more or fewer elements, and this application does not make specific restrictions.

[0142] In another implementation, the projection source device runs a first application and a second application, the first application runs in the background, the screen of the projection source device displays the interface of the second application, and the corresponding projection window of the projection destination device displays the interface of the second application. For example, the first application can be a music application that plays in the background, and the second application is a desktop application. The screen of the projection source device displays the interface of the desktop application, and the corresponding projection window of the projection destination device displays the interface of the desktop application. Fig.6D As shown, the screen projection window 610 of the screen projection destination device 601 may include elements such as a system status bar 651 (status bars), a dock bar 652 (favorites tray), and application icons (icons) 653. In some embodiments, the screen projection window 610 of the screen projection destination device 601 may also include elements such as application widgets not shown in the figure. In some embodiments, the main interface may include more or fewer elements, and this application does not make specific restrictions. Among them, the system status bar 651 refers to Fig. 6A. The dock bar 652 may include application icons such as a camera, address book, phone, and information. In some embodiments, the user can manually change the displayed application icon. The desktop page mark is used to display the number of pages contained in the desktop and the location of the current page. The application icon may include at least one application icon. The application icon 653 may include at least one of the applications of "Clock", "Calendar", "Photos", "Memorandum", "File Management", "E-mail", "Music", "Calculator", "HUAWEI Video", "Health", "Weather", "Browser", "AI Life", "Settings", "Recorder", "App Store", "Camera", "Contacts", "Phone" and "Message" or a combination thereof. In some embodiments, the application icons displayed by the application icons may be different. For example, the number of application icons may be increased or decreased, and the categories of the icons may also be different.

[0143] Next, in step 402, the first application running on the screen projection source device uses the first multimedia device of the screen projection source device itself to process multimedia information.

[0144] Among them, the first multimedia device of the screen projection source device itself may include an audio device, such as a speaker, microphone, etc. of the screen projection source device itself; the first multimedia device of the screen projection source device itself may also include a camera device, such as a camera of the screen projection source device itself. Using multimedia devices to process multimedia information may include using audio devices to collect audio information, or using audio devices to play audio information, or using audio devices to collect audio information and play audio information at the same time. The above-mentioned use of multimedia devices to process multimedia information may also include using camera devices to collect image information, or using camera devices to collect video information.

[0145] Optionally, the first multimedia device using the screen projection source device also includes using a device that establishes a connection with the screen projection source device, such as audio devices such as headphones and speakers that establish a wireless or wired connection with the screen projection source device, or imaging devices such as cameras or camcorders that establish a wireless or wired connection with the screen projection source device.

[0146] In one implementation, the first application running on the screen projection source device calls the first audio device and the first camera device of the screen projection source device to collect audio information, play audio information, and collect video information. Fig. 6A In the scenario, the first application is a video call application. At this time, the module diagram of the device call of the screen projection source device is as follows Fig. 7A As shown. The video call application 713 sends a service dispatch to the audio service 721 and the camera service 723 indicating the call of the audio capability and the camera capability; the audio service 721 and the camera service 723 respond to the dispatch from the video call application 714 and configure the device selection of the audio access component 722 and the camera access component 724 to use the local device according to the current device scheduling policy; the audio access component 722 and the camera access component 724 respond to the device selection configuration and choose to call the local audio HAL 731 and the local camera HAL 732. Subsequently, the audio access component 722 can send the audio stream to the local audio device (such as the speaker of the screen projection source device itself) through the local audio HAL 731 for playback, or receive the audio stream collected from the local audio device (such as the microphone of the screen projection source device itself) through the local audio HAL 731; correspondingly, the camera access component 724 can receive the video stream collected from the local camera device (such as the camera of the screen projection source device itself) through the local camera HAL 732.

[0147] In another implementation, the first application on the screen projection source device calls the audio device of the screen projection source device to play audio information, or to collect audio information, or to play audio information and collect audio information at the same time. Figure 6B In the scenario, the first application is a video application. In this way, the module diagram of the device call of the screen projection source device is as follows: Figure 7BAs shown. The video application 711 of the screen projection source device sends the service schedule indicating the use of audio capabilities to the audio service 721; the audio service 721 responds to the schedule from the video application and configures the audio access component 722 to use local devices according to the current device scheduling policy; the audio access component 722 selects to call the local Audio HAL 731 in response to the device selection configuration. Subsequently, the audio access component 722 can send the audio stream to the local audio device (such as the speaker of the screen projection source device itself) through the local Audio HAL731 to play the audio information. For another example, the above-mentioned first application can also be a recording application, and accordingly, the recording application can realize the audio information collection through the audio device (such as a microphone) of the screen projection source device itself through the above process. For another example, the above-mentioned first application can also be a call application, and accordingly, the call application can realize the audio information playback and audio information collection through the audio device (such as a speaker and a microphone) of the screen projection source device itself through the above process.

[0148] In another implementation, the first application on the screen projection source device calls the camera device of the screen projection source device to collect image information or video information. Figure 6C In the scenario, the first application is the camera application. In this way, the module diagram of the device call of the screen projection source device is as follows: Figure 7C As shown. The camera application 712 of the screen projection source device sends the service schedule indicating the use of the camera capability to the Camera service 723; the Camera service 723 responds to the schedule from the camera application and configures the device selection for the Camera access component 724 to use the local device according to the current device scheduling policy; the Camera access component 724 selects to call the local Camera HAL 732 in response to the device selection configuration. Subsequently, the Camera access component 724 can receive image information or video information collected from the local camera device (such as the camera of the screen projection source device itself) through the local Camera HAL 732.

[0149] In another implementation, the first application can be running in the background and still using the audio device of the projection source device. At this time, the screen of the projection source device displays the content of the second application, and the projection window of the projection destination device displays the content of the second application. For example, the music application in the background is still calling the audio device to play the audio in the background, while the screen of the projection source device and the projection window of the projection destination device display the interface of the desktop application. For another example, the recording application in the background is still calling the audio device to collect audio information in the background, while the screen of the projection source device and the projection window of the projection destination device display the interface of the desktop application. At this time, the working principle and Figure 6BThe principle of device calling in the same scenario is the same and will not be repeated here.

[0150] It should be noted that the order of step 401 and step 402 can be swapped.

[0151] Next, in step 403, the screen projection destination device generates a first operation message in response to the first operation on the screen projection window and sends it to the screen projection source device.

[0152] Among them, the first operation can specifically be a user's touch operation, air gesture operation, voice control operation, etc. on the projection window. When the projection source device is a laptop computer, it can also be a mouse click operation on the projection window. The above touch operations include single touch, double touch, triple touch, long press touch, sliding touch, two-finger touch, three-finger touch, etc. The above touch can be performed by the user's fingers, knuckles, or by using a stylus. The above air gesture operations include hovering gestures, approach gestures, distance gestures, sliding gestures, etc. The above mouse click operation can be a single click, double click, left click, right click, middle click, etc. The first operation can also be other operations on the projection window, which is not limited in this application.

[0153] Figure 8A-8C Different types of first operations on the projection window 810 are shown. The content displayed in the projection window takes the interface corresponding to the video call application as an example.

[0154] Fig. 8A The touch operation on the projection window 810 may be a touch operation within the projection window 810. For example, the touch operation may be a touch operation on the system status bar 811, the video call display content 812, the video call control button 813 and other areas in the projection window 810. The touch operation may also be an operation of sliding gestures on the projection window, such as sliding up and down to scroll the screen, sliding left and right to scroll the screen, etc. The touch operation may also be a gesture of sliding down to open a menu at the top of the projection window. The touch screen operation may also be a click operation on other areas 814 in the projection window 810.

[0155] Figure 8B It is an air gesture operation on the projection window, such as hovering the video call control button 813 in the projection window 810 to hang up the video call or switch the video call display interface.

[0156] Optional, such as Figure 8CAs shown, when the screen projection destination device 801 is a laptop computer, the first operation can also be a click operation performed by controlling the mouse in the screen projection window 810, or a click operation performed by operating the touchpad in the screen projection window. The click operation by the mouse can be a click operation on the system status bar 811, the video call display content 812, the video call control button 813 and other areas in the screen projection window 810. The click operation by the mouse can also be a click operation on other areas 814 in the screen projection window 810.

[0157] Optional, such as Fig.8D As shown, the first operation can also be a voice control operation on the screen projection target device. For example, taking the screen projection target device as a laptop, the user can first trigger the voice assistant of the laptop, and then the user can say voice control sentences such as "switch to use the laptop's device", "use the laptop to play sound", "use the laptop's camera" to the laptop.

[0158] In some embodiments, in the projection state, the first operation performed on the projection window on the projection destination device will trigger a corresponding event, and accordingly, a first operation message will be generated and sent to the projection source device. The above-mentioned first operation message can instruct the projection source device to complete the same operation and thus trigger the corresponding event. For example, the triggered event can be to open an application in the projection window on the projection destination device, and accordingly, on the projection source device, this application will also be opened. The triggered event can also be an event triggered by a gesture operation performed on the projection destination device, and accordingly, it will also be performed on the projection source device, such as a drop-down menu operation triggered in response to a user's sliding gesture on the upper end of the projection window. It can be understood that the present application does not specifically limit the triggered event.

[0159] In some embodiments, in the screen casting state, operations that do not trigger any events are performed on the screen casting window on the screen casting destination device (such as Fig. 8A A single touch operation performed on other areas of the projection window), similarly, a first operation message will be generated to notify the projection source device that a certain position of the current projection window is touched.

[0160] Furthermore, in response to the above-mentioned first operation on the projection window, the first operation message generated by the projection destination device also includes the identification information of the projection destination device. The above-mentioned identification information is similar to a device identification code, which can uniquely identify and distinguish each device in the network. For example, the identification information can be a device ID (Identification), a media access control (MAC) address, or a product serial number (SN) of the device.

[0161] Combine the following Fig. 9 , which introduces in detail the working principle of the screen projection destination device 901 responding to the first operation on the screen projection window, generating a first operation message and sending it to the screen projection source device.

[0162] First, the screen projection destination device 901 can obtain the input event of the first operation for the screen projection window through the HAL layer input device access node 910. For example, the screen projection destination device 901 can obtain the touch event obtained from the touch screen through the device access node "dev / input / eventX" 911, and can also obtain the click event obtained from the mouse through the device access node "dev / input / tsX" 912. The above-mentioned device access node 910 reports the above-mentioned first operation input event to the input management service (Input Manager Service, hereinafter referred to as IMS) module 921 located in the display service 920.

[0163] Specifically, the IMS module 921 may include an InputReader module and an InputDispatcher module. Among them, the InputReader can continuously read the input events of the operations reported by the device access node 910, translate and process them into Android input events, and distribute the events to the InputDispatcher module. The InputDispatcher module can hand over the events to the appropriate window for processing according to the window information provided by the Window Manager Service (hereinafter referred to as WMS) module 922. The window is then sent to the projection shell application 931 located in the application layer along the control tree according to ViewRootImpl. The projection shell application 931 is a projection application established by the projection source device when the projection is performed on the projection destination device 901.

[0164] Subsequently, after receiving the input event of the first operation, the above-mentioned screen projection shell application 931 reports the event to the distributed display management service 923 (Distributed Display Manager Service, hereinafter referred to as DDMS) located in the display service 920. The DDMS module 923 can complete the coordinate conversion of the input event according to the screen resolution information of the screen projection source device and the screen projection destination device 901. Subsequently, the DDMS module 923 generates a first operation message, which includes the input event that completes the coordinate conversion and the identification information of the screen projection destination device 901, and sends the first operation message to the codec transmission module 925 located in the screen access component 924. The above-mentioned codec transmission module 925 can encode the above-mentioned first operation message and send it to the screen projection source device through the established network connection.

[0165] Through the above process, the screen projection destination device can detect the user's first operation on the screen projection window, and generate a first operation message including the input event of the above first operation and the identification information of the screen projection destination device in response to the first operation, and then send the first operation message to the screen projection source device.

[0166] Next, in step 404, the screen projection source device receives a first operation message, and in response to the first operation message, the first application of the screen projection source device switches to use the second multimedia device of the screen projection destination device to process multimedia information.

[0167] The second multimedia device of the screen projection destination device may include an audio device, such as a speaker, a microphone, etc. of the screen projection destination device; the second multimedia device of the screen projection destination device may also include a camera device, such as a camera of the screen projection destination device, etc. For the introduction of using multimedia devices to process multimedia information, please refer to the relevant description in step 402, which will not be repeated here.

[0168] Optionally, the second multimedia device used in the screen projection destination device also includes a device used to establish a connection with the screen projection destination device, such as audio devices such as headphones and speakers that establish a wireless or wired connection with the screen projection destination device, or video devices such as cameras or camcorders that establish a wireless or wired connection with the screen projection source device.

[0169] It should be understood that the switching of the first application of the screen projection source device to the second multimedia device of the screen projection destination device for multimedia information processing refers to stopping or pausing the use of the first multimedia device of the screen projection source device for multimedia information processing, thereby using the second multimedia device of the screen projection destination device for multimedia information processing.

[0170] In one implementation, the screen projection source device receives a first operation message, and in response to the first operation message, the first application of the screen projection source device switches to call the audio device and the camera device of the screen projection destination device to collect audio information, play audio information, and collect video information. Fig. 6A In the scenario, the first application is a video call application. In response to the first operation message, the video call application stops calling the audio device of the screen projection source device (such as the microphone and speaker of the screen projection source device itself) to collect audio information and play audio information, and stops calling the camera device of the screen projection source device (such as the camera of the screen projection source device itself) to collect video information. At the same time, in response to the first operation message, the video call application starts calling the audio device of the screen projection destination device (such as the microphone and speaker of the screen projection destination device) to collect audio information and play audio information, and calls the camera device of the screen projection destination device (such as the camera of the screen projection destination device) to collect video information.

[0171] In another implementation, the screen projection source device receives a first operation message, and in response to the first operation message, the first application of the screen projection source device switches to call the audio device of the screen projection destination device to play audio information, or to collect audio information, or to play audio information and collect audio information at the same time. Figure 6B In the scenario, the first application is an audio and video application. In response to the first operation message, the audio and video application stops calling the audio device of the screen projection source device (such as the speaker of the screen projection source device itself) to play audio information; at the same time, in response to the first operation message, the audio and video application starts calling the audio device of the screen projection destination device (such as the speaker of the screen projection destination device) to play audio information. Optionally, when the first application is a recording application, in response to the first operation message, the recording application stops calling the audio device of the screen projection source device (such as the microphone of the screen projection source device itself) to collect audio information; at the same time, in response to the first operation message, the recording application starts calling the audio device of the screen projection destination device (such as the speaker of the screen projection destination device) to play audio information. Optionally, when the first application is a call application, in response to the first operation message, the call application stops calling the audio device of the screen projection source device (such as the microphone and speaker of the screen projection source device itself) to collect and play audio information; at the same time, in response to the first operation message, the call application starts calling the audio device of the screen projection destination device (such as the microphone and speaker of the screen projection destination device) to collect and play audio information.

[0172] In another implementation, the screen projection source device receives a first operation message, and in response to the first operation message, the first application of the screen projection source device switches to call the camera device of the screen projection destination device to collect video information or image information. Figure 6C In the scenario, the first application is a camera application. In response to the first operation message, the camera application stops calling the camera device of the screen projection source device (such as the camera of the screen projection source device itself) to collect video information or image information. At the same time, in response to the first operation message, the camera application starts calling the camera device of the screen projection destination device (such as the camera of the screen projection destination device) to collect video information or image information.

[0173] In another implementation, the screen projection source device receives a first operation message, and in response to the first operation message, a first application running in the background of the screen projection source device calls an audio device of the screen projection destination device to play or collect audio information. Fig.6D In the scenario, the first application is a music application running in the background. In response to the first operation message, the music application stops calling the audio device of the screen projection source device (such as the speaker of the screen projection source device itself) to play audio information; at the same time, in response to the first operation message, the audio and video application starts calling the audio device of the screen projection destination device (such as the speaker of the screen projection destination device) to play audio information. Optionally, the first application can be a recording application running in the background. In response to the first operation message, the recording application stops calling the audio device of the screen projection source device (such as the microphone of the screen projection source device itself) to collect audio information; at the same time, in response to the first operation message, the recording application starts calling the audio device of the screen projection destination device (such as the microphone of the screen projection destination device) to collect audio information.

[0174] In another implementation, the first application running on the screen projection source device can call at least one of the audio device and the camera device of the screen projection destination device according to the device capabilities of the screen projection destination device. For example, when the screen projection destination device is a device with only an audio device but no camera device, in the following example, Figure 6C In the video call scenario, the first application of the screen projection source device only starts to call the audio device of the screen projection destination device, and continues to call the camera device of the screen projection source device. For example, when the screen projection destination device is a device with only a camera device but no audio device, it is still Figure 6C In the scenario, the first application of the screen projection source device starts to call the camera device of the screen projection destination device, while continuing to call the audio device of the screen projection source device itself.

[0175] In another implementation, the first application of the screen projection source device can call at least one of the audio device and the camera device of the screen projection destination device according to the settings. The screen projection source device can receive a selection of the device switching function from the user, and based on the user's selection, call the device that the user wants to call on the screen projection destination device. For example, when the user uses a mobile phone to project the screen on a laptop, he chooses to switch to only the camera device of the laptop. At this time, if the user operates the screen projection window of the laptop and triggers the intelligent switching of the device, the application on the mobile phone will only switch to the camera device of the laptop, and continue to use the audio device of the mobile phone. For another example, when the user uses a mobile phone to project the screen on a tablet computer, he chooses to switch to only the audio device of the tablet computer. At this time, if the user operates the screen projection window of the tablet computer and triggers the intelligent switching of the device, the application on the mobile phone will only switch to the audio device of the tablet computer, and continue to use the camera device of the mobile phone.

[0176] Specifically, Fig.10 As shown, when a user uses a screen projection source device to project the screen on a screen projection destination device for the first time, such as when a user uses a mobile phone to project the screen on a laptop computer for the first time, a setting window will pop up, prompting the user to set the device smart switching function when projecting the screen on the laptop computer. The user can choose to turn on or off the smart switching function of audio and video devices, and save the setting. When the screen projection source device that has been set responds to the operation message of the screen projection destination device, it will selectively switch to use at least one of the audio device and camera device of the screen projection destination device according to the user's settings.

[0177] like Fig.11 As shown, the user can also set the intelligent switching of devices in the system settings of the screen projection source device. The setting list includes the screen projection destination device that has established a screen projection relationship with the screen projection source device and the corresponding on / off button for the transfer of audio and camera devices for different screen projection destination devices. The user can choose to set the device switching function of audio and camera when projecting on different screen projection destination devices to turn on or off, and save the setting. When responding to the operation message of the screen projection destination device, the screen projection source device will selectively switch to use at least one of the audio device and camera device of the screen projection destination device according to the user's settings.

[0178] The above method can realize personalized settings for intelligent switching of devices when the user uses the screen projection source device to project the screen on different screen projection target devices. That is, the screen projection source device can switch to use at least one of the audio device and camera device of the screen projection target device according to the user's settings, so that the user can get a customized screen projection experience.

[0179] In some embodiments, the screen projection source device can generate a first switching instruction in response to the identification information of the screen projection destination device in the received first operation message, and call the first multimedia abstract module according to the first switching instruction. The above-mentioned first multimedia abstract module is used to call the second multimedia device of the screen projection destination device.

[0180] Combine the following Fig.12 and Figure 13A-13C The working principle of the screen projection source device receiving the first operation message and the first application of the screen projection source device calling the second multimedia device of the screen projection destination device in response to the first operation message is introduced in detail.

[0181] First, according to Fig.12 The present invention introduces the working principle that a screen projection source device receives a first operation message, and in response to the first operation message, the screen projection source device generates a first switching instruction.

[0182] First, the codec transmission module 1211 located in the screen access component 1210 in the screen projection source device 1201 receives a first operation message sent from the screen projection destination device, which includes a first operation input event and identification information of the screen projection destination device, and decodes the first operation message to obtain the first operation input event and identification information of the screen projection destination device.

[0183] Subsequently, the codec transmission module 1211 sends the first operation input event to the IMS module of the display service 1220 of the screen projection source device 1201, and the IMS module distributes the event to the WMS module, and the WMS module reports the event to the application corresponding to the input event at the application layer. Through the above processing of the first operation input event, the user's first operation on the screen projection destination device can be synchronized to the screen projection source device, and the screen projection destination device can realize the synchronous control operation of the screen projection source device.

[0184] Furthermore, the codec transmission module 1211 also sends the decoded identification information of the screen projection destination device to the switching identification component 1212. The switching identification component 1212 saves a record of the identification information received last time. Exemplarily, the record of the identification information received last time can be stored in the screen projection source device 1201 in the form of a list, as shown in Table 1. The above identification information takes the device ID as an example.

[0185] Table 1

[0186] The last received identification information 2A11682E2F54F…(screen projection source device ID)

[0187] As shown in Table 1, before the switching identification component 1212 receives the identification information of the screen projection destination device sent by the codec transmission module 1211, the device corresponding to the last received identification information is the screen projection source device 1201 itself.

[0188] Furthermore, after the switching identification component 1212 receives the identification information of the screen projection destination device 901 sent by the codec transmission module 1211, the switching identification component 1212 compares the currently received identification information with the identification information received last in the record. If the currently received identification information is the same as the identification information received last in the record, the record is not updated; if the currently received identification information is not the same as the identification information received last in the record, the currently received identification information is updated in the record as the identification information received last, and at the same time, the switching identification component sends a first switching instruction to the intelligent switching module 1231 located in the distributed device management service, which includes the device identification information received last at the moment.

[0189] Table 2

[0190] The last received identification information 1A21412E2F54F…(Destination device ID)

[0191] As shown in Table 2, after the switching identification component 1212 determines that the identification information received from the screen projection destination device is different from the identification information last received in the record, the identification information of the screen projection destination device is updated in the record.

[0192] Furthermore, the intelligent switching module 1231 located in the distributed device management service receives the first switching instruction from the switching identification component 1212. The correspondence between the device identification information and the multimedia device HAL is stored in the distributed device management service. Among them, the above-mentioned devices include the screen projection source device 1201 itself and other devices connected to the screen projection source device 1201, such as the screen projection destination device. When the screen projection source device 1201 and the screen projection destination device establish a connection, the distributed device management service of the screen projection source device 1201 will create a corresponding virtual multimedia device HAL in the screen projection source device 1201 according to the multimedia device capabilities of the screen projection destination device, such as virtual Audio1 HAL and virtual Camera1 HAL. At the same time, the distributed device management service of the screen projection source device 1201 will also update the mapping relationship between the identification information of the screen projection destination device and the virtual multimedia device HAL to the corresponding relationship between the above-mentioned device identification information and the multimedia device HAL.

[0193] Exemplarily, the correspondence between the device identification information and the multimedia device HAL can be stored in the screen projection source device 1201 in the form of a list, as shown in Table 3.

[0194] Table 3

[0195] Device identification information Multimedia Device HAL 2A11682E2F54F…(screen projection source device ID) Local Audio HAL 2A11682E2F54F…(screen projection source device ID) Native Camera HAL 1A21412E2F54F…(Destination device ID) Virtual Audio1 HAL 1A21412E2F54F…(Destination device ID) Virtual Camera1 HAL … …

[0196] Exemplarily, the correspondence between the identification information of the screen projection source device 1201 and the screen projection destination device and the multimedia device HAL is recorded in Table 3. Among them, the multimedia device HAL corresponding to the identification information of the screen projection source device 1201 includes the local Audio HAL and the local Camera HAL; the multimedia device HAL corresponding to the identification information of the screen projection destination device includes the virtual Audio1HAL and the virtual Camera1 HAL.

[0197] Further, in response to the first switching instruction sent from the switching identification component 1212, the intelligent switching module 1231 extracts the virtual Audio1 HAL and the virtual Camera1 HAL corresponding to the identification information of the screen projection destination device from the correspondence between the device identification information and the multimedia device HAL stored in the distributed device management service according to the identification information of the screen projection destination device contained in the above-mentioned device switching instruction, generates a device switching scheduling instruction for indicating the device capabilities of calling the virtual Audio1 HAL and the virtual Camera1 HAL, and sends the device switching scheduling instruction to the audio service module 1241 and the Camera service module 1243.

[0198] Combine the following Figure 13A-13C The working principle of a first application on a screen projection source device calling a second multimedia device on a screen projection destination device in response to a first switching instruction is introduced.

[0199] In response to the first switching instruction, the intelligent switching module 1231 sends a device switching scheduling instruction to the audio service module 1241 and the Camera service module 1243 to indicate the device capability of calling the virtual multimedia HAL associated with the screen projection destination device.

[0200] In one implementation, Fig.13AAs shown, the first application is a video call application, and the video call application 1301 sends a service dispatch requesting audio capability and camera capability to the audio service module 1241 and the Camera service module 1243. In response to the service dispatch, the audio service module 1241 determines that the current video call application 1301 has a demand for audio service. The audio service module 1241 also responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the audio device of the screen projection destination device by calling the virtual Audio1HAL1252, and performs device selection configuration on the audio access component 1242. The above configuration is used to instruct the audio access component 1242 to call the virtual Audio1 HAL 1252. After completing the above device selection configuration, the audio access component 1242 stops sending the audio stream to the local Audio HAL 1251 located in the HAL layer and stops receiving the audio stream from the local Audio HAL 1251; the audio access component 1242 starts sending the audio stream to the virtual Audio1 HAL 1252 located in the HAL layer and receives the audio stream from the virtual Audio1 HAL 1252. The above virtual Audio1 HAL1252 can send the audio stream to the audio device of the screen projection destination device, and can also receive the audio stream collected by the audio device of the screen projection destination device. Correspondingly, the Camera service module 1243 determines in response to the service scheduling that the current video call application 1301 has a demand for camera services. The Camera service module 1243 simultaneously responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the camera device of the screen projection destination device by calling the virtual Camera1 HAL 1254, and performs device selection configuration on the camera access component 1244, and the above configuration is used to instruct the camera access component 1244 to call the virtual Camera1HAL 1254. After completing the above device selection configuration, the Camera access component 1244 stops receiving the video stream from the local Camera HAL1253; the Camera access component 1244 starts to receive the video stream from the virtual Audio1 HAL 1254. The above virtual Camera1 HAL 1254 can receive the video stream collected by the camera device of the screen projection destination. In some embodiments, the audio access component 1242 and the Camera access component 1244 can also send relevant control instructions or configuration instructions to the virtual Audio1 HAL 1252 and the virtual Camera1 HAL 1254 to control or configure the audio device or camera device of the screen projection destination device.

[0201] In another implementation, Fig. 13BAs shown, the first application is a video application, and the video application 1302 sends a service dispatch requesting audio capabilities to the audio service module 1241. In response to the service dispatch, the audio service module 1241 determines that the current video application 1302 has a demand for audio services. At the same time, in response to the device switching scheduling instruction, the audio service module 1241 determines that the current device scheduling strategy is to use the audio device of the screen projection destination device by calling the virtual Audio1 HAL 1252, and performs device selection configuration on the audio access component 1242, and the above configuration is used to instruct the audio access component 1242 to call the virtual Audio1HAL 1252. After completing the above device selection configuration, the audio access component 1242 stops sending the audio stream to the local Audio HAL 1251 located in the HAL layer; the audio access component 1242 starts sending the audio stream to the virtual Audio1 HAL 1252 located in the HAL layer. The above virtual Audio1 HAL 1252 can send the audio stream to the audio device of the screen projection destination device. Although the Camera service module 1243 receives the device switching scheduling instruction, it does not receive the service scheduling issued by the application, so no camera device is being called. In some embodiments, the audio access component 1242 can also send relevant control instructions or configuration instructions to the virtual Audio1 HAL 1252 to control or configure the audio device of the screen projection destination device.

[0202] In another implementation, Fig. 13CAs shown, the first application is a camera application, and the camera application 1303 sends a service schedule to the Camera service module 1243. In response to the service schedule, the Camera service module 1243 determines that the current camera application 1303 has a demand for camera services. The Camera service module 1243 also responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the camera device of the screen projection destination device by calling the virtual Camera1 HAL 1254, and performs device selection configuration on the camera access component 1244, and the above configuration is used to instruct the camera access component 1244 to call the virtual Camera1 HAL1254. After completing the above device selection configuration, the Camera access component 1244 stops receiving images or videos from the local Camera HAL 1253; the Camera access component 1244 starts to receive images or videos from the virtual Audio HAL1254. The above virtual Camera1 HAL 1254 can receive images or videos captured by the camera device of the screen projection destination. Although the audio service module 1241 has received the device switching scheduling instruction, it has not received the service scheduling issued by the application, so no audio device is being called. In some embodiments, the Camera access component 1244 can also send relevant control instructions or configuration instructions to the virtual Camera1 HAL 1254 to control or configure the camera device of the screen projection destination device.

[0203] Table 4

[0204] Device identification information Multimedia Device HAL 2A11682E2F54F…(screen projection source device ID) Local Audio HAL 2A11682E2F54F…(screen projection source device ID) Native Camera HAL 1A21412E2F54F…(Destination device ID) Virtual Audio1 HAL … …

[0205] In another implementation, Fig.14 As shown and described in Table 4, when the multimedia device capabilities of the screen projection destination device 901 are limited, the intelligent switching module 1231 can send device switching scheduling instructions only to the audio service module 1241 or the Camera service module 1243 based on the correspondence between the above-mentioned device identification information and the multimedia device HAL. For example, the screen projection destination device is a TV device that only has an audio device. When the TV device and the screen projection source device 1201 establish a network connection, the screen projection source device 1201 only establishes a virtual Audio1HAL 1252 in the hardware abstraction layer of this end based on the multimedia device capabilities of the TV device, and then only saves the correspondence between the identification information of the TV device and the virtual Audio1 HAL in the above-mentioned correspondence. Therefore, the intelligent switching module 1231 only sends device switching scheduling instructions to the audio service module 1241. As shown in Fig. 6A In the scenario, the video call application calls the audio device and camera device of the screen projection source device at the same time. If the screen projection destination device is a TV device, such as Fig.14As shown, the audio service module 1241 determines the current device scheduling strategy in response to the service scheduling and device switching scheduling instructions of the video call application, which is to use the audio device of the screen projection destination device by calling the virtual Audio1 HAL 1252, and calls the virtual Audio1 HAL 1252 through the audio access component 1242, thereby using the audio device of the screen projection destination device to play and collect audio information; the Camera service module 1243 responds to the scheduling of the video call application and still calls the local Camera HAL 1253 of the screen projection source device 1201 through the Camera access component 1244, thereby using the camera device of the screen projection source device to collect video information.

[0206] In another implementation, Fig.15 As shown, after receiving the device switching scheduling instruction for indicating the device capability of using the virtual HAL, the audio service module 1241 and the camera service module 1243 call at least one of the virtual Audio1 HAL 1252 and the virtual Camera1 HAL 1254 through the audio access component 1242 and the camera access component 1244 according to the device switching function configuration, so as to use at least one of the audio device and the camera device corresponding to the screen projection target device. Still taking the screen projection target as a TV as an example, as Fig.11 In the settings, when the screen projection destination device 1201 is a TV, it is set to enable the audio device that allows the TV to be switched, and disable the camera device that allows the TV to be switched. Fig. 6A In the scenario, after the audio service module 1241 receives the device switching scheduling instruction from the smart switching module 1231, according to the setting of allowing audio switching in the device switching setting to be turned on, the audio service module 1241 stops calling the local Audio HAL 1251 through the audio access component 1242, and starts calling the virtual Audio1HAL 1252 through the audio access component 1242, thereby using the audio device of the screen projection destination device to play and collect audio information; after the Camera service module 1243 receives the device switching scheduling instruction from the smart switching module 1231, according to the setting of allowing camera switching in the device switching setting to be turned off, the Camera service module 1243 still calls the local Camera HAL 1253 through the Camera access component 1244, so the camera device of the screen projection source device 1201 is still used to collect video information.

[0207] Figure 16A-16CThe working principle diagram of the screen projection source device 1201 calling the screen projection destination device 901 device after the device switching is completed is shown. The video call application 1601 on the screen projection source device 1201 continuously sends instructions or data streams to the audio service module 1241 and the Camera service module 1243. The audio service module 1241 and the Camera service module 1243 respond to the instructions or data streams and call the virtual Audio1HAL 1252 and the virtual Camera1 HAL 1254 through the audio access component 1242 and the Camera component 1244. Subsequently, the virtual Audio1 HAL 1252 and the virtual Camera1 HAL 1254 can send the instructions or data streams to the screen projection destination device 901 through the network connection, and the screen projection destination device 901 receives and processes the instructions and data streams from the screen projection source device 1201. The screen projection source device 1201 can also receive the data stream returned from the screen projection destination device 901 through the virtual Audio1 HAL 1252 and the virtual Camera1 HAL 1254.

[0208] Specifically, Fig.16A As shown, the video call application 1601 can send the original audio data to the audio service module 1241, and the audio service module 1241 can parse, decapsulate or decode the original audio data to obtain frames of PCM data. The above data can also be further mixed, resampled, sound effects set, etc., and the processed audio data is the audio data to be played. The audio service module 1241 sends the above audio data to be played to the virtual Audio1 HAL 1252 through the audio access component 1242. The above virtual Audio1 HAL 1252 can transmit the received audio data to the screen projection destination device 901 through the established connection, and use the audio device of the screen projection destination device 901 to play the above audio data.

[0209] Specifically, Fig. 16BAs shown, the video call application 1601 can also issue an audio information collection instruction, and the audio service module 1241 responds to the audio information collection instruction and sends the audio information collection instruction to the virtual Audio1 HAL 1252 through the audio access component 1242. The virtual Audio1 HAL 1252 can transmit the received audio information collection instruction to the screen projection destination device 901 through the established connection, so that the screen projection destination device 901 uses its audio device to execute the audio information collection instruction, collect audio data, and send the audio data back to the screen projection source device 1201. The screen projection source device 1201 can receive the audio data collected by the screen projection destination device 901 through the virtual Audio1 HAL 1252 and send it back to the video call application for subsequent processing.

[0210] Specifically, Fig. 16C As shown, the video call application 1601 can send the video information acquisition instruction to the Camera service module 1243, and the above video information acquisition instruction can include video format information (including the number of data streams, the resolution of each data stream, frame rate, color space and other information). The Camera service module 1243 can send the above video information acquisition instruction to the virtual Camera1 HAL 1254 through the Camera access component 1244. Subsequently, the virtual Camera1 HAL1254 can transmit the above video information acquisition instruction to the screen projection destination device 901 through the established connection, and the screen projection destination device 901 configures its camera device according to the video format information contained in the above video information acquisition instruction and uses its camera device to collect video data. Optionally, when the above video call application has a request for multi-channel video stream acquisition, the virtual Camera1 HAL1254 can also select a video stream format with appropriate resolution and send it to the screen projection destination device 901 according to the bandwidth of the network to reduce bandwidth requirements and delays. The screen projection destination device 901 sends the video data collected by its camera device back to the screen projection source device 1201, and the virtual Camera1 HAL 1254 sends the video data back to the Camera service module 1243 through the Camera access component 1244, and the Camera service module 1243 forwards the video data to the video call application corresponding to the application layer. Optionally, the virtual Camera1 HAL 1254 can also convert the resolution and color space of the received video.

[0211] In another implementation, the video application may execute as follows Fig.16A The working principle shown is to play the audio data of the video application on the screen projection target device. The recording application can execute the following Fig. 16BThe working principle shown is to enable the recording application to collect audio data through the screen projection target device. The call application can execute the following Fig.16A and Fig. 16B The working principle shown is to realize the playback of audio data of the call application on the screen projection target end and the collection of audio data through the screen projection target end.

[0212] In another implementation, the camera application may execute the following Fig. 16C The working principle shown is to enable the camera application to collect image or video information through the screen projection target device. In this implementation, the camera application can send an image or video information collection instruction containing image or video format information to complete the image or video information collection.

[0213] It should be noted that if Figure 16A-16C The working principle of the screen projection source device 1201 calling the screen projection destination device 901 device can be implemented in two ways. In one of the ways, the screen projection destination device sends the device call instruction or data stream to the audio service and camera service of the screen projection destination device, and the audio service and camera service perform device selection configuration on the audio access component and camera access component of the screen projection destination device. The audio access component and the camera access component call the local audio HAL and local camera HAL of the screen projection destination device in response to the above configuration, thereby realizing the call of the multimedia device of the screen projection destination device. The advantage of this method is that it can optimize the device call of the screen projection destination device according to the device configuration of the screen projection destination device. In another way, the screen projection destination device sends the device call instruction and data stream directly to the local audio HAL and local camera HAL of the screen projection destination device to realize the call of the multimedia device of the screen projection destination device. The advantage of this method is that it minimizes the path of hardware virtualization, realizes multiple instances of a set of code, increases the reusability of the code, and saves software resources.

[0214] Through the device switching process of the above steps 403 and 404, the screen projection source device responds to the first operation message from the screen projection destination device, and intelligently switches to use the device of the screen projection destination device, thereby realizing rapid switching of device capabilities between the screen projection source device and the screen projection destination device, and optimizing the user experience of the screen projection destination device in the screen projection scenario.

[0215] Next, in step 405, the screen projection source device detects a second operation on the screen projection source device.

[0216] The second operation may be a touch operation, an air gesture operation, etc. Fig.17Taking the scenario shown in the figure as an example, the screen projection source device 1702 can be a mobile phone, and the screen projection destination device 1701 can be a laptop computer. The mobile phone is using the multimedia device of the laptop computer to process multimedia information. Fig.17 As shown, when a second operation is performed on the mobile phone screen, the mobile phone will generate a second operation message. Fig.17 In the figure, only the projection window is taken as the interface corresponding to the video call application as an example. It should be understood that the projection window can also contain audio and video application interfaces, camera application interfaces, etc.

[0217] Specifically, the second operation may be a user's touch operation or air gesture operation on the screen 1710 of the screen projection source device 1702. The touch operation includes a single touch, a double touch, a triple touch, a long press touch, a sliding touch, a two-finger touch, a three-finger touch, etc. The touch position may be any position on the screen 1710 of the screen projection source device 1702, such as the system status bar 1711, the video call display content 1712, the video call control button 1713, or other areas 1714 on the screen 1710 of the screen projection source device 1702. The air gesture operation includes a hover gesture, a close gesture, a far gesture, a sliding gesture, etc.

[0218] Optionally, the second operation may also be a voice control operation on the screen projection source device 1702. For example, taking the screen projection source device 1702 as a mobile phone, the user may first trigger the voice assistant of the mobile phone, and then the user may say voice control sentences such as "switch to use the device of the mobile phone", "use the mobile phone to play sound", and "use the camera of the mobile phone" to the mobile phone.

[0219] Optionally, the second operation may also be a pressing operation on a physical button of the screen projection source device. For example, still taking the screen projection source device as a mobile phone as an example, the user may press the audio adjustment button on the side of the mobile phone.

[0220] In some embodiments, performing a second operation on the projection source device can trigger a corresponding event, and accordingly, a second operation message will be generated, and the second operation message can instruct the projection source device to complete the corresponding event triggered by the second operation. For example, the triggered event can be opening an application on the projection source device. The triggered event can also be an event triggered by a gesture operation performed on the projection source device, such as a drop-down menu operation triggered in response to a user's sliding gesture on the upper end of the projection source screen. It will be understood that the present application does not specifically limit the triggered event.

[0221] Finally, in step 406, in response to the second operation, the first application of the screen projection source device resumes using the first multimedia device of the screen projection source device itself to process multimedia information.

[0222] It should be understood that the first application of the screen projection source device resumes the use of the second multimedia device of the screen projection destination device for multimedia information processing, which means stopping or pausing the use of the second multimedia device of the screen projection destination device for multimedia information processing, thereby using the first multimedia device of the screen projection source device for multimedia information processing. At this time, the screen projection source device and the screen projection destination device are still in the screen projection connection state, that is, the screen projection connection between the screen projection source device and the screen projection destination device will not be disconnected due to the second operation on the screen projection source device.

[0223] The introduction of the first multimedia device and the use of the multimedia device to process multimedia information is detailed in the related description of step 402 and will not be repeated here.

[0224] In some embodiments, the screen projection source device can generate a second switching instruction in response to the received second operation, and call the second multimedia abstract module according to the second switching instruction. The second multimedia abstract module is used to call the first multimedia device of the screen projection source device.

[0225] Combine the following Figures 18A-18D The working principle of the screen projection source device detecting the second operation on the screen projection source device and responding to the second operation in step 405 and step 406, and the first application of the screen projection source device resuming to call the multimedia device of the screen projection source device itself is described in detail. Figures 18A-18D The screen projection source device 1201 can be Fig.17 Mobile phone in.

[0226] First pass Fig.18A The working principle of the screen projection source device detecting a second operation on the screen projection source device and generating a second switching instruction in response to the second operation is introduced.

[0227] like Fig.18A As shown, the screen projection source device can detect the user's second operation on the screen projection source device through the HAL layer input device access node 1810. For example, the screen projection source device can obtain the touch input event obtained from the touch screen through the device access node "dev / input / eventX" 1811. The above HAL layer input device access node 1810 reports the input event of the second operation to the IMS module 1221 located in the display service 1220.

[0228] First, the InputReader in the IMS module 1221 can continuously read the second operation input event reported by the HAL layer input device access node 1810, translate and process it into an Android input event, and distribute the event to the InputDispatcher module. The InputDispatcher module delivers the event to the appropriate window for processing according to the window information provided by the WMS module 1222. The window is then sent to the corresponding application at the application layer along the control tree according to ViewRootImpl. The above application processes the input event and performs related operations corresponding to the input event.

[0229] In some embodiments, the IMS module 1221 can also send the above-mentioned input event to the DDMS module 1223. After receiving the input event, the above-mentioned DDMS module 1223 sends the identification information of the screen projection source device 1201 itself to the switching identification component 1212 located in the screen access component 1210. Among them, the description of the identification information is detailed in the relevant instructions in step 402, which will not be repeated here. At this time, the record of the identification information received for the last time saved in the switching identification component 1212 is the identification information of the screen projection destination device. Exemplarily, as shown in Table 5, the record of the identification information received for the last time saved in the current switching identification component 1212 is the device ID of the screen projection destination device 901.

[0230] Table 5

[0231] The last received identification information 1A21412E2F54F…(the ID of the device to which the screen is projected)

[0232] Further, after receiving the identification information of the screen projection source device 1201 from the DDMS module 1223, the switching identification component 1212 compares the identification information of the screen projection source device 1201 with the identification information received last in the record, and determines that the identification information received from the screen projection source device 1201 is different from the identification information received last in the record, thereby updating the identification information received last in the record to the identification information of the screen projection source device 1201. Exemplarily, the updated record is shown in Table 6.

[0233] Table 6

[0234] The last received identification information 2A11682E2F54F…(ID of the source device)

[0235] Subsequently, the switching identification component 1212 sends a second switching instruction to the intelligent switching module 1231 located in the distributed device management service. The instruction includes the device identification information received last time. At this time, the identification information is the identification information of the screen projection source device.

[0236] Further, the intelligent switching module 1231 located in the distributed device management service receives the second switching instruction from the switching identification component 1212. The intelligent switching module 1231 extracts the local multimedia HAL corresponding to the identification information of the screen projection source device 1201 from the correspondence between the device identification information and the multimedia device HAL stored in the distributed device management service, such as the Audio HAL and the local Camera HAL, based on the identification information of the screen projection source device 1201 contained in the second switching instruction, generates a device switching scheduling instruction for indicating the device capability of using the local multimedia HAL, and sends the device switching scheduling instruction to the audio service module 1241 and the Camera service module 1243. Exemplarily, the correspondence between the device identification information and the multimedia device HAL stored in the distributed device management service is shown in Table 7.

[0237] Table 7

[0238]

[0239]

[0240] According to the following Figure 18B-Figure 18D , introduces the working principle of the first application of the screen projection source device resuming the call of its own first multimedia device in response to the second switching instruction.

[0241] In response to the second switching instruction, the intelligent switching module 1231 sends a device switching scheduling instruction for instructing the device capability of calling the local multimedia HAL to the audio service module 1241 and the Camera service module 1243 .

[0242] In one implementation, Fig.18BAs shown, the first application is a video call application, and the video call application 1801 sends a service dispatch requesting audio capability and camera capability to the audio service module 1241 and the Camera service module 1243. In response to the service dispatch, the audio service module 1241 determines that the current video call application 1801 has a demand for audio service. The audio service module 1241 also responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the audio device of the screen projection destination device by calling the local AudioHAL1251, and performs device selection configuration on the audio access component 1242. The above configuration is used to instruct the audio access component 1242 to call the local Audio HAL 1251. After completing the above-mentioned device selection configuration, the audio access component 1242 stops sending the audio stream to the virtual Audio1 HAL 1252 located in the HAL layer and stops receiving the audio stream from the virtual Audio1 HAL 1252; the audio access component 1242 starts sending the audio stream to the local Audio HAL 1251 located in the HAL layer and receives the audio stream from the local Audio HAL 1251. The above-mentioned local Audio HAL 1251 can send the audio stream to the audio device of the screen projection source device itself, and can also receive the audio stream collected by the audio device of the screen projection source device itself. Correspondingly, the Camera service module 1243 determines that the current video call application 1801 has a demand for camera services in response to the service scheduling instruction. The Camera service module 1243 simultaneously responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the camera device of the screen projection destination device by calling the local Camera HAL 1253, and performs device selection configuration on the camera access component 1244, which is used to instruct the camera access component 1244 to call the local CameraHAL 1253. After completing the above device selection configuration, the Camera access component 1244 stops receiving the video stream from the virtual Camera1 HAL1254; the Camera access component 1244 starts to receive the video stream from the local Audio HAL 1253. The above local Camera HAL 1253 can receive the video stream collected by the camera device of the screen projection source end itself. In some embodiments, the audio access component 1242 and the Camera access component 1244 can also send relevant control instructions or configuration instructions to the local Audio HAL 1251 and the local Camera HAL 1253 to control or configure the audio device or camera device of the screen projection source end device itself.

[0243] In another implementation, Fig.18CAs shown, the first application is a video application, and the video application 1802 sends a service dispatch requesting audio capabilities to the audio service module 1241. In response to the service dispatch, the audio service module 1241 determines that the current video application 1802 has a demand for audio services. At the same time, in response to the device switching scheduling instruction, the audio service module 1241 determines that the current device scheduling strategy is to use the audio device of the screen projection source device itself by calling the local Audio HAL 1251, and configures the device selection for the audio access component 1242. The above configuration is used to instruct the audio access component 1242 to call the local Audio HAL 1251. After completing the above device selection configuration, the audio access component 1242 stops sending the audio stream to the virtual Audio1 HAL 1252 located in the HAL layer; the audio access component 1242 starts sending the audio stream to the local Audio HAL 1251 located in the HAL layer. The above local Audio HAL 1251 can send the audio stream to the audio device of the screen projection source device itself. Although the Camera service module 1243 has received the device switching scheduling instruction, it has not received the service scheduling issued by the application, so no camera device is being called. In some embodiments, the audio access component 1242 can also send relevant control instructions or configuration instructions to the local Audio1 HAL 1251 to control or configure the audio device of the projection source device itself.

[0244] In another implementation, Fig.18DAs shown, the first application is a camera application, and the camera application 1803 sends a service schedule to the Camera service module 1243. In response to the service schedule, the Camera service module 1243 determines that the current camera application 1803 has a demand for camera services. The Camera service module 1243 also responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the camera device of the projection source device itself by calling the local Camera HAL 1253, and performs device selection configuration on the camera access component 1244, and the above configuration is used to instruct the camera access component 1244 to call the local Camera HAL1253. After completing the above device selection configuration, the Camera access component 1244 stops receiving images or videos from the virtual Camera1 HAL 1254; the Camera access component 1244 starts to receive images or videos from the local Audio HAL1253. The above local Camera HAL 1253 can receive images or videos captured by the camera device of the projection source device itself. Although the audio service module 1241 has received the device switching scheduling instruction, it has not received the service scheduling issued by the application, so no audio device is being called. In some embodiments, the Camera access component 1244 can also send relevant control instructions or configuration instructions to the local Camera HAL 1253 to control or configure the camera device of the projection source device itself.

[0245] Through the device switching process of steps 405 and 406 above, the screen projection source device can respond to the user's second operation on the screen projection source device itself, and intelligently switch the device capability from the screen projection destination device back to the screen projection source device, thereby optimizing the user experience when operating between the screen projection source device and the screen projection destination device.

[0246] By using Figure 4 The device capability intelligent switching method shown in the figure can realize that in the scenario where the screen projection source device is used to project the screen on the screen projection target device, the screen projection target device switches to use the device of the screen projection target device in response to the user's first operation on the screen projection window of the screen projection target device, and switches to use the device of the screen projection target device in response to the user's second operation on the screen projection source device. The above method improves the continuity of the user when switching between the screen projection source device and the screen projection target device, thereby improving the user's operating experience.

[0247] Next, refer to Fig.19 , another device switching method provided by an embodiment of the present application is specifically introduced. The method can be executed by any electronic device with a display screen and data processing capabilities. Fig.19As shown, the method includes: step 1901, the screen projection source device runs a first application, the screen projection source device displays the interface of the first application on its screen, the screen projection source device projects the content of the first application to the first screen projection destination device and the second screen projection destination device, the first screen projection destination device displays a first projection window on its display screen, the second screen projection destination device displays a second projection window on its display screen, and the first projection window and the second projection window display the interface of the first application; step 1902, the first application running on the screen projection source device uses the first multimedia device of the screen projection source device itself to process multimedia information; step 1903, the first screen projection destination device generates a first operation message in response to the first operation on the projection window and sends it to the screen projection source device; step 1904, the screen projection source The screen projection source device receives the first operation message, and in response to the first operation message, the first application of the screen projection source device switches to use the second multimedia device of the first screen projection destination device to process multimedia information; step 1905, the second screen projection destination device generates a second operation message in response to the second operation on the screen projection window and sends it to the screen projection source device; step 1906, the screen projection source device receives the second operation message, and in response to the second operation message, the first application of the screen projection source device switches to use the third multimedia device of the second screen projection destination device to process multimedia information; step 1907, the screen projection source device detects the third operation on the screen projection source device; step 1908, in response to the third operation, the first application of the screen projection source device resumes using the first multimedia device of the screen projection source device itself to process multimedia information. It should be understood that the screen projection destination device and the screen projection source device are both electronic devices.

[0248] First, step 1901, the screen projection source device runs the first application, the screen projection source device displays the interface of the first application on its screen, the screen projection source device projects the content of the first application to the first screen projection destination device and the second screen projection destination device, the first screen projection destination device displays the first projection window on its display screen, the second screen projection destination device displays the second projection window on its display screen, and the first projection window and the second projection window display the interface of the first application.

[0249] Among them, the first application running on the above-mentioned projection source device can be a video call application, an audio and video application, a camera application, etc. Correspondingly, when the above-mentioned first application is running on the projection source device, the projection source device displays the interface corresponding to the above-mentioned first application on the screen. The projection source device can project the content of the above-mentioned first application on the first projection destination device and the second projection destination device at the same time, so that the screen display of the first projection destination device and the projection window on the second projection destination device both display the interface corresponding to the first application running on the projection source device. For example, if Fig. 20As shown, the screen projection source device 2001 is a mobile phone, the first screen projection destination device 2002 is a laptop computer, and the second screen projection destination device 2003 is a tablet computer. The mobile phone can project the content of the first application it is running on the laptop computer and the tablet computer at the same time. It should be noted that Fig. 20 In the example, the first application displayed in the projection window is the interface corresponding to the video call application. It should be understood that in some other implementations, the projection window can also display the interface corresponding to the audio and video application, the interface corresponding to the camera application, and the interface corresponding to the desktop application. For details about the projection window and the corresponding interface of the application displayed in the projection window, see Figure 4 The relevant description in step 401 in will not be repeated here.

[0250] For a detailed description of the screen projection steps, see Figure 4 Step 401 in will not be repeated here.

[0251] Steps 1902-1904 can refer to Figure 4 Steps 402 to 404 in the method are not described in detail here.

[0252] Step 1905: The second screen projection destination device generates a second operation message in response to the second operation on the screen projection window and sends it to the screen projection source device.

[0253] The second projection destination device responds to the second operation, such as the touch operation on the projection window on the second projection destination device, by sending the generated second operation message to the projection source device through the codec transmission module in the screen access component, wherein the second operation message includes the second operation input event and the identification information of the second projection destination device. For a detailed description of the second projection destination device detecting and responding to the second operation to generate the second operation message, see Fig. 9 Related description.

[0254] Step 1906: The screen projection source device receives a second operation message. In response to the second operation message, the first application of the screen projection source device switches to use the third multimedia device of the second screen projection destination device to process multimedia information.

[0255] The third multimedia device of the second screen projection destination device may include an audio device, such as a speaker, a microphone, etc. of the second screen projection destination device; the third multimedia device of the second screen projection destination device may also include a camera device, such as a camera of the second screen projection destination device, etc. For the introduction of using multimedia devices to process multimedia information, please refer to the relevant description in step 402 of the previous embodiment, which will not be repeated here.

[0256] Optionally, the third multimedia device using the second screen projection destination device also includes using a device that establishes a connection with the second screen projection destination device, such as audio devices such as headphones and speakers that establish a wireless or wired connection with the second screen projection destination device, or imaging devices such as cameras or camcorders that establish a wireless or wired connection with the second screen projection destination device.

[0257] In some embodiments, the screen projection source device can generate a second switching instruction in response to the identification information of the second screen projection destination device in the received second operation message, and call the second multimedia abstract module according to the second switching instruction. The second multimedia abstract module is used to call the third multimedia device of the second screen projection destination device.

[0258] According to the following Figure 21A-Figure 21D , describes in detail the working principle of step 1906. Among them, the screen projection source device 2101 can be Fig. 20 The first screen projection destination device 2103 may be Fig. 20 The second screen projection destination device 2102 may be a laptop computer in the Fig. 20 Tablet computer in the room.

[0259] First, according to Fig.21A The present invention introduces the working principle that a screen projection source device receives a second operation message and generates a second switching instruction in response to the second operation message.

[0260] First, the encoding and decoding transmission module 2121 of the screen projection source device 2101 receives and decodes the second operation message 2104 from the second screen projection destination device 2102 to obtain the second operation input event and the identification information of the second screen projection destination device.

[0261] In some embodiments, the codec transmission module 2121 can send the second operation input event to the IMS module of the display service 2130 of the screen projection source device 2101, and the IMS module distributes the event to the WMS module, and the WMS module reports the event to the application corresponding to the input event at the application layer. Through the above processing of the second operation input event, the user's operation on the second screen projection destination device can be synchronized to the screen projection source device.

[0262] Subsequently, the codec transmission module 2121 sends the decoded identification information of the second screen projection destination device 2102 to the switching identification component 2122. At this time, the record of the identification information received last time in the switching identification component 2122 is the identification information of the first screen projection destination device 2103. Exemplarily, as shown in Table 8, the record of the identification information received last time in the current switching identification component 2122 is the device ID of the first screen projection destination device.

[0263] Table 8

[0264] The last received identification information 1A21412E2F54F…(ID of the first screen projection destination device)

[0265] When the switching identification component 2122 receives the identification information of the second screen projection destination device 2102 from the codec transmission module 2121, the identification information is compared with the identification information received last in the record, and it is determined that the identification information received from the second screen projection destination device 2102 is different from the identification information received last in the record, so the identification information received last in the record is updated to the identification information of the second screen projection destination device 2102. Exemplarily, the updated record is shown in Table 9.

[0266] Table 9

[0267] The last received identification information 3A36174C2A54E…(ID of the second screen projection destination device)

[0268] Furthermore, the switching identification component 2122 sends a second switching instruction to the intelligent switching module 2140 located in the distributed device management service, and the instruction includes the device identification information received last time, that is, the identification information of the second screen projection destination device.

[0269] The intelligent switching module 2140 located in the distributed device management service receives the second switching instruction from the switching identification component 2122. The intelligent switching module 2140 extracts the multimedia device HAL corresponding to the identification information of the second screen projection destination device 2102 from the correspondence between the device identification information and the multimedia device HAL stored in the distributed device management service, such as the virtual Audio2 HAL and the virtual Camera2 HAL, based on the identification information of the second screen projection destination device 2102 contained in the second switching instruction, and generates a device switching scheduling instruction for indicating the use of the multimedia device HAL corresponding to the identification information of the second screen projection destination device 2102, and sends the device switching scheduling instruction to the audio service module 2151 and the Camera service module 2153. Exemplarily, the correspondence between the device identification information and the multimedia device HAL stored in the distributed device management service is shown in Table 10.

[0270] Table 10

[0271] Device identification information Multimedia Device HAL 2A11682E2F54F…(screen projection source device ID) Local Audio HAL 2A11682E2F54F…(screen projection source device ID) Native Camera HAL 1A21412E2F54F…(ID of the first screen projection destination device) Virtual Audio1 HAL 1A21412E2F54F…(ID of the first screen projection destination device) Virtual Camera1 HAL 3A36174C2A54E…(ID of the second screen projection destination device) Virtual Audio2 HAL 3A36174C2A54E…(ID of the second screen projection destination device) Virtual Camera2 HAL … …

[0272] Combine the following Figure 21B-Figure 21D The working principle of a first application of a screen projection source device calling a multimedia device of a second screen projection destination device in response to a second switching instruction is introduced.

[0273] In response to the second switching instruction, the intelligent switching module 2140 sends a device switching scheduling instruction to the audio service module 2151 and the Camera service module 2153 to indicate the calling and second screen projection destination device 2102 corresponding to the multimedia device HAL identification information.

[0274] In one implementation, Fig.21BAs shown, the first application is a video call application, and the video call application 2171 sends a service dispatch requesting audio capability and camera capability to the audio service module 2151 and the Camera service module 2153. In response to the service dispatch, the audio service module 2151 determines that the current video call application 2171 has a demand for audio service. The audio service module 2151 also responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the audio device of the second screen projection destination device by calling the virtual Audio2HAL2162, and performs device selection configuration on the audio access component 2152, and the above configuration is used to instruct the audio access component 2152 to call the virtual Audio2 HAL 2162. After completing the above device selection configuration, the audio access component 2152 stops sending the audio stream to the virtual Audio1 HAL 2161 located in the HAL layer and stops receiving the audio stream from the virtual Audio1 HAL 2161; the audio access component 2152 starts sending the audio stream to the virtual Audio2 HAL 2162 located in the HAL layer and receives the audio stream from the virtual Audio2 HAL 2162. The above virtual Audio2 HAL2162 can send the audio stream to the audio device of the second screen projection destination device, and can also receive the audio stream collected by the audio device of the second screen projection destination device. Correspondingly, the Camera service module 2153 determines in response to the service scheduling that the current video call application 2171 has a demand for camera services. The Camera service module 2153 simultaneously responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the camera device of the second screen projection destination device by calling the virtual Camera2 HAL 2164, and performs device selection configuration on the camera access component 2154, and the above configuration is used to instruct the camera access component 2154 to call the virtual Camera2HAL 2164. After completing the above device selection configuration, the Camera access component 2154 stops receiving the video stream from the virtual Camera1 HAL2163; the Camera access component 2154 starts to receive the video stream from the virtual Audio2 HAL 2164. The above virtual Camera2 HAL 2164 can receive the video stream collected by the camera device of the second screen projection destination. In some embodiments, the audio access component 2152 and the Camera access component 2154 can also send relevant control instructions or configuration instructions to the virtual Audio2 HAL2162 and the virtual Camera2 HAL 2164 to control or configure the audio device or camera device of the second screen projection destination device.

[0275] In another implementation, Fig. 21CAs shown, the first application is a video application, and the video application 2172 sends a service schedule to the audio service module 2151. In response to the service schedule, the audio service module 2151 determines that the current video application 2172 has a demand for audio services. At the same time, in response to the device switching scheduling instruction, the audio service module 2151 determines that the current device scheduling strategy is to use the audio device of the second screen projection destination device by calling the virtual Audio2 HAL 2162, and performs device selection configuration on the audio access component 2152, and the above configuration is used to instruct the audio access component 2152 to call the virtual Audio2 HAL2162. After completing the above device selection configuration, the audio access component 2152 stops sending the audio stream to the virtual Audio1 HAL2161 located in the HAL layer; the audio access component 2152 starts sending the audio stream to the virtual Audio2 HAL2162 located in the HAL layer. The above virtual Audio2 HAL 2162 can send the audio stream to the audio device of the second screen projection destination device. In some embodiments, the audio access component 2152 may also send relevant control instructions or configuration instructions to the virtual Audio2 HAL 2162 to control or configure the audio device of the second screen projection destination device.

[0276] In another implementation, Fig.21D As shown, the first application is a camera application, and the camera application 2173 sends a service schedule to the Camera service module 2151. The Camera service module 2153 determines that the current video call application 2171 has a demand for camera services in response to the service schedule. The Camera service module 2153 also responds to the device switching scheduling instruction, determines that the current device scheduling strategy is to use the camera device of the second screen projection destination device by calling the virtual Camera2 HAL 2164, and performs device selection configuration on the camera access component 2154, and the above configuration is used to instruct the camera access component 2154 to call the virtual Camera2 HAL 2164. After completing the above device selection configuration, the Camera access component 2154 stops receiving the video stream from the virtual Camera1 HAL 2163; the Camera access component 2154 starts to receive the video stream from the virtual Audio2 HAL2164. The above virtual Camera2 HAL 2164 can receive the video stream collected by the camera device of the second screen projection destination. In some embodiments, the Camera access component 2154 may also send relevant control instructions or configuration instructions to the virtual Camera2 HAL 2164 to control or configure the camera device of the second screen projection destination device.

[0277] Through the above-mentioned device switching process of steps 1902 to 1906, in a multi-screen projection scenario, the projection source device responds to operation messages from different projection destination devices, intelligently switches and calls the device capabilities of different projection destination devices, and realizes the rapid switching of device capabilities between the projection source device and different projection destination devices, thereby optimizing the user experience of the projection destination devices in a multi-screen projection scenario.

[0278] Step 1907 and step 1908 can refer to Figure 4 Steps 405 and 406 in the method are not described in detail here.

[0279] In this embodiment, two screen projection target devices are taken as an example to describe the device switching method in a multi-screen projection scenario. It should be noted that the screen projection target devices in the multi-screen projection scenario can be three or more, and this application does not impose any restrictions.

[0280] In this embodiment, the screen projection source device can still switch to use at least one of the audio devices and camera devices of different screen projection destination devices according to the multimedia device capabilities of the screen projection destination device and according to the device switching settings.

[0281] Through the above-mentioned device switching process, the screen projection source device can respond to the user's control operations on the screen projection source device itself, and intelligently switch the device capabilities from multiple screen projection destination devices back to the screen projection source device, thereby optimizing the user experience when operating between the screen projection source device and multiple screen projection destination devices in multi-screen projection scenarios.

[0282] Fig. 22 An example is given Figure 4 An example scenario of a method for using a device between a screen projection source device and a screen projection destination device in the example scenario. In the example scenario, the screen projection source device is in the form of a mobile phone, and the screen projection destination device is in the form of a laptop computer. Of course, the screen projection source device and the screen projection destination device can be in the form of other electronic devices, such as tablet computers, car display screens, smart TVs, and other devices.

[0283] like Fig. 22 As shown in (A), the mobile phone is currently using a video call application, and the interface of the video call application is displayed on its display screen. The mobile phone projects the interface of the video call application onto the laptop computer, and the laptop computer displays a projection window on its display screen, and displays the interface of the video call application in the projection window. The video call application enables the mobile phone to use its own camera device (such as a camera) to capture video, and also enables the mobile phone to use its own audio device (such as a speaker, a microphone) to play and capture voice. Correspondingly, the video call interface in the projection window of the laptop computer displays the video screen captured by the mobile phone's own camera device.

[0284] like Fig. 22 As shown in (B), when the user touches the projection window on the laptop display, the switching of the device usage will be triggered. It should be understood that in this example, only the touch operation is used as an example to illustrate the user's operation on the projection window. The operation can also be a hover gesture operation, a mouse click operation, a voice control operation, etc.

[0285] like Fig. 22 As shown in (C), in response to the user's touch operation on the projection window on the laptop display, the mobile phone will switch to using the laptop's camera device (such as a camera) for video capture and the laptop's audio device (such as a speaker, a microphone) for voice playback and capture. Correspondingly, the video call interface in the projection window of the laptop displays the video image captured by the laptop's camera device.

[0286] like Fig. 22 As shown in (D), when the user touches the screen of the mobile phone, the switching of the device usage will be triggered. In response to the user's touch operation on the screen of the mobile phone, the mobile phone will switch back to using the mobile phone's own camera device for video acquisition and the mobile phone's own audio device for voice playback and acquisition, that is, it will return to the scene of (A). It should be understood that in this example, only the touch operation is used as an example to illustrate the user's operation on the mobile phone. The operation can also be a hover gesture operation, a voice control operation, etc.

[0287] It is important to understand that Figure 4 The device usage method between the screen projection source device and the screen projection target device is not only applicable to the above Fig. 22 The scenario shown in which a video call application interface is being displayed on a mobile phone and a laptop computer is also applicable to different scenarios in which a camera application interface, an audio and video playback application interface, and a desktop application interface are being displayed on a mobile phone and a laptop computer.

[0288] Fig.23 An example is given Fig.19 An example scenario of a method for using a device between a screen projection source device and multiple screen projection destination devices. In the example scenario, the screen projection source device takes the form of a mobile phone, the first screen projection destination device takes the form of a laptop computer, and the second screen projection destination device takes the form of a tablet computer. Of course, the screen projection source device and the first and second screen projection destination devices can take the form of other electronic devices, such as tablet computers, car display screens, smart TVs and other devices.

[0289] like Fig.23As shown in (A), the mobile phone is currently using a video call application, and the interface of the video call application is displayed on its display screen. The mobile phone projects the interface of the video call application onto the laptop and tablet computer. The laptop displays a projection window on its display screen, and displays the interface of the video call application in the projection window. The tablet computer also displays a projection window on its display screen, and displays the interface of the video call application in the projection window. The implementation of the video call application enables the mobile phone to use its own camera device (such as a camera) to capture video, and also enables the mobile phone to use its own audio device (such as a speaker, microphone) to play and capture voice. Correspondingly, the video call interface in the projection window of the laptop and tablet computer displays the video screen captured by the mobile phone's own camera device.

[0290] like Fig.23 As shown in (B), when the user touches the projection window on the laptop display, the switching of the device usage will be triggered. It should be understood that in this example, only the touch operation is used as an example to illustrate the user's operation on the projection window. The operation can also be a hover gesture operation, a mouse click operation, a voice control operation, etc.

[0291] like Fig.23 As shown in (C), in response to the user's touch operation on the projection window on the laptop display, the mobile phone will switch to using the laptop's camera device (such as a camera) for video capture and the laptop's audio device (such as a speaker, a microphone) for voice playback and capture. Correspondingly, the video call interface in the projection window of the laptop and the tablet computer displays the video image captured by the laptop's camera device.

[0292] like Fig.23 As shown in (D), when the user touches the projection window on the tablet display, the switching of the device usage will be triggered. It should be understood that in this example, only the touch operation is used as an example to illustrate the user's operation on the projection window. The operation can also be a hover gesture operation, a mouse click operation, a voice control operation, etc.

[0293] like Fig.23 As shown in (E), in response to the user's touch operation on the projection window on the tablet display, the mobile phone will switch to using the tablet's camera device (such as a camera) for video capture and the tablet's audio device (such as a speaker, a microphone) for voice playback and capture. Correspondingly, the video call interface in the projection window of the laptop and tablet displays the video image captured by the tablet's camera device.

[0294] like Fig.23As shown in (F), when the user touches the screen of the mobile phone, the switching of the device usage will be triggered. In response to the user's touch operation on the screen of the mobile phone, the mobile phone will switch back to using the mobile phone's own camera device to collect video information and use the mobile phone's own audio device to play and collect voice, that is, restore to the scene of (A). It should be understood that in this example, only the touch operation is used as an example to illustrate the user's operation on the mobile phone. The operation can also be a hover gesture operation, a voice control operation, etc.

[0295] It is important to understand that Fig.19 The device usage method between the screen projection source device and the screen projection target device is not only applicable to the above Fig.23 The scenario shown in which a video call application interface is being displayed on a mobile phone and a laptop computer is also applicable to different scenarios in which a camera application interface, an audio and video playback application interface, and a desktop application interface are being displayed on a mobile phone and a laptop computer. Fig.23 The method of using the device is illustrated by taking the scenario in which the screen projection source device projects the screen on two screen projection target devices as an example. It can be understood that this method is also applicable to the scenario in which the screen projection source device projects the screen on three or more screen projection target devices.

[0296] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0297] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0298] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0299] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of a device or in the form of a software functional unit.

[0300] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0301] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0302] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A device use method, applied to a screen projection source device, characterized in that: include: The screen projection source device runs a first application, and the screen projection source device projects the interface of the first application to the screen projection destination device, so that the screen projection destination device displays a screen projection window on its display screen, and the screen projection window displays the interface of the first application, wherein the first application uses the first multimedia device of the screen projection source device to process multimedia information, and the screen projection source device and the screen projection destination device are in a screen projection connection state; The screen projection source device receives a first operation message from the screen projection destination device, wherein the first operation message is a message generated by the screen projection destination device in response to a first operation on the screen projection window detected by the screen projection destination device; In response to the first operation message, the first application of the screen projection source device stops or suspends using the first multimedia device of the screen projection source device to process multimedia information, and uses the second multimedia device of the screen projection destination device to process multimedia information; The screen projection source device detects a second operation on the screen projection source device, and in response to the second operation, the first application of the screen projection source device resumes using the first multimedia device of the screen projection source device to process multimedia information, wherein the screen projection source device and the screen projection destination device are still in a screen projection connection state; Wherein, the first multimedia device includes a first audio device, the second multimedia device includes a second audio device, the first audio device and the second audio device are used for audio information collection and / or audio information playback, and the multimedia information processing includes audio information collection and / or audio information playback; and / or The first multimedia device includes a first camera device, the second multimedia device includes a second camera device, the first camera device and the second camera device are used for video information acquisition or image information acquisition, and the multimedia information processing includes video information acquisition or image information acquisition.

2. The method according to claim 1, characterized in that: The first operation message includes the first identification information of the screen projection destination device.

3. The method according to claim 2, characterized in that In response to the first operation message, the first application of the screen projection source device switches to use the second multimedia device of the screen projection destination device to process multimedia information, including: The screen projection source device determines that the received first identification information of the screen projection destination device is different from the last received identification information saved by the screen projection source device, and generates a first switching instruction; In response to the first switching instruction, the first application of the screen projection source device calls the second multimedia device of the screen projection destination device.

4. The method according to claim 3, characterized in that After the screen projection source device determines that the first identification information received from the screen projection destination device is different from the last received identification information saved by the screen projection source device, the last received identification information is updated to the first identification information of the screen projection destination device.

5. The method according to claim 3 or 4, characterized in that: In response to the first switching instruction, the first application of the screen projection source device calls the second multimedia device of the screen projection destination device, including: The first application of the screen projection source device calls the first multimedia abstract module of the hardware abstraction layer of the screen projection source device according to the first corresponding relationship, Among them, the first corresponding relationship is the mapping relationship between the first identification information of the screen projection destination device and the first multimedia abstract module, the first multimedia abstract module includes a first audio abstract module and / or a first camera abstract module, the first audio abstract module is used to call the second audio device of the screen projection destination device to collect audio information and / or play audio information, and the first camera abstract module is used to call the second camera device of the screen projection destination device to collect video information or image information.

6. The method according to claim 1, characterized in that In response to the second operation, the first application of the screen projection source device resumes using the first multimedia device of the screen projection source device to process multimedia information, including: In response to the second operation, the screen projection source device generates second identification information; The screen projection source device determines that the second identification information of the screen projection source device is different from the identification information received last time and stored by the screen projection source device, and generates a second switching instruction; In response to the second switching instruction, the first application of the screen projection source device calls the first multimedia component of the screen projection source device.

7. The method according to claim 6, characterized in that After the screen projection source device determines that the second identification information received from the screen projection source device is different from the identification information last received and saved by the screen projection source device, the identification information last received is updated to the second identification information of the screen projection source device.

8. The method according to claim 6 or 7, characterized in that: In response to the second switching instruction, the first application of the screen projection source device calls the first multimedia device of the screen projection source device, including: The first application of the screen projection source device calls the second multimedia abstract module of the hardware abstraction layer of the screen projection source device according to the second corresponding relationship, Among them, the second corresponding relationship is the mapping relationship between the second identification information of the screen projection source device and the second multimedia abstract module, the second multimedia abstract module includes a second audio abstract module and / or a second camera abstract module, the second audio abstract module is used to call the first audio device of the screen projection source device to collect audio information and / or play audio information, and the second camera abstract module is used to call the first camera device of the screen projection source device to collect video information or image information.

9. The method according to any one of claims 1-4, 6-7, characterized in that: The screen projection source device projects the interface of the first application to the screen projection destination device further comprising: The screen projection source device projects the interface of the first application to another screen projection destination device, so that the other screen projection destination device displays another screen projection window on its display screen, and the other screen projection window displays the interface of the first application; Before the screen projection source device detects the second operation on the screen projection source device, the following further includes: The screen projection source device receives a third operation message from the other screen projection destination device, wherein the third operation message is a message generated by the other screen projection destination device in response to a third operation on the other screen projection window detected by the other screen projection destination device; In response to the third operation message, the first application of the screen projection source device switches to use the third multimedia device of the other screen projection destination device to process multimedia information.

10. A device use method, applied to a screen projection destination device, characterized in that: include: The screen projection destination device receives the projection data from the screen projection source device, and in response to the projection data, the screen projection destination device displays a projection window on its display screen, wherein the projection window displays the interface of the first application running on the screen projection source device, wherein the screen projection source device uses the first multimedia device of the screen projection source device to process multimedia information; The screen projection destination device detects a first operation on the screen projection window; In response to the first operation, the screen projection destination device generates first operation information and sends the first operation information to the screen projection source device, wherein the first operation information is used to trigger the screen projection source device to stop or suspend using the first multimedia device of the screen projection source device to process multimedia information, and use the second multimedia device of the screen projection destination device to process multimedia information; Wherein, the first multimedia device includes a first audio device, the second multimedia device includes a second audio device, the first audio device and the second audio device are used for audio information collection and / or audio information playback, and the multimedia information processing includes audio information collection and / or audio information playback; and / or The first multimedia device includes a first camera device, the second multimedia device includes a second camera device, the first camera device and the second camera device are used for video information acquisition or image information acquisition, and the multimedia information processing includes video information acquisition or image information acquisition.

11. The method according to claim 10, characterized in that The first operation information includes the first identification information of the screen projection destination device.

12. A screen projection source device, characterized in that: It includes a processor, a memory, a transceiver and a display screen; wherein the memory is used to store computer execution instructions; when the screen projection source device is running, the processor executes the computer execution instructions stored in the memory, so that the screen projection source device executes the method described in any one of claims 1-9.

13. A screen projection destination device, characterized in that: It includes a processor, a memory, a transceiver and a display screen; wherein the memory is used to store computer execution instructions; when the screen projection destination device is running, the processor executes the computer execution instructions stored in the memory, so that the screen projection destination device executes the method described in claim 10 or 11.

14. A screen projection system, characterized in that: It includes the screen projection source device as described in claim 12 and the screen projection destination device as described in claim 13.

15. A computer storage medium, characterized in that The computer storage medium includes computer instructions, and when the computer instructions are executed on a device, the device is caused to perform the method of any one of claims 1-9 or the method of any one of claims 10-11.

Images