Wireless screen projection method, mobile device and computer-readable storage medium

Abstract

The present application relates to a mobile device, a wireless screen projection method and a computer-readable storage medium. The mobile device runs a first application in the foreground, and wirelessly projects the screen to an electronic device in a first wireless screen projection mode. The mobile device includes: a processor; a memory; a computer program stored in the memory. When the computer program is executed by the processor, the mobile device executes: after detecting that the first application is a first type of application, the mobile device automatically outputs a first prompt message, and the first prompt message is used to prompt the first wireless screen projection mode to be switched to a second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode. The mobile device can automatically identify the scene, determine the most suitable screen projection mode for the currently running application in the foreground, automatically output prompt information, or automatically switch the screen projection mode, taking into account the characteristics of different applications and different screen projection modes, and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of screen projection technology, and in particular to a wireless screen projection method, a mobile device, and a computer-readable storage medium. Background Art

[0002] Wireless screen projection allows mobile devices such as mobile phones and tablets to project multimedia content locally or on the Internet to electronic devices such as PCs and smart screens that have audio and video playback capabilities, and play the above multimedia data on electronic devices. Typical wireless screen projection includes mirror projection (such as miracast) and online projection (such as DLNA).

[0003] In mirror screen projection, electronic devices do not need to be connected to the Internet or local area network, but during the whole process, both mobile devices and electronic devices must process the projection data in real time. For example, mobile devices must encode and send projection data in real time, and electronic devices must receive and decode projection data in real time. This results in more participating devices, a longer delay in projection data, and the projection effect is easily affected by mobile devices. In online screen projection, mobile devices only participate in the initial network address transmission and do not participate in subsequent processes. The delay in projection data is smaller, the projection effect is not easily affected by mobile devices, and the projection effect is better, but the electronic device needs to be connected to the Internet or local area network. Summary of the invention

[0004] After long-term research, the inventor found that different applications focus on different requirements, but all use the same projection method in screen projection, which brings a bad experience to users. For example, game applications focus on real-time, have high requirements for low latency, and have relatively low requirements for audio and video synchronization; while video applications (such as Tencent Video) focus on audio and video synchronization, and have relatively low real-time requirements, that is, relatively low requirements for low latency. If the same projection method is used in screen projection, after the user switches to different applications, the same projection method will be suitable for the application before the switch, but not for the application after the switch. For example, after the user projects the mobile phone screen to a smart TV (also called a large screen) through mirror projection, he opens the game application and plays the game with the mobile phone and the large screen. At this time, the latency is low and the user experience is good; afterwards, the user switches the game application to the video application, and the audio and video will be out of sync, and the user experience is not good.

[0005] In order to solve the above technical problems, taking into account the characteristics of mirror screen projection and online screen projection, as well as the requirements of different applications, the present application provides a wireless screen projection method, a mobile device and a computer-readable storage medium, which automatically identifies the category of the current application, automatically gives suggestions, and automatically prompts the user whether to change, or even automatically changes the screen projection method, so that the changed screen projection method is most suitable for the current application, thereby improving the user experience. For example, after the user projects the mobile phone screen to the large screen through mirror screen projection, the user opens the game application and uses the mobile phone and the large screen to play the game. At this time, the scene is automatically identified, and it is judged that the mirror screen projection is the most suitable screen projection method for the game application, and it remains unchanged; afterwards, the user switches the game application to the video application (such as Tencent Video). At this time, the scene is automatically identified, and it is judged that the online screen projection is the most suitable screen projection method for the video application. The user is automatically prompted to change, and even the current mirror screen projection method is automatically changed to the online screen projection method. In this way, the requirements of different applications, as well as the characteristics of mirror screen projection and online screen projection are taken into account, and the user experience is high.

[0006] In the first aspect, the present application provides a mobile device, the mobile device runs a first application in the foreground, the mobile device wirelessly projects the screen to the electronic device in a first wireless projection mode, the mobile device includes: a processor; a memory; and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the mobile device performs the following steps: after detecting that the first application belongs to the first category of applications, the mobile device automatically outputs a first prompt information, the first prompt information is used to prompt the first wireless projection mode to be switched to a second wireless projection mode; or, the mobile device automatically switches the first wireless projection mode to the second wireless projection mode, and wirelessly projects the screen to the electronic device in the second wireless projection mode; or, the mobile device automatically switches the first wireless projection mode to the second wireless projection mode, and wirelessly projects the screen to one or more electronic devices that support the second wireless projection mode detected by the mobile device in the second wireless projection mode. In this way, the mobile device can automatically identify the category of the application running in the foreground, and automatically prompt the user whether to change it, or automatically change the projection mode, so that the changed projection mode is most suitable for the projection of the current application, thereby taking into account the requirements of different applications, as well as the characteristics of mirror projection and online projection, and improving the user experience. In this solution, the specific execution subject is the operating system currently running on the mobile device, or the default system-level application on the mobile device (for example, the system-level application that is started when the mobile device is turned on).

[0007] The third scheme of the first aspect is schematically illustrated by way of example. For example, there are electronic devices 1, 2, and 3 around the mobile device; the mobile device runs a first application, and the mobile device wirelessly projects the screen to the electronic device 1 in a first wireless screen projection mode; after detecting that the first application belongs to the first category of applications, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, the mobile device disconnects the wireless screen projection with the electronic device 1, and wirelessly projects the screen to at least one of the electronic devices 2 and 3 in the second wireless screen projection mode. Alternatively, after detecting that the first application belongs to the first category of applications, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and the mobile device wirelessly projects the screen to at least one of the electronic devices 1, 2, and 3 in the second wireless screen projection mode.

[0008] According to the first aspect, after the mobile device automatically outputs the first prompt information, the mobile device also performs the following steps: detecting a first user input, the first user input is used to switch the first wireless screen projection mode to the second wireless screen projection mode; in response to the first user input, the mobile device automatically outputs the identification of one or more electronic devices, the electronic device is an electronic device that supports the second wireless screen projection mode detected by the mobile device; detecting a second user input, the second user input is used to select an identification of an electronic device from the identifications of the electronic devices; in response to the second user input, the mobile device switches the first wireless screen projection mode to the second wireless screen projection mode, and projects the screen to the selected electronic device in the second wireless screen projection mode. In this way, after the mobile device outputs the prompt information, it provides the user with the opportunity to choose whether to switch the screen projection mode, and to select the switched screen projection mode and the electronic device to receive the screen projection. The user can make a selection based on the prompt.

[0009] According to the first aspect, or any implementation of the first aspect above, the mobile device also performs the following steps: after detecting that the second application belonging to the second category of applications is started, or after detecting that the second application belonging to the second category of applications is switched to the foreground application, the mobile device automatically outputs a second prompt message, and the second prompt message is used to prompt the second wireless screen projection mode to be switched to the first wireless screen projection mode; or, the mobile device automatically switches the second wireless screen projection mode to the first wireless screen projection mode, and wirelessly projects the screen to the electronic device in the first wireless screen projection mode. In this way, after the application category running in the foreground of the mobile device changes again, the mobile device automatically identifies it and automatically prompts the user whether to change it, or automatically changes the screen projection mode, so that the changed screen projection mode is most suitable for the screen projection of the current application, thereby improving the user experience.

[0010] According to the first aspect, or any implementation of the first aspect above, the mobile device also performs the following steps: after detecting that the first application is a third-category application, the first application of the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode; or, the first application of the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to one or more electronic devices that support the second wireless screen projection mode detected by the mobile device in the second wireless screen projection mode. In this way, after the category of the application running in the foreground of the mobile device changes, the mobile device automatically identifies it and automatically prompts the user whether to change it, or automatically changes the projection mode, so that the changed projection mode is most suitable for the projection of the current application, thereby improving the user experience.

[0011] According to the first aspect, or any implementation of the first aspect above, the mobile device also performs the following steps: after detecting that the first application belonging to the first category of applications is switched to the foreground application, or after detecting that the third application belonging to the first category of applications is started, or after detecting that the third application belonging to the first category of applications is switched to the foreground application, the mobile device automatically outputs a third prompt message, and the third prompt message is used to prompt the first wireless screen projection mode to be switched to the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode. In this way, after the application category running in the foreground of the mobile device changes back, the mobile device automatically identifies it and automatically prompts the user whether to change it back, or automatically changes the projection mode back, so that the changed projection mode is most suitable for the projection of the current application, thereby improving the user experience.

[0012] According to the first aspect, or any implementation of the first aspect above, after the mobile device automatically outputs the first prompt information, the mobile device also performs the following steps: detecting a first user input, the first user input is used to switch the first wireless screen projection mode to the second wireless screen projection mode; in response to the first user input, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to one or more electronic devices that support the second wireless screen projection mode detected by the mobile device in the second wireless screen projection mode. In this way, another solution for changing the screen projection mode is provided, which can also improve the user experience.

[0013] According to the first aspect, or any implementation of the first aspect above, after detecting that the first application belongs to the first category of applications, the mobile device also performs the following steps: after detecting that the mobile device plays a network video through the first application, the mobile device automatically outputs a first prompt message, and the first prompt message is used to prompt the first wireless screen projection mode to be switched to the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to one or more electronic devices that support the second wireless screen projection mode detected by the mobile device in the second wireless screen projection mode. In this way, the mobile device also automatically prompts the user whether to change, or automatically changes the projection mode, so that the changed projection mode is most suitable for the projection of the current application, thereby taking into account the requirements of different applications, as well as the characteristics of mirror projection and online projection, and improving the user experience.

[0014] According to the first aspect, or any implementation of the first aspect above, the first category of applications is non-customized video applications, the second category of applications is game applications, and the third category of applications is customized video applications; the first wireless screen projection method is a mirror screen projection method, and the second wireless screen projection method is an online screen projection method; the one or more electronic devices include the electronic device, or the one or more electronic devices do not include the electronic device; the input form of the first user input and the second user input includes touch input and voice input. In this way, there is no need to improve the first application that provides video services, and it can be ensured that the mobile device prompts the user to change when running the first application, and even automatically changes the current mirror screen projection method to the online screen projection method, which improves the screen projection efficiency, and can ensure the image quality of the electronic device playing the network video, so as to ensure that the user gets the best screen projection experience, and can also reduce costs. In addition, after the mobile device changes the screen projection method, it can continue to project the screen to the original electronic device, or it can change to other electronic devices for projection, to meet the different needs of users.

[0015] According to the first aspect, or any implementation of the first aspect above, the mirror screen projection method may be miracast developed by the Wi-Fi Alliance, and the online screen projection method may be DLNA.

[0016] According to the first aspect, or any implementation of the first aspect above, the mobile device stores a whitelist, the whitelist is used to identify which applications belong to the first category of applications, and the whitelist includes one or more applications of the first category of applications. The whitelist is preset and can be updated.

[0017] According to the first aspect, or any implementation of the first aspect above, the first prompt information includes but is not limited to: interface elements displayed on the display screen, audio played, flashing indicator light, motor vibration, etc.

[0018] In a second aspect, a mobile device is provided. The mobile device wirelessly projects a screen to an electronic device in a first wireless projection mode, and the mobile device includes: a processor; a memory; and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the mobile device performs the following steps: after detecting that a first application is started and the first application belongs to a first category of applications, the mobile device automatically outputs a first prompt message, and the first prompt message is used to prompt that the first wireless projection mode is switched to a second wireless projection mode; or the mobile device automatically switches the first wireless projection mode to the second wireless projection mode, and wirelessly projects the screen to the electronic device in the second wireless projection mode.

[0019] In a third aspect, a wireless screen projection method is provided, which is applied to a mobile device, the mobile device includes a processor and a memory, the mobile device runs a first application in the foreground, and the mobile device wirelessly projects the screen to the electronic device in a first wireless screen projection mode. The method includes: after detecting that the first application belongs to the first category of applications, the mobile device automatically outputs a first prompt message, the first prompt message is used to prompt to switch the first wireless screen projection mode to a second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to one or more electronic devices that support the second wireless screen projection mode detected by the mobile device in the second wireless screen projection mode.

[0020] According to the third aspect, after the mobile device automatically outputs the first prompt information, the wireless screen projection method also includes: detecting a first user input, the first user input is used to switch the first wireless screen projection mode to a second wireless screen projection mode; in response to the first user input, the mobile device automatically outputs the identification of one or more electronic devices, the electronic device is an electronic device that supports the second wireless screen projection mode detected by the mobile device; detecting a second user input, the second user input is used to select an identification of an electronic device from the identifications of the electronic devices; in response to the second user input, the mobile device switches the first wireless screen projection mode to the second wireless screen projection mode, and projects the screen to the selected electronic device in the second wireless screen projection mode. In this way, after the mobile device outputs the prompt information, it provides the user with the opportunity to choose whether to switch the screen projection mode, and to select the switched screen projection mode and the electronic device to receive the screen projection. The user can make a selection based on the prompt.

[0021] According to the third aspect, or any implementation of the third aspect above, the wireless screen projection method also includes: after detecting that the second application belonging to the second category of applications is started, or after detecting that the second application belonging to the second category of applications is switched to the foreground application, the mobile device automatically outputs a second prompt message, and the second prompt message is used to prompt the second wireless screen projection mode to be switched to the first wireless screen projection mode; or, the mobile device automatically switches the second wireless screen projection mode to the first wireless screen projection mode, and wirelessly projects the screen to the electronic device in the first wireless screen projection mode. In this way, after the application category running in the foreground of the mobile device changes again, the mobile device automatically identifies it and automatically prompts the user whether to change it, or automatically changes the screen projection mode, so that the changed screen projection mode is most suitable for the screen projection of the current application, thereby improving the user experience.

[0022] According to the third aspect, or any implementation of the third aspect above, the wireless screen projection method also includes: after detecting that the first application is a third-category application, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to one or more electronic devices that support the second wireless screen projection mode detected by the mobile device in the second wireless screen projection mode. In this way, after the category of the application running in the foreground of the mobile device changes, the mobile device automatically identifies it and automatically prompts the user whether to change it, or automatically changes the screen projection mode, so that the changed screen projection mode is most suitable for the screen projection of the current application, thereby improving the user experience.

[0023] According to the third aspect, or any implementation of the third aspect above, the wireless screen projection method also includes: after detecting that the first application belonging to the first category of applications is switched to the application running in the foreground, or, after detecting that the third application belonging to the first category of applications is started, or, after detecting that the third application belonging to the first category of applications is switched to the application running in the foreground, the mobile device automatically outputs a third prompt message, and the third prompt message is used to prompt the first wireless screen projection mode to be switched to the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode. In this way, after the application category running in the foreground of the mobile device changes back, the mobile device automatically identifies it and automatically prompts the user whether to change it back, or automatically changes the projection mode back, so that the changed projection mode is most suitable for the projection of the current application, thereby improving the user experience.

[0024] According to the third aspect, or any implementation of the third aspect above, after the mobile device automatically outputs the first prompt information, the wireless screen projection method also includes: detecting a first user input, the first user input is used to switch the first wireless screen projection mode to a second wireless screen projection mode; in response to the first user input, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to one or more electronic devices that support the second wireless screen projection mode detected by the mobile device in the second wireless screen projection mode. In this way, another solution for changing the screen projection mode is provided, which can also improve the user experience.

[0025] According to the third aspect, or any implementation of the third aspect above, after detecting that the first application belongs to the first category of applications, the wireless screen projection method also includes: after detecting that the mobile device plays a network video through the first application, the mobile device automatically outputs a first prompt information, and the first prompt information is used to prompt the first wireless screen projection mode to be switched to the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to one or more electronic devices that support the second wireless screen projection mode detected by the mobile device in the second wireless screen projection mode. In this way, the mobile device also automatically prompts the user whether to change, or automatically changes the projection mode, so that the changed projection mode is most suitable for the projection of the current application, thereby taking into account the requirements of different applications, as well as the characteristics of mirror projection and online projection, and improving the user experience.

[0026] According to the third aspect, or any implementation of the third aspect above, the first category of applications is non-customized video applications, the second category of applications is game applications, and the third category of applications is customized video applications; the first wireless screen projection method is a mirror screen projection method, and the second wireless screen projection method is an online screen projection method; the one or more electronic devices include the electronic device, or the one or more electronic devices do not include the electronic device; the input form of the first user input and the second user input includes touch input and voice input. In this way, there is no need to improve the first application that provides video services, and it can be ensured that the mobile device prompts the user to change when running the first application, and even automatically changes the current mirror screen projection method to the online screen projection method, which improves the screen projection efficiency, and can ensure the image quality of the electronic device playing the network video, so as to ensure that the user gets the best screen projection experience, and can also reduce costs. In addition, after the mobile device changes the screen projection method, it can continue to project the screen to the original electronic device, or it can change to other electronic devices for projection, to meet the different needs of users.

[0027] According to the third aspect, or any implementation of the third aspect, the mirror screen projection method may be miracast developed by the Wi-Fi Alliance, and the online screen projection method may be DLNA.

[0028] According to the third aspect, or any implementation of the third aspect, the mobile device stores a whitelist, the whitelist is used to identify which applications belong to the first category of applications, and the whitelist includes one or more applications belonging to the first category of applications. The whitelist is pre-set and can be updated. The whitelist can be added, deleted, etc. by the user.

[0029] According to the third aspect, or any implementation of the third aspect above, the first prompt information includes but is not limited to: interface elements displayed on the display screen, audio played, flashing indicator light, motor vibration, etc.

[0030] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0031] In a fourth aspect, a wireless screen projection method is provided. The wireless screen projection method is applied to a mobile device, the mobile device includes a processor and a memory, the mobile device wirelessly projects the screen to the electronic device in a first wireless screen projection mode, and the method includes: after detecting that a first application is started and the first application belongs to a first category of applications, the mobile device automatically outputs a first prompt message, the first prompt message is used to prompt the first wireless screen projection mode to be switched to a second wireless screen projection mode; or, the mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode.

[0032] In a fifth aspect, the present application provides a wireless screen projection method. The wireless screen projection method is applied to a mobile device, the mobile device includes a processor and a memory, and the mobile device wirelessly projects the screen to an electronic device in a first wireless screen projection mode; the wireless screen projection method includes: the mobile device runs a first application in the foreground, the mobile device identifies a scene corresponding to the application running in the foreground, and notifies the electronic device of the scene, so that the electronic device uses a playback strategy corresponding to the scene to play the multimedia content.

[0033] In a sixth aspect, an embodiment of the present application provides a screen projection method, which is applied to an electronic device. The method may include: the electronic device receives multimedia content sent by a mobile device in a first wireless screen projection mode; receives a scene recognized by the mobile device and sent by the mobile device; and plays the multimedia content using a playback strategy corresponding to the scene.

[0034] In the seventh aspect, the present application provides a computer-readable storage medium, including a computer program, characterized in that when the computer program is run on a mobile device, the mobile device executes the method of any one of the implementation methods of the third aspect, the fourth aspect, the fifth aspect and the third aspect.

[0035] In an eighth aspect, the present application provides a computer program product. When the computer program product is run on a mobile device, the mobile device executes the method of the third aspect, the fourth aspect, the fifth aspect and any one of the implementation modes of the third aspect.

[0036] The technical solution provided by this application is an improvement from the system side, and no third-party application needs to be adapted. In addition, the technical solution provided by this application automatically selects the optimal screen projection method according to the type of application running in the foreground, and even whether the application running in the foreground plays online videos, etc., or automatically outputs prompt information to allow users to make their own choices, so that the mobile device screen projection effect is the best, the user's screen projection experience is improved, and the cost is reduced and the screen projection efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1A A schematic diagram of the principle of sharing network video based on mirroring screen projection for the provided mobile device and electronic device;

[0038] Figure 1B A schematic diagram of the principle of sharing network video based on online screen projection for the provided mobile device and electronic device;

[0039] Figure 2 A schematic diagram of a scenario of a wireless screen projection method provided in an embodiment of the present application;

[0040] Figure 3A A schematic diagram of the hardware structure of a mobile device provided in an embodiment of the present application;

[0041] Figure 3B A schematic diagram of the software structure of a mobile device provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 5A-Figure 5C A schematic diagram of a flow chart of a wireless screen projection method provided in an embodiment of the present application;

[0044] Figure 6A-6C , Figure 7A-7C , Figure 8A-8H A schematic diagram of a user interface of a mobile device in a wireless screen projection method provided in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of a user interface of an electronic device in a wireless screen projection method provided in an embodiment of the present application;

[0046] Figure 10A-10B A schematic diagram of the interaction of internal modules of a mobile device in a wireless screen projection method provided in an embodiment of the present application;

[0047] Fig.11 A schematic diagram of the interaction of internal modules of a mobile device in a wireless screen projection method provided in an embodiment of the present application;

[0048] Fig.12 A schematic diagram of a flow chart of another wireless screen projection method provided in an embodiment of the present application;

[0049] Figure 13A-13B A schematic diagram of the interaction between internal modules of a mobile device and an electronic device in another wireless screen projection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means or. For example, A / B can mean A or B. "And / or" in the text is just a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0051] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0052] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0053] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on a mobile device or electronic device, and finally presented as content that the user can recognize. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of a mobile device or electronic device.

[0054] In an embodiment of the present application, through wireless screen projection, the content displayed on the mobile device (such as the multimedia content being played) can be displayed on the electronic device (such as continuing to play the multimedia content), and then the mobile device can continue to display the content or stop displaying the content. Wireless screen projection can also be replaced by other terms, such as multi-screen interaction, etc.; this embodiment of the present application is not limited. Among them, the mobile device can also be referred to as the output end or source end (source end), and the electronic device can also be referred to as the input end or receiving end (sink end).

[0055] After wireless screen projection, the multimedia content displayed or played by the electronic device may include any one or more of the following: video, text, picture, photo, audio or table, etc. For example, the multimedia content may be a movie, TV series, short video, musical, etc.

[0056] Multimedia content can be network multimedia content, local multimedia content, or a combination of network multimedia content and local multimedia content. Network multimedia content refers to multimedia content obtained by a mobile device from the network. For example, when a mobile device runs a video application, it obtains a video from a server that provides audio and video services. Local multimedia content refers to multimedia content stored or generated locally on a mobile device. For example, pictures or tables stored locally on a mobile device.

[0057] First, two screen projection methods involved in the embodiments of the present application are introduced: mirroring technology and online screen projection.

[0058] 1. Mirror screen projection: A mobile device (such as a smart phone, tablet computer, etc.) establishes a communication connection with an electronic device (such as a smart TV, smart screen, etc.). The mobile device obtains multimedia content from the network side or the local side, and then encodes the multimedia content and transmits it to the electronic device by wireless transmission such as point-to-point transmission; the electronic device outputs (such as display, playback, etc.) after decoding.

[0059] The multimedia content may be network multimedia content, local multimedia content or a combination of the two. The point-to-point transmission method between the mobile device and the electronic device may include but is not limited to: wireless fidelity direct (Wi-Fi direct) (also known as wireless fidelity peer-to-peer (Wi-Fi P2P)) communication connection, Bluetooth communication connection, near field communication (NFC) connection, etc.

[0060] Mirror screen projection may include miracast developed by the Wi-Fi Alliance, private mirror screen projection solutions developed by various companies, such as Huawei's cast+ and Apple's AirPlay. Among them, miracast is based on the basic technical standards developed by the Wireless Fidelity Wi-Fi Alliance and the real-time streaming protocol (RTSP). Wi-Fi basic technical standards may include wireless transmission technology 802.11n, 802.11ac, Wi-Fi direct / Wi-Fi P2P, tunneled direct link setup (TDLS), WPA2 (Wi-Fi protected access 2) encryption for management security, and WMM (Wi-Fi multimedia) technology that provides quality of service and traffic management.

[0061] Combine the following Figure 1A , further elaborating on mirror screen projection. Figure 1A The example shows a scenario where a mobile device and an electronic device share a network video based on mirroring. Figure 1AAs shown, the mobile device establishes a communication connection (such as a Wi-Fi P2P connection) with the electronic device. At the same time, the mobile device accesses the routing device, and obtains streaming media from the server through the routing device. Among them, the mobile device accesses the routing device, which can be specifically an access point (AP) provided by the mobile device to the routing device. Afterwards, the mobile device obtains the screen recording content and the audio recording content through screen recording, audio recording, etc. during the process of playing the streaming media itself, and then encodes the screen recording content and the audio recording content in real time, and sends them to the electronic device through a connection such as Wi-Fi P2P; after the electronic device receives it, it plays and displays it in real time.

[0062] from Figure 1A It can be seen that when sharing network videos based on mirroring, the network videos received by the electronic device have undergone multiple network transmissions and codec conversions, which is a cumbersome process. In addition, the image quality (such as resolution) of the network video played by the electronic device is limited by the mobile device, and the playback effect may be poor, affecting the user experience. However, the audio and video synchronization effect of mirroring is better.

[0063] In some embodiments, mirroring screen projection may also be referred to as full sharing screen projection, wireless display, etc.

[0064] 2. Online screen projection: Both the mobile device and the electronic device are connected to the Internet or a local area network. The mobile device only sends the network address corresponding to the multimedia resource to be projected, such as a uniform resource locator (URL), to the electronic device; the electronic device obtains the corresponding multimedia content from the Internet or the local area network according to the network address, and then outputs it (such as playing or displaying it).

[0065] The multimedia content may be network multimedia content, local multimedia content, or a combination of the two. Here, the electronic device may access the same local area network formed by a wireless Wi-Fi access point AP as the mobile device, or may connect to different networks with the mobile device, such as different local area networks formed by different APs, which are interconnected via the Internet.

[0066] Online screen casting can include DLNA, as well as private online screen casting solutions developed by various companies, such as Google's Google cast and Apple's AirPlay. Among them, DLNA is built on the universal plug and play (UPnP) protocol.

[0067] Combine the following Figure 1B , further elaborating on online screen projection. Figure 1BThe example shows a scenario where a mobile device and an electronic device share a network video based on online screen projection. Figure 1B As shown, the mobile device and the electronic device are jointly connected to the local area network formed by the Wi-Fi AP. The mobile device sends the URL of the played network video to the electronic device through the AP, and then the electronic device obtains the multimedia content according to the URL. The URL is the address of the server that provides network audio and video services. When sharing network videos based on online screen projection, electronic devices can directly obtain multimedia content from the network side without multiple network transmissions and codec conversions of network videos. The process is simple, and the electronic device can have a better playback effect when playing network videos, and the user experience is better. However, compared with mirrored screen projection, the audio and video synchronization of online screen projection is relatively poor.

[0068] In some embodiments, online screen projection may also be referred to as network screen projection, etc.

[0069] In the embodiment of the present application, the mirror screen projection mode can be referred to as the first wireless screen projection mode, and the online screen projection mode can be referred to as the second wireless screen projection mode.

[0070] Figure 2 A schematic diagram of a scenario of a wireless screen projection method provided in an embodiment of the present application. Figure 2 As shown, the mobile device 100 can project multimedia content to the electronic device 200 through mirroring screen projection, and can also project multimedia content to the electronic device 200 through online screen projection.

[0071] When the mobile device 100 projects multimedia content to the electronic device 200 through mirroring, a Wi-Fi P2P connection can be established between the mobile device 100 and the electronic device 200; other short-range communication direct connections can also be established, such as Bluetooth, ZigBee, etc. In mirroring, the multimedia content projected by the mobile device 100 can come from itself; it can also come from the server 400, in which case the mobile device 100 is connected to the server 400 via the Wi-Fi access point 300.

[0072] When the mobile device 100 delivers multimedia content to the electronic device 200 through online screen projection, both the mobile device 100 and the electronic device 200 can access the Wi-Fi access point 300, and thus are in the same local area network. In some other embodiments, the mobile device 100 and the electronic device 200 can also access different networks, which is not limited in the embodiments of the present application. It should be emphasized that, unless otherwise specified, in the following embodiments of the present application, the mobile device 100 and the electronic device 200 both access the same local area network through the Wi-Fi access point 300. In online screen projection, the multimedia content delivered by the mobile device 100 comes from the server 400.

[0073] The server 400 provides network audio and video services. Exemplarily, the server 400 may be a server storing a variety of multimedia content. For example, the server 400 may be a Tencent video server providing audio and video services. The number of servers 400 may be one or more.

[0074] In some embodiments, when the mobile device 100 shares a network video with the electronic device 200 through mirroring, the mobile device 100 can automatically or under the trigger of the user switch the mirroring to online projection, and continue to share the network video with the electronic device 200 through online projection. Exemplarily, the network video can be obtained from the server 400 when the mobile device 100 runs a video application (APP).

[0075] In some embodiments, after the mobile device 100 switches the mirror screen projection to the online screen projection, in some cases, the mobile device 100 can also switch the online screen projection back to the mirror screen projection.

[0076] The mobile devices of the present application include but are not limited to smart phones, tablet computers, personal digital assistants (PDAs), wearable electronic devices with wireless communication functions (such as smart watches and smart glasses), etc. The mobile device may be a portable electronic device running Linux or other operating systems. The mobile device may also be other portable electronic devices, such as a laptop computer, etc. It should also be understood that in some other embodiments, the mobile device may not be a portable electronic device, but a desktop computer.

[0077] For example, Figure 3A FIG. 1 shows the hardware structure of the mobile device 100 provided in the embodiment of the present application. Figure 3AAs shown, the mobile device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0078] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile device 100. In other embodiments of the present application, the mobile device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0079] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0080] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0081] The wireless communication function of the mobile device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0082] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0083] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the mobile device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0084] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), NFC, infrared technology (IR), etc., which are applied to the mobile device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0085] In some embodiments, the antenna 1 of the mobile device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the mobile device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0086] The wireless communication module 160 can be used to establish a communication connection (such as a Wi-Fi direct communication connection, a Bluetooth communication connection, etc.) with the electronic device 200, and based on the communication connection, encode the data collected by the mobile device 100 through screen recording and audio recording and send it to the electronic device 200. That is, the wireless communication module 160 can support the mobile device 100 and the electronic device 200 to share multimedia content based on mirroring (such as miracast).

[0087] The wireless communication module 160 can also access the local area network or other networks formed by the Wi-Fi access point 300, and can send the URL of the currently playing multimedia content to the electronic device 200 through the network. Afterwards, the electronic device 200 can directly obtain the multimedia content through the URL. That is, the wireless communication module 160 can support the mobile device 100 and the electronic device 200 to share multimedia content based on online screen projection (such as DLNA).

[0088] In some embodiments of the present application, the processor 110 is used to identify the current scene when the mobile device 100 shares multimedia content with the electronic device 200 based on mirroring and projection, and notify the electronic device 200 of the scene through the wireless communication module 160, so that the electronic device 200 adaptively selects the corresponding playback strategy to play the multimedia content according to the scene. The way in which the processor 110 identifies the current scene and the way in which the wireless communication module 160 notifies the electronic device 200 of the current scene can be referred to the relevant description of the subsequent method embodiments, which will not be repeated here.

[0089] In some other embodiments of the present application, the processor 110 is further used to automatically or under the trigger of the user instruct the wireless communication module 160 to switch the mirror screen projection to online screen projection during the process of the mobile device 100 sharing the network video with the electronic device 200 based on the mirror screen projection. The specific implementation of the wireless communication module 160 switching the mirror screen projection to the online screen projection can refer to the relevant description of the subsequent method embodiments, which will not be repeated here.

[0090] In some embodiments, after the wireless communication module 160 switches the mirror screen projection to the online screen projection, in some cases, the processor 110 may also be used to instruct the wireless communication module 160 to switch the online screen projection back to the mirror screen projection. For the case where the wireless communication module 160 switches the online screen projection back to the mirror screen projection, reference may be made to the relevant description of the subsequent method embodiments.

[0091] The mobile device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0092] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an automatic matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0093] In the embodiment of the present application, the display screen 194 is used to display the user interface mentioned in the embodiment of the present application implemented on the mobile device 100. The specific implementation of the user interface can refer to the relevant description of the subsequent method embodiment.

[0094] Video codecs are used to compress or decompress digital videos. The mobile device 100 may support one or more video codecs. Thus, the mobile device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0095] The mobile device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0096] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0097] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the mobile device 100, which is different from the position of the display screen 194.

[0098] The software system of the mobile device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the software structure of the mobile device 100. Figure 3B 1 is a schematic block diagram of a software structure of a mobile device 100 provided in an embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, namely, from top to bottom, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0099] like Figure 3B As shown, the application package may include screen projection services, video applications, game applications, office applications, cameras, galleries, calendars, calls, maps, navigation, WLAN, Bluetooth, music, short messages and other applications. For the sake of simplicity, the video application will be referred to as the video application, and the game application will be referred to as the game application.

[0100] The screen projection service provides a mirroring screen projection function of the mobile device 100. The screen projection service supports the mobile device 100 to share multimedia content with the electronic device 200 based on the mirroring screen projection. The screen projection service can call the wireless communication module 160 of the mobile device 100 to provide the mirroring screen projection function.

[0101] The video application program may be referred to as a video application, and is used to provide audio and video services for the mobile device 100. The mobile device 100 may run the video application and obtain network videos from the server corresponding to the video application. The number of video applications may be one or more. For example, the video application may include Tencent Video.

[0102] The video application can provide an online screen projection function. The video application supports the mobile device 100 to share multimedia content with the electronic device 200 through online screen projection. Specifically, when the mobile device 100 is running the video application and playing the online video therein, if the user turns on the online screen projection function of the video application, the mobile device 100 can send the URL of the online video to the electronic device 200.

[0103] The game application program may be referred to as a game application, and is used to provide game services for the mobile device 100. The mobile device 100 may run the game application and obtain game resources locally or from a server corresponding to the game application. The number of game applications may be one or more.

[0104] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0105] like Figure 3B As shown, the application framework layer may include a scene awareness module, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0106] Specifically, the scene perception module is used to listen to the operation of the currently used application, and identify the application currently used by the mobile device 100 based on this, so as to determine the scene in which the mobile device 100 is located. The scene perception module is an optional module. In some embodiments, the function of the scene perception module can be integrated into the screen projection service of the application layer.

[0107] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0108] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0109] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0110] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0111] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0112] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0113] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0114] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0115] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0116] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0117] A 2D graphics engine is a drawing engine for 2D drawings.

[0118] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0119] It should be emphasized that Figure 3B This is only an illustrative example; the software structure of the mobile device 100 provided in the embodiment of the present application may also adopt other software architectures, such as Software architecture of Linux or other operating systems.

[0120] The electronic devices of the embodiments of the present application include but are not limited to tablet computers, desktop computers, portable electronic devices (such as laptop computers, Laptop), smart TVs (such as smart screens), car computers, smart speakers, augmented reality (AR) devices, virtual reality (VR) devices, electronic billboards with display screens, projectors used alone (for example, projected on a wall) or in combination with a display device (such as a screen), other smart devices with display screens, and other smart devices with speakers. Exemplary embodiments of electronic devices include but are not limited to those equipped with In some embodiments, the electronic device 200 may be a television set equipped with a TV box, the TV box is used to receive multimedia content from the mobile device 100 or the server 400 and provide a screen projection function, and the television set only provides a display function. In some embodiments, the electronic device 200 may also be used in conjunction with a remote controller. The remote controller and the electronic device 200 may communicate via infrared signals.

[0121] For example, Figure 4 2 shows the hardware structure of the electronic device 200 provided in the embodiment of the present application. Figure 4 As shown, the electronic device 200 may include: a video codec 221, a processor 222, a memory 223, a wireless communication processing module 224, a power switch 225, a wired LAN communication processing module 226, a high definition multimedia interface (HDMI) communication processing module 227, a USB communication processing module 228, a display screen 229, and an audio module 230. Each module may be connected via a bus. Among them:

[0122] The processor 222 can be used to read and execute computer-readable instructions. In a specific implementation, the processor 222 may mainly include a controller, an arithmetic unit, and a register. Among them, the controller is mainly responsible for decoding instructions and issuing control signals for operations corresponding to the instructions. The arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, etc., and can also perform address operations and conversions. The register is mainly responsible for storing register operands and intermediate operation results temporarily stored during the execution of instructions. In a specific implementation, the hardware architecture of the processor 222 can be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, or an NP architecture, etc.

[0123] The wireless communication processing module 224 may include a WLAN communication processing module 224A, and may also include a Bluetooth (BT) communication processing module 224B, an NFC processing module 224C, a cellular mobile communication processing module (not shown), and the like.

[0124] In some embodiments, the wireless communication processing module 224 may be used to establish a communication connection with the mobile device 100, and receive the encoded data sent by the mobile device 100 based on the communication connection. For example, the WLAN communication processing module 224A may be used to establish a Wi-Fi direct communication connection with the mobile device 100, the Bluetooth (BT) communication processing module 224B may be used to establish a Bluetooth communication connection with the mobile device 200, and the NFC processing module 224C may be used to establish an NFC connection with the mobile device 100, etc. That is, the wireless communication processing module 224 may support the mobile device 100 and the mobile device 200 to share multimedia content through mirroring (such as miracast).

[0125] In one embodiment, the wireless communication processing module 224 can monitor the signals transmitted by the mobile device 100, such as detection requests and scanning signals, discover the mobile device 100, and establish a communication connection with the mobile device 100. In another embodiment, the wireless communication processing module 224 can also transmit signals, such as detection requests and scanning signals, so that the electronic device 200 can discover the mobile device 100 and establish a communication connection (such as a Wi-Fi P2P connection) with the mobile device 100.

[0126] In some embodiments, when the mobile device 100 and the electronic device 200 share multimedia content through mirroring (such as miracast), the wireless communication processing module 224 (such as the WLAN communication processing module 224A) can also receive the scene notified by the mobile device 100. The processor 222 can parse and learn the scene, and adaptively select a playback strategy corresponding to the scene, and use the playback strategy to call the display screen 229, audio module 230 and other modules to play the multimedia content sent by the mobile device 100.

[0127] In some embodiments, the wireless communication processing module 224 (such as the WLAN communication processing module 224A) can also access the local area network or other network formed by the Wi-Fi access point 300, and receive the URL of the network video sent by the mobile device 100 through the Wi-Fi access point 300, and then directly obtain the network video from the server corresponding to the URL. That is, the WLAN communication processing module 224A can support the mobile device 100 and the electronic device 200 to share the network video through online screen projection (such as DLNA).

[0128] The video codec 221 is used to compress or decompress digital video. In the embodiment of the present application, the video codec 221 can decompress multimedia content from the mobile device 100 or the server 400. The electronic device 200 can support one or more video codecs and can play videos in one or more encoding formats. For example: MPEG1, MPEG2, MPEG3, MPEG4, etc.

[0129] The processor 222 can be used to parse the signal received by the wireless communication processing module 224, such as the detection request broadcast by the electronic device 200. The processor 222 can be used to perform corresponding processing operations according to the analysis results, such as generating a detection response, etc. The processor 222 can be used to drive the display screen 229 to perform display according to the decompression result of the video codec 221.

[0130] The memory 223 is coupled to the processor 222 and is used to store various software programs and / or multiple sets of instructions. In a specific implementation, the memory 223 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 223 may store an operating system, such as an embedded operating system such as uCOS, VxWorks, RTLinux, Harmony, Android, etc. The memory 223 may also store a communication program, which may be used to communicate with the electronic device 200, one or more servers, or an additional device.

[0131] The power switch 225 may be used to control the supply of power to the electronic device 200 .

[0132] The wired LAN communication processing module 226 may be used to communicate with other devices in the same LAN via the wired LAN, and may also be used to connect to a WAN via the wired LAN and communicate with devices in the WAN.

[0133] The HDMI communication processing module 227 may be used to communicate with other devices via an HDMI interface (not shown).

[0134] The USB communication processing module 228 may be used to communicate with other devices via a USB interface (not shown).

[0135] The display screen 229 can be used to display images, videos, etc. The display screen 229 can be LCD, OLED, AMOLED, FLED, QLED, etc. The content displayed on the display screen 229 can refer to the relevant description of the subsequent method embodiment.

[0136] The audio module 230 can be used to output audio signals through the audio output interface, so that the electronic device 200 can support audio playback. The audio module 230 can also be used to receive audio data through the audio input interface. The audio module 230 may include but is not limited to: a microphone, a speaker, a receiver, etc.

[0137] In some embodiments, the electronic device 200 may further include a serial interface such as an RS-232 interface, which can be connected to other devices, such as an audio player such as a speaker, so that the display and the audio player cooperate to play audio and video.

[0138] It is understandable that Figure 4 The illustrated structure does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0139] The software system of the electronic device 200 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Exemplarily, the software system of the electronic device 200 includes but is not limited to Linux or other operating systems. It is Huawei's Hongmeng system.

[0140] The application layer of the software system of the electronic device 200 may include a screen projection service and a screen projection player. The screen projection service supports the electronic device 200 to receive multimedia content projected by the mobile device 100 through mirror projection and online projection. Specifically, the screen projection service can call the wireless communication processing module 224 to provide the mirror projection function and the online projection function. The screen projection player is used to play multimedia content from the mobile device 100 or the server 400.

[0141] In some embodiments, when multimedia content is shared between the electronic device 200 and the mobile device 100 through mirroring, the screen projection service can instruct the screen projection player to play the multimedia content according to the corresponding playback strategy based on the current scene of the mobile device 100.

[0142] The wireless screen projection method provided in the embodiment of the present application is described in detail below. In the embodiment of the present application, based on whether there is a customized online screen projection method, the video application can be divided into: customized video applications and non-customized video applications. Among them, customized video applications refer to applications in which the application itself has the ability to initiate screen projection. Exemplarily, the customized video application itself integrates a software development kit (SDK). For example, Tencent Video. The screen projection status of the mobile device 100 may include: whether the mobile device 100 is currently sharing multimedia content with the electronic device 200 through mirror projection. Non-customized video applications refer to applications in which the application itself does not have the ability to initiate screen projection. Exemplarily, the non-customized video application itself does not have an SDK integrated.

[0143] There is no video application configured with the SDK that can query the current screen projection status of the current device.

[0144] As explained above, the inventors have found through long-term research that different applications focus on different requirements. For example, game applications focus on real-time, have high requirements for low latency, and have relatively low requirements for audio and video synchronization; while video applications focus on audio and video synchronization, and have relatively low real-time requirements, that is, relatively low requirements for low latency. Therefore, the wireless screen projection method provided in the embodiment of the present application can automatically identify the category of the application currently running in the foreground, and prompt the user to select the recommended wireless screen projection method, or automatically change to the appropriate wireless screen projection method.

[0145] Figure 5A A flowchart of a wireless screen projection method provided in an embodiment of the present application is shown below. Figure 5A As shown, the method may include:

[0146] S001-S007, the mobile device 100 shares multimedia content with the electronic device 200 through screen mirroring.

[0147] S001, the mobile device 100 detects a user operation to enable a mirroring screen function.

[0148] Fig. 6A and Figure 6B The example shows a user operation of starting the mirroring screen function detected by the mobile device 100. Fig. 6AAn exemplary user interface 61 for displaying installed applications on a mobile device 100 is shown. The user interface 61 displays: a status bar, a calendar indicator, a weather indicator, a tray with commonly used application icons, a navigation bar, an icon 601 for a video application, an icon 602 for a game application, and icons for other applications. Among them, the status bar may include: one or more signal strength indicators of a mobile communication signal (also referred to as a cellular signal), an operator name (such as "China Mobile"), one or more signal strength indicators of a Wi-Fi signal, a battery status indicator, a time indicator, etc. The navigation bar may include system navigation keys such as a back key, a home screen key, and a multitasking key. In some embodiments, Fig. 6A The user interface 61 exemplarily shown may be a home screen.

[0149] like Fig. 6A and Figure 6B As shown, when the mobile device 100 detects a downward sliding gesture on the display screen, in response to the sliding gesture, the mobile device 100 displays a window 603 on the user interface 61. Figure 6B As shown, a control 603a may be displayed in window 603, and control 603a may receive an operation (such as a touch operation, a click operation) to turn on / off the mirror screen projection function of mobile device 100. The expression form of control 603a may include an icon and / or text (such as the text "mirror screen projection", "wireless screen projection", "multi-screen interaction", etc.). Window 603 may also display switch controls for other functions such as Wi-Fi, Bluetooth, flashlight, etc. Figure 6B As shown, the mobile device 100 can detect a user operation on the control 603a, that is, detect a user operation to turn on the mirroring screen function. In some embodiments, after the mobile device 100 detects a user operation on the control 603a, the display form of the control 603a can be changed, such as adding a shadow when displaying the control 603a.

[0150] Not limited to Fig. 6A On the main interface shown, the user can also input a downward sliding gesture on other interfaces to trigger the mobile device 100 to display a window.

[0151] Not limited to Fig. 6A and Figure 6B The user operation shown is performed on the control 603a in the window 603. In the embodiment of the present application, the user operation of enabling the mirror screen projection function can also be implemented in other forms, which is not limited by the embodiment of the present application.

[0152] For example, the mobile device 100 can also display a settings interface provided by a settings application, which may include a control provided to the user for turning on / off the mirroring screen projection function of the mobile device 100. The user can turn on the mirroring screen projection function of the mobile device 100 by inputting user operations on the control.

[0153] For another example, the user may also place the mobile device 100 close to the NFC tag of the electronic device 200 to trigger the mobile device 100 to enable the mirroring screen function.

[0154] S002, the mobile device 100 discovers nearby electronic devices.

[0155] Upon detecting the user operation of turning on the mirror screen projection function in S001, the mobile device 100 turns on one or more of Wi-Fi Direct (not shown), Bluetooth, or NFC in the wireless communication module 160, and discovers electronic devices that can be projected near the mobile device 100 through one or more of Wi-Fi Direct, Bluetooth, and NFC. For example, the mobile device 100 can discover nearby electronic devices 200 and other electronic devices through Wi-Fi Direct.

[0156] S003, the mobile device 100 displays the identifications of the discovered nearby electronic devices.

[0157] In addition to displaying the identification of the electronic device discovered by the mobile device 100 that is acceptable for mirroring, the mobile device 100 may also display other information, such as an image of the discovered electronic device, etc., which is not limited in the embodiments of the present application.

[0158] Then, illustratively, Figure 6C As shown, a window 605 pops up on the mobile device. The window 605 includes: an interface indicator 605a, an icon 605b, an image 605c of one or more electronic devices, and a logo 605d.

[0159] The embodiment of the present application does not limit the order of S002 and S003, and the two can be executed simultaneously or sequentially. When the mobile device 100 has not found any nearby electronic devices, the number of electronic devices displayed in the window 605 is 0.

[0160] S004 , the mobile device 100 detects a user operation of selecting the electronic device 200 .

[0161] For example, Figure 6C As shown, the user operation of selecting the electronic device 200 may be a user operation acting on the image 605c and / or the logo 605d corresponding to the electronic device 200. The user operation of selecting the electronic device 200 may also be implemented in other forms, which are not limited in the embodiment of the present application.

[0162] S005 , in response to the detected user operation of selecting the electronic device 200 , the mobile device 100 and the electronic device 200 establish a communication connection.

[0163] In response to the user operation, the mobile device 100 can establish a communication connection with the electronic device 200 through one or more wireless communication technologies such as Wi-Fi Direct, Bluetooth, and NFC. For example, the mobile device 100 and the electronic device 200 establish a Wi-Fi Direct communication connection. After the mobile device 100 and the electronic device 200 establish a communication connection, they can perform capability negotiation based on the communication connection, including the encoding format, resolution, audio format, etc. supported by both parties, to facilitate the transmission of multimedia content that is subsequently executed.

[0164] S006 , the mobile device 100 sends the currently displayed multimedia content to the electronic device 200 based on the communication connection between the mobile device 100 and the electronic device 200 .

[0165] Specifically, the mobile device 100 can obtain the currently displayed multimedia content (including images and / or audio) by recording the screen, recording the audio, etc., and then compress the obtained multimedia content and send it to the electronic device 200 through the communication connection between the mobile device 100 and the electronic device 200. Taking the sharing of multimedia content based on miracast between the mobile device 100 and the electronic device 200 as an example, the mobile device 100 can obtain the image displayed on the display screen by recording the screen according to the provisions of the miracast protocol, and compress the image using the H.264 encoding algorithm; collect the audio played by the mobile device 100, and compress the audio using the advanced audio coding (AAC) algorithm; then encapsulate the compressed audio data and image data into a transport stream (TS), and then encode the TS stream according to the real-time transport protocol (RTP) and send the encoded data to the electronic device 200 through a Wi-Fi direct connection. That is, the multimedia content is transmitted in a streaming media manner.

[0166] S007, the electronic device 200 plays the received multimedia content.

[0167] After the electronic device 200 receives the multimedia content sent by the mobile device 100 based on the communication connection, it can perform decoding processing on the multimedia content to obtain the multimedia content. Taking the mobile device 100 and the electronic device 200 sharing multimedia content based on miracast as an example, the electronic device 200 can receive the RTP-encoded TS stream based on the Wi-Fi direct communication connection between the mobile device 100, and can perform RTP decoding, TS stream decapsulation, audio and video quality processing / delay synchronization processing on it in sequence, and finally output audio and video, that is, play the multimedia content. It is understandable that S006 and S007 will continue to be executed until the mobile device 100 turns off the mirror screen projection function, turns on the online screen projection function, etc.

[0168] During the execution of S007 by the electronic device 200, the multimedia content played by the electronic device 200 is the same as the multimedia content played by the mobile device 100. When the multimedia content played by the mobile device 100 changes with the user operation, the multimedia content played by the electronic device 200 also changes accordingly.

[0169] It is understandable that the process of sharing multimedia content based on mirroring screen projection by the mobile device 100 and the electronic device 200 shown in S001-S007 is only an example; other implementations are also possible and will not be described in detail here.

[0170] S008-S012, the mobile device 100 prompts the user to select or automatically selects an appropriate screen projection method according to the category of the currently launched application.

[0171] S008, the mobile device 100 starts an application, or switches an application.

[0172] The application launched by the mobile device 100 depends on the user, and may be a game application or a video application. The video application may include a non-customized video application and a non-customized video application.

[0173] The mobile device 100 may respond to Fig. 6A The user operation (such as a click operation, a touch operation, etc.) detected on the icon of the application in the user interface 61 shown in the figure starts the application corresponding to the icon. The corresponding application can also be started in response to other user operations (such as voice commands), which are not limited here. For example, Figure 7A-7B As shown, the mobile device 100 can launch the video application in response to a user operation on an icon 601 of the video application on the main interface. Figure 7B The user interface 71 displayed after the mobile device 100 starts the video application is exemplarily shown. The user interface 71 is the main page provided by the video application. Figure 7BAs shown, one or more video images 701 are displayed in the user interface 71. The video image can be dynamic or static. In addition, the user interface 71 can also display a bottom menu bar, a search box, a sub-channel entrance, etc., which are not limited in this embodiment of the present application. Figure 7B As shown, the mobile device 100 can detect the user operation on the video image 701, obtain the network video indicated by the video image 701 from the server corresponding to the video application through the network, and play the network video. The network video indicated by the video image 701 monitored by the user operation is the network video selected by the user. The following embodiments will be described by taking the server 400 corresponding to the video application as an example. That is to say, the website address of the network video obtained by the mobile device 100 is the address of the server 400.

[0174] Figure 7C The user interface 72 displayed when the mobile device 100 plays a network video selected by the user is shown by way of example. The user interface 72 may be displayed by the mobile device 100 in response to the user switching the mobile device 100 from a portrait mode to a landscape mode, or when the user clicks on a full-screen play control displayed in the lower right corner of the user interface displayed by the mobile device 100. In some embodiments, the user interface 72 may also include an online screen projection switch control 702, and the control 702 is used to monitor user operations (such as click operations, touch operations, etc.) to turn on / off the online screen projection function of the video application. Figure 7C As shown, the display status of the control 702 indicates that the online screen projection function of the current mobile device 100 is turned on.

[0175] S009, the mobile device 100 determines whether the started application, or the switched application, is a video application.

[0176] Specifically, the mobile device 100 can monitor the application program running in the foreground or the application program corresponding to the currently used window, and identify whether the application program running in the foreground of the mobile device 100 is a video application. If it is a video application, execute S010.

[0177] The specific identification after starting and switching applications can be combined with Fig. 10A , Fig. 10B Explain. Fig. 10A , Fig. 10BThe scene perception module can perceive application startup and application switching by calling APIs of the operating system. For example, taking the perception of application switching as an example, the scene perception module perceives the sending of the "android.ActivityState.CHANGE" message to subscribe to the application switching event. After the scene perception module perceives the application switching, it calls the API provided by the operating system to query the top-level APP name seen by the user. For example, the task and the PackageName corresponding to the task are queried through the API provided by ActivityManager. The scene perception module perceives whether the queried application name is in the pre-made database or table to determine whether it is a video application. The database or the table can be added, deleted, updated, etc. by the user as needed.

[0178] S010, the mobile device 100 determines whether the started video application is a non-customized video application.

[0179] In some embodiments, the mobile device 100 may store a whitelist, which may include one or more non-customized video applications. Thus, when an application running in the foreground is in the whitelist, the mobile device 100 may determine that the application running in the foreground of the mobile device 100 is a non-customized video application. The non-customized video application in the whitelist may be installed in the mobile device 100 or may not be installed in the mobile device 100. The mobile device 100 may update the whitelist as needed.

[0180] If it is a non-customized video application, execute S011; if it is a customized video application, execute S012.

[0181] S011, Execution Figure 5B The process;

[0182] S012, Execution Figure 5C process.

[0183] Next, we will introduce Figure 5B process. Figure 5B The process of switching the current mirror screen projection of the mobile device 100 to online screen projection for non-customized video applications is described. Figure 5B In the embodiment of the present invention, when the mobile device 100 shares a network video provided by a non-customized video application with the electronic device 200 through mirroring, the mirroring can be switched to online projection under the triggering of the user, and the network video can be shared based on the online projection. Figure 5B The process shown may include:

[0184] S101, the mobile device 100 outputs prompt information, where the prompt information is used to prompt the user to switch the mirror screen projection to online screen projection.

[0185] In S101, the implementation form of the prompt information may include but is not limited to: interface elements displayed on the display screen of the mobile device 100, audio played, indicator light flashing, motor vibration, etc. The scenarios in which the mobile device 100 outputs prompt information may include the following three scenarios:

[0186] Scenario 1: After the mobile device 100 starts the video application, it outputs prompt information.

[0187] The mobile device 100 may adopt S009, or the method of S009 and S010, and when it is identified that the application program running in the foreground of the mobile device 100 is a non-customized video application, the mobile device 100 outputs prompt information.

[0188] Fig. 8A The prompt information displayed by the mobile device 100 in scene 1 is exemplarily shown. The prompt information is a window 703. The window 703 may include: text 703a. The text 703a may be, for example, "Click the online screen casting button when playing a video to switch the mirror screen casting to online screen casting", "Click the online screen casting button when playing a video to get a clearer screen casting effect", etc.

[0189] In some embodiments, window 703 may also include image 703b and image 703c, and image 703b and image 703c are respectively used to indicate the effects of sharing network videos based on mirroring and online screen casting. It can be seen that image 703c is clearer and has better effects than image 703b. In this way, the user can be reminded of the difference between mirroring and online screen casting, which is conducive to the user to choose a more suitable online screen casting to share network videos. The prompt information displayed on the display screen of the mobile device 100 can automatically disappear after being displayed for a period of time (such as 5 seconds) without user interaction. The mobile device 100 can also stop displaying the prompt information in response to the user clicking on other areas other than the prompt information on the display screen. It is understandable that when outputting prompt information in scene 1, the mobile device 100 can display the following information in sequence according to the user's operation. Fig. 7A , Fig. 8A , Figure 7B and Figure 7C In this way, after the user opens a specific video application, the user can be automatically prompted to trigger the mobile device 100 to switch the mirror screen projection to online screen projection, ensuring that the user gets the best screen projection experience when watching online videos. Figure 7C The "9:21" in the video is used to indicate the last playback record, that is, the playback of the online video was exited at "9:21" last time.

[0190] Scenario 2: After the mobile device 100 starts the video application and plays the online video selected by the user, it outputs prompt information.

[0191] Similarly, the mobile device 100 may adopt S009, or the method of S009 and S010, and when it is identified that the application program running in the foreground of the mobile device 100 is a non-customized video application, the mobile device 100 outputs prompt information. Figure 8B The prompt information displayed by the mobile device 100 in scene 2 is exemplarily shown. Figure 8B As shown, the prompt information 704 is a window. The specific content of the prompt information 704 is the same as Fig. 8A It is similar to the window 703 of Scenario 2, and will not be described in detail. It is understandable that when outputting prompt information in Scenario 2, the mobile device 100 can display the prompt information in sequence according to the user operation. Fig. 7A , Figure 7B , Figure 7C and Figure 8B In this way, after starting to play a specific online video, the user can be automatically prompted to trigger the mobile device 100 to switch the mirror screen to online screen casting, ensuring that the user gets the best screen casting experience when watching online videos. Figure 8B In the example, if the online screen projection button is not clicked within the preset time, the window 703 is closed. Figure 8B In the default time (for example, 20 seconds), if the online screen projection button is not clicked, the mirror screen projection mode is still used, or it is automatically switched to the online screen projection mode. The above 20 seconds is only an illustrative example, and any time can be the preset time, which is not limited in this application.

[0192] Scenario 3: After the mobile device 100 continues to run the video application in the foreground for more than a first preset time, a prompt message is output.

[0193] The mobile device 100 may adopt S009, or the method of S009 and S010, and after identifying that the application running in the foreground of the mobile device 100 is a non-customized video application, and the continuous running time in the foreground exceeds a first preset time, the mobile device 100 outputs a prompt message. The first preset time can be, for example, 10 seconds, 30 seconds, 1 minute, etc., which is not limited in the embodiments of the present application. After the mobile device 100 continues to run the video application in the foreground for a period of time exceeding the first preset time, the mobile device 100 may still display the main page provided by the video application, or may play the online video in response to the user's operation of selecting an online video. Therefore, if Fig. 8A As shown, the prompt information displayed by the mobile device 100 can be displayed in Figure 7B In the video application main page 71 shown; Figure 8B As shown, this prompt information can also be displayed in Figure 7C In the user interface 72 shown. In this way, when the user is watching an online video, the mobile device 100 can be prompted to switch the mirror screen projection to online screen projection, ensuring that the user gets the best screen projection experience when watching the online video.

[0194] Unless otherwise specified or there is no contradiction, the contents not described in Scenario 2 and Scenario 3 are the same as those in Scenario 1 and will not be repeated here.

[0195] S102, the mobile device 100 receives a user operation to enable an online screen projection function.

[0196] For example, reference Figure 8C , the user operation for starting the online screen projection function of the video application may be, for example, a user operation (such as a click operation, a touch operation, etc.) acting on the online screen projection control 702. The user operation for starting the online screen projection control may also be in other forms, such as a shaking gesture, a voice command, etc., which is not limited in the embodiment of the present application.

[0197] S103-S108, in response to the user operation of the online screen projection function, the mobile device 100 switches the mirror screen projection to the online screen projection, and continues to share the network video with the electronic device 200 based on the online screen projection.

[0198] In response to the user operation of enabling the online screen projection function of the video application in S102 , the mobile device 100 may enable the online screen projection function of the video application.

[0199] S103, the mobile device 100 discovers nearby electronic devices that support online screen projection and displays the identification of the discovered electronic devices.

[0200] Exemplarily, the mobile device 100 can send a user datagram protocol (UDP) broadcast. Nearby electronic devices that support online screen projection, such as the electronic device 200 connected to the Wi-Fi access point 300, and other electronic devices (not shown in the figure) can respond to the UDP broadcast and reply to a UDP message carrying their own relevant information (such as a device identifier) ​​so that the mobile device 100 can discover itself. After the mobile device 100 discovers a nearby electronic device that supports online screen projection, it displays the identifier of the discovered electronic device, and can also display other information such as an image of the electronic device, which is not limited in the embodiments of the present application. Fig.8D The user interface displayed by the mobile device 100 in response to the user operation of starting the mirroring screen function detected in S101 is exemplarily shown. Fig.8D As shown, the user interface may display the identification of the electronic device discovered by the mobile device 100 .

[0201] S104 , the mobile device 100 detects a user operation of selecting the electronic device 200 .

[0202] For example, Fig.8D As shown, the user operation of selecting the electronic device 200 may be to act on Fig.8DThe user operation on the logo corresponding to the electronic device 200 in the user interface shown. The user operation of selecting the electronic device 200 can also be implemented in other forms, which are not limited in the embodiment of the present application.

[0203] In one embodiment, when the mobile device 100 starts the online screen projection function of the video application for the first time, the mobile device executes S103 and S104 to share the network video with the electronic device selected by the user based on the online screen projection.

[0204] In another embodiment, when the mobile device 100 is not starting the online screen projection function of the video application for the first time, the mobile device 100 automatically projects the screen to the electronic device connected to the previous online screen projection, and shares the network video through online screen projection. That is, in another embodiment, S103-S104 may not be executed, and S105 and its subsequent steps may be directly executed after S102.

[0205] In yet another embodiment, when the mobile device 100 detects that there is only one electronic device nearby that supports online screen projection, it automatically and wirelessly projects the screen to the electronic device in an online screen projection manner to share the network video.

[0206] S105 , a transmission control protocol (TCP) connection is established between the mobile device 100 and the electronic device 200 .

[0207] S106, the mobile device 100 sends the URL of the played network video to the electronic device 200 based on the TCP connection.

[0208] Here, the website address of the network video may be a URL, and the URL locates a server corresponding to the video application, such as server 400 .

[0209] In some embodiments, the mobile device 100 may also send the time node of the currently playing network video to the electronic device 200 so that the electronic device 200 continues to play the network video from the time node.

[0210] In some embodiments, the mobile device 100 responds to the user operation of turning on the online screen projection function detected in S102, establishes a TCP connection with the electronic device 200 and sends the URL of the network video to the electronic device 200, and can also change the display form of the control 702, such as adding a shadow, changing the color of the control 702, etc., so as to prompt the user that the mobile device 100 is currently sharing the network video based on the online screen projection. In other embodiments, the mobile device 100 can also prompt the user that the mobile device 100 is currently sharing the network video based on the online screen projection by displaying text or other means, which is not limited in the embodiments of the present application.

[0211] S107, the electronic device 200 obtains the network video from the website address of the network video.

[0212] The electronic device 200 can request the server 400 to obtain the network video according to the website address, and the server 400 can respond to the request by encoding the network video and sending it to the electronic device 200 through the network (such as the local area network formed by the Wi-Fi access point 300).

[0213] S108, the electronic device 200 plays the network video.

[0214] During the process of sharing the online video between the mobile device 100 and the electronic device 200 based on online screen projection, the user can continue to operate the mobile device 100, and the operation of the mobile device 100 by the user does not affect the continued playing of the online video by the electronic device 200. For example, the user can operate the mobile device 100 to exit playing the online video, exit the running video application, and start the game application.

[0215] Through the above steps S101-S108, the mobile device 100 can switch the mirror screen to online screen casting when the user selects an inappropriate mirror screen casting to share the network video, thereby improving the screen casting efficiency and ensuring the quality of the network video played by the electronic device 200, thereby ensuring that the user gets the best screen casting experience. Figure 5B The screen projection method shown can lower the user threshold and ensure user experience.

[0216] In some embodiments, the mobile device 100 may also switch the online screen projection back to the mirror screen projection in some cases. The following describes the case of switching the online screen projection back to the mirror screen projection through optional step S109.

[0217] S109, after detecting that the game application is started, or after detecting that the game application is switched to the foreground application running, the mobile device 100 switches the online screen projection to the mirror screen projection under the triggering of the user.

[0218] Specifically, the mobile device 100 can adopt S009, or S009 and S010. After detecting that the application running in the foreground of the mobile device 100 is a game application, the mobile device 100 can output a prompt message, which can be used to ask the user whether to switch the online screen projection to the mirror screen projection.

[0219] Fig. 8E The following is an exemplary diagram showing the prompt information displayed after the mobile device 100 starts the game application. Fig. 8EAs shown, the prompt information may be a window 705. Window 705 may include: text 705a, control 705b, and control 705c. Text 705a may be, for example, "Do you want to switch the online screen projection to the mirror screen projection to get a smoother game projection experience?", which is used to ask the user whether to switch the online screen projection to the mirror screen projection. Control 705b is used to monitor user operations, and the mobile device 100 may respond to the user operation and not perform the operation of switching the online screen projection to the mirror screen projection. Control 705c is used to monitor user operations, and the mobile device 100 may respond to the user operation and switch the online screen projection to the mirror screen projection. The operation of the mobile device 100 switching the online screen projection to the mirror screen projection can be referred to. Figure 5A S002-S007 in the method shown are not repeated here.

[0220] In this way, under the premise of wireless screen projection, when the application running in the foreground is switched to a game application, or when the game application is started, the mobile device 100 automatically prompts the user to switch the online screen projection back to the mirror screen projection. If the user chooses to switch the online screen projection back to the mirror screen projection, a smoother and lower-latency game screen projection experience is provided to the user; if the user chooses to refuse to switch the online screen projection back to the mirror screen projection, the user can watch online videos on the electronic device 200 and play games on the mobile device 100.

[0221] In an alternative manner, S109 may also be replaced by: after detecting that the game application is started, or after detecting that the game application is switched to the foreground application running, the mobile device automatically switches back to the mirror screen projection. In this way, the user does not need to select and automatically switches back to the mirror screen projection, so that the user experience is better.

[0222] Next, we will introduce Figure 5C process. Figure 5C The process of switching the mirror screen projection currently used by the mobile device 100 to online screen projection is described for customized video applications. Figure 5C In the present invention, when the mobile device 100 shares the network video provided by the customized video application with the electronic device 200 through mirroring, the mirroring can be switched to online projection automatically or under the triggering of the user, and the network video can continue to be shared based on the online projection. Figure 5C The process shown may include:

[0223] S201-S204, during the process of playing the online video, the mobile device 100 automatically or under the trigger of the user switches the mirror screen projection to the online screen projection, and continues to share the online video with the electronic device 200 based on the online screen projection.

[0224] S201 , the mobile device 100 directly establishes a TCP connection with the electronic device 200 .

[0225] The video application is a customized video application. When the mobile device 100 is running the video application and playing the online video, the video application can query the screen casting status from the screen casting service through the SDK interface, and can also query that the device currently sharing the online video with the mobile device 100 is the electronic device 200. Therefore, in S201, the mobile device 100 can directly establish a TCP connection with the queried electronic device 200 without the need to connect to the TCP server as in S202. Figure 5A It is selected by the user or by default by the mobile device 100 based on the last online screen projection interaction.

[0226] S202-S204, with Figure 5B S106-S108 are the same as those in the previous section and will not be described in detail here.

[0227] In one implementation, S201 - S204 are performed automatically.

[0228] In some embodiments, the mobile device 100 may output a prompt message before automatically switching to prompt the user that the mobile device 100 is going to switch the mirror screen projection to the online screen projection.

[0229] In some embodiments, the mobile device 100 may output a prompt message after automatic switching to prompt the user that the screen has been switched to online projection.

[0230] The prompt information output by the mobile device 100 before and after the switch may be implemented in the form of, but not limited to: interface elements displayed on the display screen, audio, flashing indicator light, motor vibration, etc.

[0231] Figure 8F The prompt information 1001 displayed before the mobile device 100 automatically switches is exemplarily shown, and the prompt information 1001 is the text "You will be switched to online screen projection soon." Figure 8G The example shows a prompt message 1002 displayed after the mobile device 100 automatically switches, and the prompt message 1002 is the text "Switched to online screen projection".

[0232] In other implementations, the mobile device 100 may not output prompt information before or after automatic switching, and this embodiment of the present application does not limit this.

[0233] In this way, not only can the screen projection efficiency be improved and users can be guaranteed to get the best screen projection experience, but user operations can also be reduced, making the user experience simpler and more convenient.

[0234] In another embodiment, the mobile device 100 may adopt S009, or S009 and S010. When it is identified that the application running in the foreground of the mobile device 100 is a customized video application, the mobile device 100 outputs a prompt message and switches the mirror screen projection to online screen projection when triggered by the user.

[0235] Figure 8H The prompt information outputted when the mobile device 100 recognizes that a network video is being played is exemplarily shown. Figure 8H The prompt information in Figure 8B The prompt information is the same as in , and will not be repeated here. For example, Figure 8H As shown, the mobile device 100 can detect a user operation (such as a click operation, a touch operation) acting on the control 702, and in response to the user operation, switch the mirror screen projection to the online screen projection. Not limited to this, the mobile device 100 can also switch the mirror screen projection to the online screen projection in response to other user operations, such as shaking gestures, voice commands, etc., and the embodiments of the present application are not limited to this. In some embodiments, in response to user operations, after the mobile device 100 switches the mirror screen projection to the online screen projection, the display mode of the control 702 can be changed to prompt the user that it has currently switched to the online screen projection. In this way, the user can be given ample choice when switching.

[0236] In some embodiments, after S202-S204, the mobile device 100 may also switch the online screen projection back to the mirror screen projection in some cases. That is, an optional step S205 may also be included.

[0237] S205, the mobile device 100 switches the online screen projection back to the mirror screen projection.

[0238] In one embodiment, in S205, the mobile device 100 can automatically switch the online screen projection back to the mirror screen projection after the electronic device 200 finishes playing the online video. In some cases, after the electronic device 200 finishes playing the online video, the user needs to further operate the electronic device 200 to repeat the online video or play other online videos. Fig. 9 The user interface displayed after the electronic device 200 plays the online video is shown as an example. In this way, the mobile device 100 can adaptively adjust the wireless screen projection method used to provide the user with the best screen projection experience. In addition, the mobile device 100 can automatically switch between online screen projection and mirror screen projection in the background without user operation, which can bring a good screen projection experience to the user.

[0239] In another embodiment, in S205, the mobile device 100 can switch the online screen projection back to the mirror screen projection under the trigger of the user after detecting that the game application is started or after detecting that the game application is switched to the foreground running application. For specific details, please refer to the relevant description in S109, which will not be repeated here.

[0240] In the above Figure 5A-Figure 5CIn the illustrated embodiment, non-customized video applications may be referred to as first-category applications, game applications may be referred to as second-category applications, and customized video applications may be referred to as third-category applications.

[0241] Figure 5A The application launched by the mobile device 100 in S008 may be referred to as a first application.

[0242] Figure 5B The prompt information output by the mobile device 100 in S101 can be called the first prompt information, the user operation of starting the online screen projection function received by the mobile device 100 in S102 can be called the first user input, the user operation of selecting the electronic device 200 detected by the mobile device 100 in S104 can be called the second user input, the application started or switched to the foreground detected by the mobile device 100 in S109 can be called the second application, the game application can be called the second type of application, and the prompt information output by the mobile device 100 in S109 can be called the second prompt information.

[0243] exist Figure 5A or Figure 5B After the method shown, the mobile device 100 can also switch the first application belonging to the first category of applications to the foreground, or start the third application belonging to the first category of applications, or switch the third application belonging to the first category of applications to the foreground, and automatically output a third prompt information, the third prompt information is used to prompt the first wireless screen projection mode to be switched to the second wireless screen projection mode; or, the mobile device 100 automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device 200 in the second wireless screen projection mode.

[0244] Next, in conjunction with the software structure of the mobile device 100 (eg Figure 3B The software structure shown in FIG. 1 is described in detail. Figure 5B The process in is the interaction process between the internal modules. Figure 3B The software structure shown is used for illustrative purposes only, but those skilled in the art should know that other operating systems (such as Harmony, etc.) may also be applicable.

[0245] Fig. 10A The mobile device 100 is shown executing Figure 5B The process shown is the interaction process between the internal modules. Fig. 10A As shown, it may include:

[0246] Step 1: Enable the mirroring function in the screen projection service.

[0247] In step 1, after the mobile device 100 detects the user operation to turn on the mirroring screen projection function, the screen projection service will call one or more of Wi-Fi Direct, Bluetooth or NFC in the wireless communication module 160, and discover electronic devices that can mirror the screen near the mobile device 100 through one or more of Wi-Fi Direct, Bluetooth, NFC, establish a communication connection with the electronic device 200 selected by the user, and share multimedia content with the electronic device 200.

[0248] Step 2: Start the video application and obtain the network video.

[0249] The implementation of step 2 can refer to Figure 7A-7C The relevant description will not be repeated here.

[0250] In step 2, after detecting a user operation for starting a video application, the mobile device 100 starts the video application, and then acquires the network video in response to a user operation of selecting a network video to play.

[0251] Step 3: The scene perception module recognizes any one of the three scenes.

[0252] The three scenarios are: Figure 5B The three scenarios mentioned in S101 may specifically include: (1) the mobile device 100 starts a non-customized video application; (2) the mobile device 100 starts a non-customized video application and plays an online video selected by a user from the application; (3) the mobile device 100 continues to run the non-customized video application in the foreground for a period exceeding a first preset period.

[0253] The scene perception module can use S009, or S009 and S010 in real time or periodically to identify whether the application running in the foreground of the mobile device 100 is a non-customized video application, and further identify the above three scenes.

[0254] In some embodiments, the scene perception module can be integrated into the screen projection service. In this way, the screen projection service can be used to perform step 3.

[0255] Step 4: The scene perception module notifies the screen projection service of the identified scene.

[0256] Step 5: After the screen projection service learns the scene identified by the scene perception module, it outputs a prompt message to prompt the user to switch to online screen projection.

[0257] Specifically, when the screen projection service learns that the mobile device 100 has enabled the mirror screen projection function and the mobile device 100 is in one of the above three scenarios, it calls the hardware of the mobile device 100, such as the display screen, flash, motor, etc., to output prompt information, which is used to prompt the user to switch the mirror screen projection to online screen projection. The implementation form of the prompt information can refer to the relevant description of S101.

[0258] Step 6: The video application starts the online screen projection function when triggered by the user.

[0259] Step 6 can refer to Figure 5B See the relevant description of S102-S106 in .

[0260] Specifically, the video application may receive an event that the user turns on the online screen projection function of the video application. Figure 8C , the user can click on the control 702, and the click operation can be encapsulated as an event to enable the online screen projection function of the video application, and passed to the video application from the bottom layer. Afterwards, the video application can enable the online screen projection function, that is, call the wireless communication module 160 to send UDP broadcast, establish a TCP connection with the electronic device 200, and send the URL of the network video to the electronic device 200 based on the TCP connection.

[0261] Fig. 10B The mobile device 100 is shown executing Figure 5B The process shown is another interactive process between the internal modules. Fig. 10B As shown, it may include:

[0262] Step 1-3, with Fig. 10A Steps 1 to 3 are the same as in the previous step and will not be repeated here.

[0263] Step 4: The scene perception module queries the screen projection service for the screen projection status of the mobile device 100.

[0264] Specifically, the scene perception module can query from the screen projection service whether the mirror screen projection function is currently enabled on the current mobile device 100.

[0265] When the scene perception module learns that the mobile device 100 has turned on the mirror screen projection function and the mobile device 100 is in one of the three scenes in S101, it calls the hardware of the mobile device 100, such as the display screen, flash, motor, etc., to output prompt information, which is used to prompt the user to switch the mirror screen projection to online screen projection. The implementation form of the prompt information can refer to the relevant description of S101.

[0266] Step 5: Fig. 10A The steps are the same as step 5 in the previous section and will not be repeated here.

[0267] Understandably, the above Fig. 10A and Fig. 10B In this example, the function of the scene perception module can be integrated into the screen projection service. Fig. 10A and Fig. 10B The steps performed by the scene perception module are all executed by the screen projection service, and the interaction steps between the two can also be omitted.

[0268] Next, in conjunction with the software structure of the mobile device 100 (eg Figure 3B The software structure shown in FIG. 1 is described in detail. Figure 5C The process in is the interaction process between the internal modules. Figure 3B The software structure shown in FIG. 1 is used as an example to illustrate, but those skilled in the art should know that other operating systems (such as Harmony, etc.) may also be applicable. Fig.11 As shown, it may include:

[0269] 1. Enable the mirroring function in the screen projection service.

[0270] 2. Start the video application and obtain the network video.

[0271] Step 1-Step 2, with Fig. 10A Steps 1 and 2 are the same as in the previous step and will not be repeated here.

[0272] 3. The video application queries the screen projection service for the screen projection status.

[0273] The video application can query the screen projection service through the SDK for the current screen projection status and learn that the mobile device 100 is currently sharing network video based on mirrored screen projection. In some embodiments, the video application can query the screen projection service through the SDK to find out the electronic device 200 that is currently receiving the screen projection of the mobile device 100.

[0274] 4. The video application learns that the mobile device 100 shares the network video with the electronic device 200 based on mirroring.

[0275] In combination with step 2 and step 3, the video application can know that the current mobile device 100 is currently sharing a network video with the electronic device 200 based on mirroring.

[0276] 5. The video application automatically or when triggered by the user, enables the online screen projection function.

[0277] In some embodiments, before or after the video application automatically switches the mirror screen to online screen projection, the video application can also call the display screen, audio module, flash module, etc. to output prompt information. The implementation form of the prompt information can refer to Figure 8F , Figure 8G Related description.

[0278] In other embodiments, after the video application learns that the current mobile device 100 is currently sharing a network video with the electronic device 200 based on mirroring, it can call the display screen to output a prompt message to prompt the user to switch the mirroring screen to online screen casting, and then turn on the online screen casting function after the user inputs a user operation to turn on the online screen casting. Here, the display screen outputs a prompt message to prompt the user to switch the mirroring screen to online screen casting. Figure 8H Window 704 in.

[0279] Specifically, the video application turns on the online screen projection function, that is, calls the wireless communication module 160 to send a UDP broadcast, establishes a TCP connection with the electronic device 200, and sends the URL of the network video to the electronic device 200 based on the TCP connection.

[0280] In this way, when the video application started by the mobile device 100 is a non-customized video application, the scene perception module can perceive the three scenarios mentioned in S101, and then the scene perception module or the screen projection service can output a prompt message to prompt the user to switch the mirror screen projection to the online screen projection. Afterwards, the video application can turn on the online screen projection function in response to the user operation input by the user after seeing the prompt message. When the video application started by the mobile device 100 is a customized video application, the customized video application can obtain the screen projection status from the screen projection service through the SDK interface, and combined with the operation of the customized video application itself, it can identify the scenario in which the mobile device 100 shares the network video based on the mirror screen projection. Therefore, the video application can automatically or passively turn on the online screen projection function.

[0281] That is, execute Figure 5B The process shown does not require any improvement to the video application. It only requires improving the software system of the mobile device 100, adding a scene perception module, and adding a screen projection service or a function for the scene perception module to output prompt information, so as to realize the function of switching the mirror projection to the online projection in the process of sharing network video based on the mirror projection. Figure 5C The process shown requires improvements to the video application and a new SDK to implement the function of switching mirrored screen projection to online screen projection during the process of sharing network videos based on mirrored screen projection.

[0282] In some embodiments, customized video applications and non-customized video applications may not be distinguished. As long as the application started by the mobile device 100 is a video application, the mobile device 100 uses Figure 5B The process shown is used to switch the wireless screen projection mode.

[0283] It should be noted that the wireless screen projection method provided in this application does not necessarily include Figure 5A , Figure 5B and Figure 5C The process shown. Figure 5A , Figure 5B and Figure 5C The process shown and the above Figure 5A , Figure 5B and Figure 5C The parts of the text corresponding to the process shown can also independently constitute the wireless screen projection method provided in the embodiment of the present application. For example, Figure 5A and Figure 5B The process shown in the above Figure 5A and Figure 5B The text corresponding to the process shown; for example, Figure 5A and Figure 5C The process shown in the above Figure 5A and Figure 5C The text corresponding to the process shown; for example, Figure 5A The portion of the process shown (e.g., Figure 5A S009-S012), and Figure 5A The text corresponding to the part of the process shown is Figure 5B The process shown in the above Figure 5B The text corresponding to the process shown can independently constitute the wireless screen projection method provided in the embodiment of the present application.

[0284] Fig.12 A flowchart of another screen projection method provided in an embodiment of the present application. Fig.12 As shown, when the mobile device 100 shares multimedia content with the electronic device 200 based on mirroring, the mobile device 100 can identify the current scene and notify the electronic device 200 of the scene. After the electronic device 200 learns the current scene, it selects the corresponding playback strategy to play the multimedia content according to the scene adaptability. The multimedia content shared between the mobile device 100 and the electronic device 200 can be network multimedia content, local multimedia content, or a combination of the two, and this application does not limit this.

[0285] like Fig.12 As shown, the method may include the following steps:

[0286] S301-S307, and Figure 5A The S001-S007 in are the same and will not be repeated here.

[0287] S308, the mobile device 100 starts the application.

[0288] The application launched by the mobile device 100 may be any installed application. For example, the mobile device 100 may respond to the Fig. 6AThe video application is started by a user operating on the icon 601 of the video application on the main interface 61 shown in FIG. Fig. 6A The user operates on the game application icon 602 on the main interface 61 to start the game application. The video application refers to an application provided with audio and video services by a server. The game application refers to an application provided with game services by a server.

[0289] S309, the mobile device 100 identifies the scene corresponding to the launched application.

[0290] The mobile device 100 can distinguish different scenes according to different applications started. For example, when the mobile device 100 starts a video application, the mobile device 100 can identify the scene of running the video application; when the mobile device 100 starts a video application, the mobile device 100 can identify the scene of running a game application.

[0291] Specifically, the scene perception module or screen projection service of the mobile device 100 can use S009, or S009 and S010 in real time or periodically to obtain the application running in the foreground, thereby identifying the current scene.

[0292] In some embodiments, the mobile device 100 is not limited to distinguishing different scenarios based on the launched application program, and can also distinguish different scenarios based on other strategies, which is not limited in the embodiments of the present application.

[0293] S310, the mobile device 100 notifies the electronic device 200 of the recognized scene.

[0294] In some embodiments, the mobile device 100 may carry the indication information of the identified scene in the control instruction of the mirror screen projection used, thereby notifying the electronic device 200 of the identified scene. Taking miracast as an example, the mobile device 100 may carry the indication information of the identified scene in the extension field of the RTSP control instruction. For example, if the extension field value is 1, it may indicate that the mobile device 100 has identified the scene of running a video application; if the extension field value is 0, it may indicate that the mobile device 100 has identified the scene of running a game application.

[0295] S311, the electronic device 200 uses the playback strategy corresponding to the scene to play the multimedia content.

[0296] When the mobile device 100 and the electronic device 200 share multimedia content based on mirror screen projection, there may be a variety of playback strategies. Exemplarily, the playback strategies corresponding to the mirror screen projection may include: real-time display strategy, and audio and video cache synchronization strategy. Among them, when using the real-time display strategy, the electronic device 200 directly decodes and plays the data sent by the mobile device 100 after receiving it, giving priority to ensuring the user's low-latency screen projection experience in scenarios such as game screen projection. When using the audio and video cache synchronization strategy, after receiving the encoded data sent by the mobile device 100, the electronic device 200 will cache certain data to ensure the smoothness of the multimedia content, and automatically compare the timestamps (presentation time stamp, PTS) of the audio and image in the multimedia content. When the difference between the audio and video timestamps exceeds the threshold, the audio and video synchronization processing logic (such as audio double-speed playback, image frame loss, etc.) is triggered to ensure the audio and video synchronization quality when the electronic device 200 plays multimedia content.

[0297] It is understandable that the real-time display delivery strategy and the audio and video cache synchronization strategy are just words used in the embodiments of the present application. The meanings they represent have been recorded in the embodiments of the present application, and their names do not constitute any limitation to the embodiments of the present application. In addition, in some other embodiments of the present application, the real-time display delivery strategy may also be referred to as other terms such as low-latency fast display delivery mode. Similarly, the audio and video cache synchronization strategy mentioned in the embodiments of the present application may also be referred to as other names such as cache mode in some other embodiments.

[0298] For example, referring to Table 1, Table 1 shows the correspondence between the scene where the mobile device 100 is located and the playback strategy used by the electronic device 200 under the premise of wireless screen projection of the mobile device 100 to the electronic device 200. As shown in Table 1, the scene of running a game application can correspond to the real-time display strategy, and the scene of running a video application can correspond to the audio and video cache synchronization strategy.

[0299] Table 1

[0300] Scenario Playback strategy Running game application scenario Real-time display strategy Scenarios for running video applications Audio and video cache synchronization strategy

[0301] In the scenario where the mobile device 100 runs a game application, real-time performance is the main factor affecting the user's screen projection experience, so the real-time display strategy is more suitable for the scenario where the game application is running. In the scenario where the mobile device 100 runs a video application, the smoothness of the screen projection and the quality of audio and video synchronization are the main factors affecting the user's screen projection experience, so the audio and video cache synchronization strategy is more suitable for the scenario where the video application is running. Not limited to the real-time display strategy and the audio and video cache synchronization strategy, in other embodiments, the mirror screen projection may also correspond to other more playback strategies, such as ultra-low latency mode, etc., which is not limited by the embodiments of the present application. Other playback strategies corresponding to the mirror screen projection can be customized by those skilled in the art, which is not limited by the embodiments of the present application. The embodiments of the present application also do not limit the correspondence between each scene and the playback strategy. In some other embodiments, there may be other correspondences, which are not limited by the embodiments of the present application.

[0302] When S311 is specifically implemented, the screen projection service in the electronic device 200 can instruct the screen projection player to use the playback strategy corresponding to the scene recognized by the mobile device 100 to play the multimedia content.

[0303] pass Fig.12 In the screen projection method shown, when sharing multimedia content based on mirror screen projection, the electronic device 200 can adaptively select the corresponding playback strategy to play the multimedia content according to the scene in which the mobile device 100 is located, that is, adjust the playback strategy according to the actual scene, thereby ensuring the user's screen projection experience.

[0304] Next, in conjunction with the software structure of the mobile device 100 (eg Figure 3B The software structure shown in FIG. 1 is described in detail. Fig.12 The process in is the interaction process between the internal modules. Figure 3B The software structure shown in FIG. 1 is used as an example to illustrate, but those skilled in the art should know that other operating systems (such as Harmony, etc.) may also be applicable. Fig.13A As shown, it may include:

[0305] The mobile device 100 executes steps 1 to 5:

[0306] Step 1: Enable the mirroring function in the screen projection service.

[0307] Step 2: Start the application.

[0308] Step 3: The scene perception module identifies the scene corresponding to the launched application.

[0309] The implementation of step 1 can refer to Fig.12The operation of the mobile device 100 in S301-S307 of the method shown; the implementation of step 2 can refer to Fig.12 S308 in the method shown; the implementation of step 3 can refer to Fig.12 S309 in the method shown.

[0310] Step 4: The scene perception module notifies the screen projection service of the identified scene.

[0311] Step 5: The screen projection service notifies the electronic device 200 of the scene recognized by the scene perception module.

[0312] Specifically, after the screen projection service learns that the mobile device 100 has turned on the mirror screen projection function and learns the scene recognized by the scene perception module, it can notify the electronic device 200 of the scene.

[0313] The implementation of step 5 can refer to Fig.12 S310 in the method shown.

[0314] Step 6-step 7 performed by the electronic device 200:

[0315] Step 6: The screen projection service determines the playback strategy corresponding to the scene in which the mobile device 100 is located, and passes the playback strategy to the screen projection player.

[0316] Step 7: The projection player uses the playback strategy to play the multimedia content.

[0317] The implementation of steps 6 and 7 can be referred to Fig.12 S311 in the method shown.

[0318] Fig. 13B The mobile device 100 is shown executing Figure 5C It is another interactive process between internal modules. Fig. 13B and Fig.13A The difference is that in step 4, the scene perception module of the mobile device 100 queries the screen projection service for the screen projection status, and in step 5, the scene perception module notifies the electronic device 200 of the identified scene.

[0319] Understandably, the above Fig.13A and Fig. 13B In the example of , the function of the scene perception module can be integrated into the screen projection service. When the function of the scene perception module is integrated into the screen projection service, Fig.13A and Fig. 13B The steps performed by the scene perception module are all executed by the screen projection service, and the interaction steps between the two can also be omitted.

[0320] It is understandable that in the embodiments of the present application, the above Figure 5B The process shown, Figure 5CThe processes shown can be Fig.12 The methods shown can be implemented in combination or separately.

[0321] For example, when Figure 5B The process shown is Fig.12 In the combined implementation, during the process of sharing network video between the mobile device 100 and the electronic device 200 based on mirroring screen projection, if the electronic device 200 is connected to the network, that is, the electronic device 200 has the conditions for online screen projection, the mobile device 100 can execute Figure 5B If the electronic device 200 is not connected to the network, that is, the electronic device 200 does not have the conditions for online screen projection, the mobile device 100 can execute Fig.12 The screen projection method.

[0322] For example, when Figure 5C The process shown and Fig.12 In the combined implementation, during the process of sharing network video between the mobile device 100 and the electronic device 200 based on mirroring screen projection, if the electronic device 200 is connected to the network, that is, the electronic device 200 has the conditions for online screen projection, the mobile device 100 can execute Figure 5C If the electronic device 200 is not connected to the network, that is, the electronic device 200 does not have the conditions for online screen projection, the mobile device 100 can execute Fig.12 The screen projection method.

[0323] All or part of the various implementation modes and examples of the present application may be combined arbitrarily and freely.

[0324] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0325] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0326] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. A mobile device, wherein the mobile device runs a first application in the foreground, and the mobile device wirelessly projects a screen to an electronic device in a first wireless screen projection mode, characterized in that: The mobile device comprises: Scene perception module and screen projection service; processor; Memory; and a computer program, wherein the computer program is stored on the memory and when the computer program is executed by the processor, causes the mobile device to perform the following steps: If the first application is identified as a first-category application, which is a video application and does not have the ability to initiate screen projection, the scene perception module identifies one of the following scenarios: starting the first-category application, starting the first-category application and playing a video selected by a user, and running the first-category application in the foreground for more than a first preset time; querying from the screen projection service through the scene perception module that the screen projection status of the mobile device is projected in the first wireless screen projection mode, and outputting a first prompt message through the scene perception module; or, sending the identified scene to the screen projection service through the scene perception module, and after the screen projection service learns that the screen projection status of the mobile device is projected in the first wireless screen projection mode and the mobile device is in the scene, the screen projection service outputs the first prompt message; the first prompt message is used to prompt the first wireless screen projection mode to be switched to the second wireless screen projection mode; If the first application is identified as a second-category application, and the second-category application is a video application and has the ability to initiate screen projection, then the first application queries the screen projection service to find out that the screen projection status of the mobile device is projected in the first wireless screen projection mode, and the first wireless screen projection mode is switched to the second wireless screen projection mode through the first application; Among them, the first wireless screen projection mode is a mirror screen projection mode, and the second wireless screen projection mode is an online screen projection mode.

2. The mobile device according to claim 1, characterized in that After outputting the first prompt information, the mobile device further performs the following steps: A first user input is detected, where the first user input is used to switch the first wireless screen projection mode to a second wireless screen projection mode; In response to the first user input, the mobile device automatically outputs identifications of one or more electronic devices, where the electronic devices are electronic devices that support the second wireless screen projection mode and are detected by the mobile device; detecting a second user input, the second user input being used to select an identification of an electronic device from the identifications of the electronic devices; In response to the second user input, the mobile device switches the first wireless screen projection mode to the second wireless screen projection mode, and projects the screen to the selected electronic device in the second wireless screen projection mode.

3. The mobile device according to claim 1 or 2, characterized in that: The mobile device also performs the following steps: After detecting that a second application belonging to the second category of applications is started, or after detecting that the second application belonging to the second category of applications is switched to an application running in the foreground, the second category of applications is a game application; The mobile device automatically outputs second prompt information, where the second prompt information is used to prompt the user to switch the second wireless screen projection mode to the first wireless screen projection mode; or The mobile device automatically switches the second wireless screen projection mode to the first wireless screen projection mode, and wirelessly projects the screen to the electronic device in the first wireless screen projection mode.

4. The mobile device according to claim 3, characterized in that The mobile device also performs the following steps: After detecting that the first application belonging to the first category of applications is switched to the foreground application, or after detecting that the third application belonging to the first category of applications is started, or after detecting that the third application belonging to the first category of applications is switched to the foreground application, The mobile device automatically outputs third prompt information, where the third prompt information is used to prompt the user to switch the first wireless screen projection mode to the second wireless screen projection mode; or The mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode.

5. A wireless screen projection method, the wireless screen projection method is applied to a mobile device, the mobile device includes a scene perception module, a screen projection service, a processor and a memory, the mobile device runs a first application in the foreground, and the mobile device wirelessly projects the screen to the electronic device in a first wireless screen projection mode; the method includes: If the first application is identified as a first-category application, and the first-category application is a video application and does not have the ability to initiate screen projection, then the scene perception module identifies one of the following scenarios: starting the first-category application, starting the first-category application and playing a video selected by a user, and running the first-category application in the foreground for more than a first preset time; The screen projection service is queried through the scene perception module that the screen projection status of the mobile device is projected in the first wireless screen projection mode, and the first prompt information is output through the scene perception module; or, the identified scene is sent to the screen projection service through the scene perception module, and after the screen projection service learns that the screen projection status of the mobile device is projected in the first wireless screen projection mode and the mobile device is in the scene, the first prompt information is output through the screen projection service; The first prompt information is used to prompt to switch the first wireless screen projection mode to the second wireless screen projection mode; If the first application is identified as a second-category application, and the second-category application is a video application and has the ability to initiate screen projection, then the first application queries the screen projection service to find out that the screen projection status of the mobile device is projected in the first wireless screen projection mode, and the first wireless screen projection mode is switched to the second wireless screen projection mode through the first application; Among them, the first wireless screen projection mode is a mirror screen projection mode, and the second wireless screen projection mode is an online screen projection mode.

6. The method according to claim 5, characterized in that After outputting the first prompt information, the method further includes: A first user input is detected, where the first user input is used to switch the first wireless screen projection mode to a second wireless screen projection mode; In response to the first user input, the mobile device automatically outputs identifications of one or more electronic devices, where the electronic devices are electronic devices that support the second wireless screen projection mode and are detected by the mobile device; detecting a second user input, the second user input being used to select an identification of an electronic device from the identifications of the electronic devices; In response to the second user input, the mobile device switches the first wireless screen projection mode to the second wireless screen projection mode, and projects the screen to the selected electronic device in the second wireless screen projection mode.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: After detecting that a second application belonging to the second category of applications is started, or after detecting that the second application belonging to the second category of applications is switched to an application running in the foreground, the second category of applications is a game application; The mobile device automatically outputs second prompt information, where the second prompt information is used to prompt the user to switch the second wireless screen projection mode to the first wireless screen projection mode; or The mobile device automatically switches the second wireless screen projection mode to the first wireless screen projection mode, and wirelessly projects the screen to the electronic device in the first wireless screen projection mode.

8. The method according to claim 7, characterized in that The method further comprises: After detecting that the first application belonging to the first category of applications is switched to the foreground application, or after detecting that the third application belonging to the first category of applications is started, or after detecting that the third application belonging to the first category of applications is switched to the foreground application, The mobile device automatically outputs third prompt information, where the third prompt information is used to prompt the user to switch the first wireless screen projection mode to the second wireless screen projection mode; or The mobile device automatically switches the first wireless screen projection mode to the second wireless screen projection mode, and wirelessly projects the screen to the electronic device in the second wireless screen projection mode.

9. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed on a mobile device, the mobile device is enabled to execute the method according to any one of claims 5 to 8.

10. A computer program product, characterized in that When the computer program product is executed on a mobile device, the mobile device is enabled to execute the method according to any one of claims 5 to 8.

Images