Screen projection method and mobile phone

Abstract

The present application provides a screen projection method and a mobile phone, which relate to the field of terminals and can conveniently and quickly project data associated with applications in an electronic device to other electronic devices that have established wireless communication connections for playback. The method includes: in response to a first operation of a user, obtaining the positions of a first electronic device and a second electronic device; determining the positions of the first electronic device icon and the second electronic device icon in a multitasking interface according to the positions of the first electronic device and the second electronic device; obtaining a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the mobile phone; displaying the icons of the above electronic devices and thumbnails of the applications on the multitasking interface; in response to a second operation of the user, projecting multimedia data associated with the first application to the first electronic device.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a screen projection method and a mobile phone. Background Art

[0002] With the development of smart home technology, a user or family often has multiple electronic devices that can communicate with each other. Each type of electronic device generally has its own device characteristics. For example, mobile phones are more portable, TV screens have better display effects, and speakers have better sound quality. In order to give full play to the device characteristics of different electronic devices, electronic devices can realize the interaction of multimedia data between multiple devices through functions such as screen projection, airplay (wireless playback) or Bluetooth.

[0003] Taking the screen projection function as an example, users can install the screen projection software in their mobile phones to send photos, videos and other multimedia data in their mobile phones to other controlled devices (such as smart TVs) that support the screen projection function for display. In one implementation, after the user selects the smart TV as the controlled device, the mobile phone will send the display data on the mobile phone screen to the smart TV in real time for display, that is, the display content in the mobile phone and the smart TV are exactly the same. This screen projection method can easily lead to the leakage of private information in the mobile phone. In another implementation, after the user selects the smart TV as the controlled device, the mobile phone can prompt the user to select the application to be projected in the form of a list. After the user selects a specific application, the mobile phone can project the application selected by the user on the smart TV for display. This screen projection method requires the user to manually set the specific projection content before each projection, which makes the operation of the projection process more cumbersome. Summary of the invention

[0004] The present application provides a screen projection method and a mobile phone, which can conveniently and quickly project data associated with an application to an electronic device for playback in a multi-tasking interface, reducing the operations in the screen projection process, improving the efficiency of human-computer interaction, and enhancing the intelligence of the electronic device and the friendliness of interaction with people.

[0005] In a first aspect, a screen projection method is provided, characterized in that it includes: in response to a first operation of a user, obtaining the positions of a first electronic device and a second electronic device; determining the positions of a first electronic device icon and a second electronic device icon in a multitasking interface according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device, and the multitasking interface includes a first display area, and the first display area is used to display the first electronic device icon and the second electronic device icon; obtaining a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the electronic device; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application, and the multitasking interface also includes a second display area, and the second display area is used to display a thumbnail of the first application and a thumbnail of the second application; displaying the multitasking interface; in response to the second operation of the user, projecting multimedia data associated with the first application to the first electronic device.

[0006] Therefore, the screen projection method provided in the embodiment of the present application, on the one hand, can further determine the positional relationship (such as direction, distance, etc.) between the mobile phone and other devices under the premise of establishing a wireless communication connection relationship between the devices, thereby improving the accuracy and efficiency of the user's selection of the receiving device; on the other hand, the user can select the application and the electronic device to receive the screen projection through the above-mentioned operation method in the multi-tasking interface of the mobile phone, thereby realizing the application screen projection to the electronic device, thereby improving the efficiency of the screen projection operation.

[0007] According to the first aspect, in a possible implementation manner, obtaining the positions of the first electronic device and the second electronic device specifically includes: obtaining the positions of the first electronic device and the second electronic device by using ultra-wideband positioning technology or antenna array technology.

[0008] That is to say, on the premise of establishing a wireless communication connection between devices, nearby electronic devices can be further screened through ultra-wideband positioning technology or antenna array technology, reducing the number of electronic devices discovered by mobile phones or other devices, avoiding the trouble of users searching and identifying target devices among a large number of devices in a large range, and avoiding the occurrence of misselection of devices when projecting the screen, thereby improving the accuracy and efficiency of users' selection of receiving devices.

[0009] According to the first aspect, in a possible implementation method, the first display area includes a coordinate system, and the positions of the first electronic device icon and the second electronic device icon in the multitasking interface are determined according to the positions of the first electronic device and the second electronic device. Specifically, the positions of the first electronic device icon and the second electronic device icon in the coordinate system in the multitasking interface are determined according to the positions of the first electronic device and the second electronic device.

[0010] Among them, the coordinate system may include a center point, a horizontal axis, and a vertical axis. The center point may represent a mobile phone, the horizontal axis represents the left and right direction relative to the mobile phone, and the vertical axis represents the front and back direction relative to the mobile phone. The multitasking interface determines the position of the first electronic device icon and the second electronic device icon in the coordinate system of the multitasking interface according to the position of the first electronic device and the second electronic device, that is, it displays the position / direction of these electronic devices relative to the mobile phone. The positions of different electronic device icons indicate the relative position of their corresponding electronic devices and the mobile phone. In addition, the distance between the electronic device icon and the center point indicates the distance between its corresponding electronic device and the mobile phone. The closer the icon is to the center point, the closer the corresponding electronic device is to the mobile phone.

[0011] That is to say, on the one hand, by screening through the positional relationship (such as direction, distance, etc.) between the mobile phone and other devices, the number of electronic devices discovered by the mobile phone is reduced, and further, the accuracy and efficiency of the user's selection of receiving devices are improved; on the other hand, the positions of the first electronic device icon and the second electronic device icon in the coordinate system in the multitasking interface are determined according to the positions of the first electronic device and the second electronic device. The user can see the relative positions and directions of nearby devices in the multitasking interface of the mobile phone, which improves the accuracy and efficiency of the user's selection of receiving devices.

[0012] According to the first aspect, in a possible implementation manner, the second operation is specifically dragging the thumbnail of the first application to the first electronic device icon.

[0013] That is to say, the user can select the application and the electronic device the phone is pointing to through the above-mentioned operation method in the multitasking interface of the mobile phone, thereby completing the screen projection of the application to the electronic device, making the entire human-computer interaction process more natural and friendly, and improving the user experience.

[0014] According to the first aspect, in a possible implementation, after projecting the multimedia data associated with the first application to the first electronic device, it also includes: in response to a third operation of the user, stopping projecting the multimedia data associated with the first application to the first electronic device, and projecting the multimedia data associated with the second application to the first electronic device.

[0015] In other words, users only need to perform one step in the multitasking interface to complete the switching of the projection application, without the need for multiple steps (for example, first open the first application, then choose to stop projection, and then open the second application to start projection), so that the projection content can be changed very conveniently and user privacy can be protected from leakage.

[0016] According to the first aspect, in a possible implementation manner, the method further includes: in response to a fourth operation of the user, projecting the multimedia data associated with the second application to the second electronic device.

[0017] That is to say, after the mobile phone has projected the first application to the first electronic device, the user can also project the second application on the mobile phone to the second electronic device through the fourth operation, realizing the convenient operation of projecting multiple applications to multiple electronic devices in the multitasking interface. At this time, the multimedia data output by the first application in the mobile phone is projected to the second device for playback, and the multimedia data output by the second application in the mobile phone is projected to the first application for playback. Users can enjoy the multimedia data played by different electronic devices at the same time.

[0018] According to the first aspect, in a possible implementation manner, the method further includes: in response to a fifth operation of the user,

[0019] Stop projecting the multimedia data associated with the first application to the first electronic device.

[0020] In other words, users only need to perform one step in the multitasking interface to stop screen projection, without the need for multiple steps (such as opening the first application first, and then choosing to stop screen projection), so that it can be ended very conveniently, improving the user experience.

[0021] According to the first aspect, in a possible implementation method, in response to the user's second operation, the multitasking interface includes a control for stopping screen projection; the fifth operation is specifically dragging the thumbnail of the first application to the control for stopping screen projection.

[0022] In other words, users only need to drag the thumbnail of the application in the multitasking interface to the control to stop screen projection, without the need for multiple steps (such as opening the first application first, and then choosing to stop screen projection), so that it can be ended very conveniently.

[0023] According to the first aspect, in a possible implementation manner, the method further includes: in response to a sixth operation of the user, projecting the multimedia data associated with the first application to the first electronic device and the second electronic device.

[0024] That is to say, in a scenario where a single application needs to be projected to multiple devices, the user only needs to perform one step in the multitasking interface to project an application to multiple electronic devices. The multiple electronic devices can be electronic devices that the mobile phone points to and has established wireless communication connections. This makes projecting to multiple devices more convenient and faster while limiting the devices that need to be projected.

[0025] According to the first aspect, in a possible implementation manner, the sixth operation is specifically sliding a thumbnail of the first application in a first direction.

[0026] The first direction may be upward or other directions, and the sliding operation may be performed by sliding with two fingers or three fingers.

[0027] That is to say, in a scenario where a single application needs to be projected to multiple devices, the user only needs to slide the application thumbnail in the first direction on the multitasking interface to project an application to multiple electronic devices. The multiple electronic devices can be electronic devices that the mobile phone is pointing to and have established wireless communication connections. This makes projecting to multiple devices more convenient and faster while limiting the devices that need to be projected.

[0028] According to the first aspect, in a possible implementation, the method further includes: in response to a seventh operation of the user, stopping the projection of multimedia data associated with the first application to the first electronic device and the second electronic device.

[0029] That is to say, in the scenario where you need to end the projection of a single application to multiple devices, the user only needs to perform one step in the multitasking interface to stop multiple devices from projecting one application, making it more convenient and faster to end the projection of the application to multiple devices.

[0030] According to the first aspect, in a possible implementation manner, the seventh operation is specifically sliding the thumbnail of the first application in the second direction.

[0031] The second direction may be downward or other directions, and the sliding operation may be performed by sliding with two fingers or three fingers.

[0032] That is to say, in a scenario where you need to end the projection of a single application to multiple devices, the user only needs to slide the thumbnail of the first application in the second direction on the multitasking interface to stop the projection of one application to multiple devices, making it more convenient and quick to end the projection of an application to multiple devices.

[0033] In a second aspect, the present application provides a mobile phone, comprising: a touch screen, a communication interface, one or more processors, a memory, and one or more computer programs; wherein the processor is coupled to the touch screen, the communication interface, the positioning device and the memory, and the one or more computer programs are stored in the memory. When the mobile phone is running, the processor executes the one or more computer programs stored in the memory so that the mobile phone executes any of the methods for playing multimedia data described above.

[0034] In a third aspect, the present application provides a computer storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the screen projection method as described in any one of the first aspects.

[0035] In a fourth aspect, the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the screen projection method as described in any one of the first aspects.

[0036] It can be understood that the terminal described in the second aspect, the computer storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the architecture of a communication system provided by an embodiment of the present application;

[0038] Figure 2A is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0039] Figure 2B is a schematic diagram of an electronic device with an antenna array provided in an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of a software structure of an electronic device provided in an embodiment of the present application;

[0041] Figure 4A This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 1 ;

[0042] Figure 4B This is a second schematic diagram of an application scenario of a screen projection method provided in an embodiment of the present application;

[0043] Figure 4C This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 3 ;

[0044] Figure 4D This is a fourth schematic diagram of an application scenario of a screen projection method provided in an embodiment of the present application;

[0045] Figure 4E This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 5 ;

[0046] Figure 4F This is a sixth schematic diagram of an application scenario of a screen projection method provided in an embodiment of the present application;

[0047] Figure 5 This is a schematic diagram of an application scenario of a screen projection method provided in an embodiment of the present application;

[0048] Fig. 6A This is a schematic diagram 8 of an application scenario of a screen projection method provided in an embodiment of the present application;

[0049] Figure 6B This is a ninth schematic diagram of an application scenario of a screen projection method provided in an embodiment of the present application;

[0050] Fig. 7A This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 10 ;

[0051] Figure 7B This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 10 one;

[0052] Fig. 8A This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 10 two;

[0053] Figure 8B This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 10 three;

[0054] Fig.9A This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 10 Four;

[0055] Fig. 9B This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 10 five;

[0056] Fig. 9C This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 10 six;

[0057] Fig.9D This is an application scenario diagram of a screen projection method provided in an embodiment of the present application. Figure 10 seven;

[0058] Fig.10 This is a schematic diagram of a screen projection method provided in an embodiment of the present application. Figure 1 ;

[0059] Fig.11 This is a second flow chart of a screen projection method provided in an embodiment of the present application;

[0060] Fig.12 This is a schematic diagram of a screen projection method provided in an embodiment of the present application. Figure 3 ;

[0061] Fig.13 This is a flowchart diagram 4 of a screen projection method provided in an embodiment of the present application;

[0062] Fig.14 This is a schematic diagram of a screen projection method provided in an embodiment of the present application. Figure 5 ;

[0063] Fig.15 6 is a flowchart of a screen projection method provided in an embodiment of the present application;

[0064] Fig.16 Schematic diagram 7 of a screen projection method provided in an embodiment of the present application;

[0065] Fig.17 This is flowchart eight of a screen projection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0067] The following describes an electronic device, a graphical user interface for such an electronic device, and embodiments for using such an electronic device. In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a wearable electronic device with wireless communication function (such as a smart watch), etc. Exemplary embodiments of portable electronic devices include but are not limited to devices equipped with Or a portable electronic device with other operating systems. The portable electronic device may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface or a touch panel. It should also be understood that in some other embodiments, the electronic device may not be a portable electronic device, but a desktop computer with a touch-sensitive surface or a touch panel.

[0068] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <head>. A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by <head> and <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0069] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0070] The following embodiments of the present application provide a screen projection method, a graphical user interface, an electronic device and a system, so that the electronic device can quickly project the screen in a multi-tasking interface, and can combine the use of direction, distance and other information to search for nearby devices for screen projection, thereby improving the efficiency of screen projection of the electronic device.

[0071] In the following embodiments of the present application, screen projection can be a service or function provided by an electronic device, which can support the electronic device to transmit data to other devices in a specified direction. In some embodiments, screen projection can support the electronic device to transmit data to nearby devices through one or more technologies such as Bluetooth, wireless fidelity direct (wireless fidelity direct, Wi-Fi direct), Wi-Fi software access point (softAP), millimeter wave (Millimeter Wave, mmWave), and ultra-wideband (UltraWide Band, UWB). Nearby devices refer to devices discovered by electronic devices through one or more technologies such as Bluetooth, Wi-Fi direct (such as Wi-Fi p2p), Wi-Fi softAP, etc., Wi-Fi LAN, millimeter wave (mmWave), and ultra-wideband (UWB).

[0072] It can be understood that screen projection is just a term used in this embodiment. Its meaning has been recorded in this embodiment, and its name does not constitute any limitation to this embodiment.

[0073] The following describes a directional screen projection method provided by an embodiment of the present application, which can be applied to Figure 1 A communication system 10 is shown.

[0074] Figure 1 The communication system 10 for directional screen projection for implementing the present invention is exemplarily shown. The communication system 10 may include an electronic device 100 and one or more nearby electronic devices, such as a first electronic device 101, a second electronic device 102, a third electronic device 103, and the like.

[0075] The electronic device 100 may be a portable electronic device such as a mobile phone or a tablet computer. Specifically, the electronic device 100 may have one or more of a Bluetooth module, a WLAN module, a millimeter wave (mmWave) antenna module, and an ultra-wideband (UWB) antenna module. The electronic device 100 may detect, scan, or directionally detect / scan devices near the electronic device 100 through one or more of the Bluetooth module, the WLAN module, the millimeter wave (MilliMeterWave) antenna module, and the ultra-wideband (UWB) antenna module. One or more of the transmission signals may be used to detect, scan, or directionally detect devices near the electronic device 100, so that the electronic device 100 may use one or more short-range wireless communication protocols such as Bluetooth, WLAN, millimeter wave (mmWave), and ultra-wideband (UWB) to discover or directionally discover nearby devices, and establish a wireless communication connection with the nearby devices, and share data to nearby devices through one or more short-range wireless communication protocols such as Bluetooth, WLAN, millimeter wave (mmWave), and ultra-wideband (UWB).

[0076] The first electronic device 101, the second electronic device 102 or the third electronic device 103 can be an electronic device such as a mobile phone, a tablet computer, a personal computer, etc. having one or more of a Bluetooth module, a WLAN module, a millimeter wave (mmWave) antenna module, and an ultra-wideband (UWB) antenna module, or can also be other devices such as a printer, a projector, a display, a speaker, etc. having one or more of a Bluetooth module, a WLAN module, a millimeter wave (mmWave) antenna module, and an ultra-wideband (UWB) antenna module.

[0077] It is understandable that the structure shown in this embodiment does not constitute a specific limitation on the communication system 10. In other embodiments of the present application, the communication system 10 may include more or fewer devices than shown. For example, the communication system 10 may also include other electronic devices having one or more of a Bluetooth module, a WLAN module, a millimeter wave (mmWave) antenna module, and an ultra-wideband (UWB) antenna module.

[0078] Figure 2A A schematic structural diagram of an electronic device 100 is shown.

[0079] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0080] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0081] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110.

[0082] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0083] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100.

[0084] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0085] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0086] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0087] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0088] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0089] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0090] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0091] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0092] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0093] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0094] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0095] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0096] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0097] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0098] In 2020, the fifth generation of communication technology (5G) will be commercialized on a large scale. The new communication technology has brought new requirements for mobile device antennas and base station antennas. The frequency bands used in 5G include the millimeter wave band (mmWave, 30GHz-100GHz). Therefore, in order to support millimeter wave communication, terminals that support 5G communication have millimeter wave antennas. Compared with traditional microwave frequency bands, the propagation of millimeter wave signals during propagation is similar to ray propagation, that is, the direct path and reflection path are the main ones, without obvious scattering and diffraction. The signal will propagate on some spatially discrete paths, and the number of effective paths here is very small. In millimeter wave channels that lack scattering, diversity gain can be obtained through space division multiplexing (such as MU-MIMO technology), thereby significantly improving the quality of the signal.

[0099] The shorter the wavelength of the millimeter wave signal, the shorter the antenna length. Based on the transmission rate requirements, 5G terminals can have multiple millimeter wave antenna modules. Figure 2B An electronic terminal having a millimeter wave antenna array is exemplarily shown. The electronic terminal can be Figure 1 The electronic terminal 100 in Figure 2B In the embodiment, the electronic terminal 100 has 8 antennas, which form an antenna array, that is, two or more single antennas working at the same frequency are fed and arranged in space to form an antenna array, so that the 5G terminal has beamforming capability, that is, by adjusting the weight of each antenna module, multiple antennas are made to coherent and cancel each other on the waveform, forming a lobe pattern with a zero point and maximum direction, so that the phased antenna array has the ability to send or receive signals in a specified direction. Multiple antennas on an electronic device can be arranged as a uniform linear array, a circular array, a rectangular array, etc.

[0100] At present, 5G millimeter wave antenna arrays are generally based on phased arrays. The specific implementation methods can be divided into three types: AoB (Antenna on Board), that is, the antenna array is located on the system motherboard, AiP (Antenna in Package), that is, the antenna array is located in the chip package, and AiM (Antenna in Module), that is, the antenna array and RFIC form a module.

[0101] In order to achieve wider spatial coverage, millimeter wave antenna arrays are generally designed with antenna types that have complementary radiation beams (such as broadside radiation and end-fire radiation), such as patch antennas and quasi-Yagi antennas, and the antenna feed points are properly designed to achieve dual-polarization coverage, thereby greatly improving the range and coverage of millimeter wave signals.

[0102] Antenna 2 may also include two or more antennas operating at the same frequency to form an antenna array.

[0103] The present invention is applicable to an electronic terminal having an antenna array, in particular to an electronic terminal having a millimeter wave antenna array.

[0104] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0105] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), ultra-wideband (UWB), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna. Exemplarily, the wireless communication module 160 may include a Bluetooth module, a Wi-Fi module, etc.

[0106] In some embodiments, a portion of the antenna of the electronic device 100 is coupled to the mobile communication module 150, and another portion of the antenna is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), millimeter wave (mmWave), BT, GNSS, WLAN, NFC, FM, UWB, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0107] In some embodiments, the Bluetooth (BT) module and WLAN module included in the wireless communication module 160 can transmit signals to detect or scan devices near the electronic device 100, so that the electronic device 100 can use wireless communication technologies such as Bluetooth or WLAN to discover nearby devices, establish wireless communication connections with nearby devices, and share data with nearby devices through the above connection. Among them, the Bluetooth (BT) module can provide solutions including one or more Bluetooth communications in classic Bluetooth or Bluetooth low energy (BLE). The WLAN module can provide solutions including one or more WLAN communications in Wi-Fi direct, Wi-Fi LAN or Wi-Fi softAP.

[0108] In some embodiments, the millimeter wave (mmWave) module included in the mobile communication module 150 can transmit or receive signals in a specified direction to detect or scan nearby devices of the electronic device 100 in the specified direction, so that the electronic device 100 can use mobile communication technologies such as millimeter waves to discover nearby devices in the specified direction, and establish a mobile communication connection or a wireless communication connection such as a WLAN direct connection with the nearby devices, and share data to the nearby devices through the above connection.

[0109] The electronic device 100 can realize the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute instructions to generate or change display information.

[0110] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0111] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0112] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0113] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0114] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0115] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG)-1, MPEG-2, MPEG-3, MPEG-4, etc.

[0116] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0117] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, data such as music, photos, and videos can be stored in the external memory card.

[0118] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can run the above instructions stored in the internal memory 121, so that the electronic device 100 executes the data sharing method provided in some embodiments of the present application, as well as various functional applications and data processing. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0119] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0120] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0121] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0122] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0123] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to realize collection of sound signals, noise reduction, identification of sound source, realization of directional recording function, etc.

[0124] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0125] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0126] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0127] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0128] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0129] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0130] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0131] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0132] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0133] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0134] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0135] The touch sensor 180K may also be referred to as a touch panel or a touch-sensitive surface. The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor may transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations may be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100, at a position different from that of the display screen 194.

[0136] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0137] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0138] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0139] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0140] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0141] Figure 2A The electronic device 100 shown in the example can display the various graphical user interfaces described in the following embodiments through the display screen 194. The electronic device 100 can detect touch operations in various graphical user interfaces through the touch sensor 180K, such as click operations in various graphical user interfaces (such as touch operations on icons, double-click operations), and for example, upward or downward sliding operations in various graphical user interfaces, or operations of performing circle gestures, etc. In some embodiments, the electronic device 100 can detect motion gestures performed by the user holding the electronic device 100, such as shaking the electronic device, through the gyroscope sensor 180B, the acceleration sensor 180E, etc. In some embodiments, the electronic device 100 can detect non-touch gesture operations through the camera 193 (such as a 3D camera, a depth camera).

[0142] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0143] Figure 3 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0144] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0145] The application layer can include a series of application packages.

[0146] like Figure 3 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0147] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0148] like Figure 3 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0149] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0150] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0151] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0152] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0153] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0154] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0155] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0156] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0157] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0158] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0159] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0160] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0161] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0162] A 2D graphics engine is a drawing engine for 2D drawings.

[0163] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0164] Figure 3 The software system shown involves application presentation using screen projection capabilities (such as video library, multi-tasking interface), as well as the application framework layer providing WLAN services, Bluetooth services, and the kernel and underlying layers providing WLAN Bluetooth capabilities and basic communication protocols.

[0165] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0166] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, timestamp of the touch operation, and other information). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch operation and the control corresponding to the touch operation is a control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0167] The following uses a mobile phone as an example of the above-mentioned electronic device 100 to describe the application scenarios involved in this application and some embodiments of implementing screen projection of the "multi-task management interface" on a mobile phone.

[0168] When using a mobile phone, a user can switch an application in the mobile phone from running in the foreground to running in the background. In addition, the user can control the mobile phone to display a multi-task management interface. The multi-task management interface is a graphical user interface used to display / manage applications running in the background on electronic devices such as smart phones and tablet computers. The graphical user interface can display thumbnails of applications currently running, and can also be called a "multi-task interface". This embodiment does not limit the name of the graphical user interface.

[0169] Figure 4A A graphical user interface on a mobile phone is exemplarily shown. The graphical user interface may be a home screen of the mobile phone, and may include: a status bar, a calendar indicator, a weather indicator, and a tray with common application icons.

[0170] Among them, a home button (Home button) 11 is arranged below the mobile phone screen, which can be used to receive user instructions. The Home button 11 can be a physical button or a virtual button.

[0171] Among them, Figure 4A and Figure 4B In response to the user's first operation, the mobile phone calls up the multitasking interface 30. Specifically, the first operation is that the user quickly presses the Home button 11 twice, and the mobile phone displays the multitasking interface. This embodiment does not limit the triggering method for the mobile phone to call up the multitasking interface, and the existing method can be referred to.

[0172] Figure 4B The multitasking interface 30 displayed on the mobile phone is shown as an example. The multitasking interface can support a variety of operations of the user on the currently running application, such as casting, cancelling casting, switching applications, viewing running applications, closing running applications, etc.

[0173] like Figure 4B As shown, the multitasking interface 30 may include a first display area 31 and a second display area 32. The first display area may include: a first electronic device icon 101', a second electronic device icon 102', and a third electronic device icon 103'; the second display area may include a thumbnail 201 of a first application, a thumbnail 202 of a second application, and a thumbnail 203 of a third application. The electronic device is an electronic device that the mobile phone points to and has established a wireless communication connection relationship, and the application thumbnail is formed by a screenshot of the interface when the application is switched to the background.

[0174] For example, the icons of the first electronic device icon 101', the second electronic device icon 102', and the third electronic device icon 103', the icons corresponding to the electronic devices within the pointing range of the mobile phone can be sorted according to a predetermined sorting rule. In some embodiments, the predetermined sorting rule can be based on the distance between the electronic device and the mobile phone. For example, three electronic device icons are displayed in the first display area 31, and the order indicates that the distance between the electronic devices corresponding to the three icons and the mobile phone is the first electronic device icon 101'<the second electronic device icon 102'<the third electronic device icon 103'. In addition, in some other embodiments, in the first display area 31, different electronic device icons can have different display effects. For example, the first electronic device icon 101' can be different from other icons in appearance (e.g., color, size, etc.) to prompt that the nearby device corresponding to the icon is closest to the electronic device or the signal strength from the nearby device is the highest. In some other embodiments, the above-mentioned predetermined sorting rule can be other rules such as the initials of the device ID, and this embodiment does not impose any restrictions on this.

[0175] Therefore, by utilizing the positioning between the mobile phone and other electronic devices to identify the electronic devices within the pointing range of the mobile phone, the target device that the user wants to project the screen can be quickly locked, and directional screen projection can be achieved, avoiding the trouble of users searching and identifying the target device among a large number of devices in a large range, and avoiding the situation of misselecting devices when projecting the screen.

[0176] It should be noted that the location recognition between the mobile phone and other electronic devices is implemented in different ways according to different hardware. In some embodiments, the location recognition between the mobile phone and other electronic devices can be achieved through the UWB chip pre-installed in the mobile phone and other electronic devices.

[0177] It is understandable that the electronic device icon can also be presented in other forms, such as the text message "printer". In addition, the electronic device icon can also include the account of the user using the device, for example, including the text message "MAC's phone", where "MAC" is the user account. In some other embodiments, the electronic device icon can also display the model of the electronic device or other custom information, such as "HUAWEI P30".

[0178] It should be noted that the mobile phone may be pointing to multiple electronic devices with which wireless communication connections have been established. However, due to the limitations of the mobile phone display interface, only some electronic device icons may be displayed in the multitasking interface. The user can trigger the mobile phone to display other electronic device icons by sliding.

[0179] It should be noted that there may be multiple applications running in the background, and due to the limitations of the electronic device display interface, only thumbnails of some applications may be displayed in the multitasking interface. Users can trigger the electronic device to display thumbnails of other applications by sliding.

[0180] In this embodiment, if Figure 4C As shown, in the first display area 31, several electronic device icons and the relative positions of their corresponding electronic devices and mobile phones can be displayed. Among them, the first display area 31 can display a coordinate system 110, which can include a center point 1111 and a horizontal axis 1112 and a vertical axis 1113. Among them, the center point 1111 can represent a mobile phone, the horizontal axis 1112 represents the left and right direction relative to the mobile phone, and the vertical axis 1113 represents the front and back direction relative to the mobile phone. The multitasking interface can also display the first electronic device icon 101', the second electronic device icon 102', and the third electronic device icon 103' to represent nearby electronic devices discovered by the mobile phone, and display the position / direction of these electronic devices relative to the mobile phone, that is, the position of different electronic device icons indicates the relative position of their corresponding electronic devices and the mobile phone. For example, the electronic device icon 103' is located in the upper right corner of the center point 1111, which indicates that the actual electronic device 103 is located in the upper right corner of the mobile phone. In addition, the distance between the electronic device icon and the center point 1111 indicates the distance between its corresponding electronic device and the mobile phone. The closer the icon is to the center point, the closer the corresponding electronic device is to the mobile phone. It is understandable that the electronic device icon may also be presented in other forms, such as a number or other information customized by the device user. In some other embodiments, the icon of the electronic device closer to the center point may have a different display effect from the icon of the electronic device farther from the center point, such as the icon closer to the center point may be different in appearance (such as icon size, color, etc.) from the icon farther from the center point.

[0181] In addition, in some implementations, the electronic device icon in the first display area can also display device information of the discovered electronic device, such as "HUAWEI P30" or "Tom's Notebook".

[0182] like Figure 4D As shown, the user performs a second operation, which is specifically dragging the thumbnail 201 of the first application to the first electronic device icon 101'. The second operation is used to start the screen projection function of the mobile phone and project the application that the user needs to project to the selected electronic device.

[0183] As an extension of the embodiment of the present application, Figure 4E As shown, the second operation may also be that the user first clicks the thumbnail 31 of the first application, and then clicks the first electronic device icon 101', thereby starting the screen projection function of the mobile phone. Alternatively, the user first clicks the first electronic device icon 101', and then clicks the thumbnail 31 of the first application. The click may be a single click, a double click, or a long press, which is not limited in the present embodiment of the application.

[0184] As an extension of the above embodiment, Figure 4F As shown, the second operation can also be that the user first clicks on the directional screen projection icon 120 to start, and then slides the thumbnail 201 of the first application in the direction of the first electronic device 101 that needs to receive the screen projection, thereby starting the screen projection function of the mobile phone. Among them, the first display area 31 can give arrow prompts based on the electronic devices detected in different directions. In some embodiments, the mobile phone determines the contact positions at the beginning and end of the user's sliding, respectively, and determines the user's sliding direction based on the contact positions at the beginning and end of the sliding. It can be understood that the method of determining the sliding direction is not limited to determining the contact positions at the beginning and end of the sliding. The mobile phone can also determine the sliding direction by recording the finger sliding trajectory and other methods. This embodiment does not impose any restrictions on this.

[0185] like Figure 5 As shown, in response to the user's second operation, the mobile phone starts the screen projection function and projects the first application to the first electronic device 101. Further, the thumbnail of the dragged first application is rebounded to the position of the second display area where it was before dragging, and the text "Screen projection is in progress" is displayed, and the third display area 33 of the multitasking interface is updated to display the control 301 ("Stop screen projection"). Further, the mobile phone can also display information including the content "The first application has been projected to the first electronic device" to facilitate the user to view the current specific screen projection status.

[0186] like Fig. 6A As shown, if the user projects the first application to the first electronic device 101 (as shown in Figure 5 As shown), if the first application needs to be switched to the second application, the second operation can be performed again in the multitasking interface to switch the application of the screen projection. Specifically, the user performs a third operation, which can be dragging the thumbnail 202 of the second application to the first electronic device icon 102'.

[0187] It is understandable that if the user needs to switch the first application to the second device 102 for screen projection after projecting the first application to the first electronic device 101, the user can drag the thumbnail 201 of the first application to the second electronic device icon 102' in the multitasking interface.

[0188] like Figure 6B As shown, in response to the user's third operation, the mobile phone can instruct the first device 101 to stop playing the first application and project the second application to the first electronic device 101. Specifically, the mobile phone may first send an end signal of "stop projecting" to the first device 101, and then transmit the multimedia data output by the second application to the first device 101. Thus, the playback of the first application on the first electronic device is switched to the playback of the second application on the first electronic device. In other words, the user only needs to perform one step in the multitasking interface to complete the switching of the projected application without multiple steps (for example, first open the first application, then choose to stop projecting, then open the second application and start projecting), so that the projected content can be changed very conveniently, user privacy is protected from leakage, and the user experience is improved.

[0189] like Fig. 7A As shown, the user performs a fourth operation, which is specifically dragging the thumbnail 201 of the first application to the second electronic device icon 102'. Figure 7B As shown, in response to the user's fourth operation, the mobile phone projects the first application to the second electronic device 102. Thus, after the mobile phone has projected the second application to the first electronic device, the user can also project the first application on the mobile phone to the second electronic device through the fourth operation, thereby realizing the convenient operation of projecting multiple applications to multiple electronic devices in the multi-tasking interface. At this time, the multimedia data output by the first application in the mobile phone is projected to the second device for playback, and the multimedia data output by the second application in the mobile phone is projected to the first application for playback. Users can enjoy the multimedia data played by different electronic devices at the same time, thereby improving the user experience.

[0190] like Fig. 8A As shown, the user performs the fifth operation, which is specifically dragging the thumbnail 201 of the first application to the control 501 ("stop screen projection"). Before the fifth operation, the first application has been projected to the second electronic device 102.

[0191] As an extension of this embodiment, the fifth operation can be sliding down or up, or selecting an application thumbnail first and then performing a circle gesture in the second display area, or selecting an application thumbnail within a fixed time (such as 1 second) after shaking the electronic device, or a voice control operation, that is, the user can say the voice command "stop screen projection". The specific form of the fifth operation is not limited in this embodiment.

[0192] like Figure 8B As shown, in response to the fifth operation of the user, the mobile phone instructs the second electronic device 102 to stop playing the first application. Alternatively, the mobile phone can stop sending the display data generated when running the first application to the first electronic device. In addition, after the first electronic device stops running the first application, the mobile phone can also switch the first application back to the mobile phone to continue displaying, so that the first application can be seamlessly connected between the mobile phone and the first electronic device.

[0193] As a refinement of this embodiment, after the fifth operation, before the first application stops projecting, the mobile phone may pop up a dialog box asking "Are you sure you want to stop projecting the first application?", which also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone stops projecting the first application; if the user selects "No", the mobile phone continues to project the first application.

[0194] For example, in a scenario (such as a multi-person conference scenario), a user needs to quickly and conveniently project an application to multiple electronic devices. Fig.9A As shown, the user performs the sixth operation, which is specifically to press and hold the thumbnail 201 of the first application with two fingers and slide upward for a distance L, and the distance L must be greater than a predetermined length so that the electronic device can recognize and determine the direction of the user's sliding. The sixth operation is used to activate the function of casting the mobile phone screen to multiple devices, and the application selected by the user can be cast to multiple electronic devices.

[0195] It should be noted that the two fingers in the sixth operation mentioned above can also be three fingers; long press can also be a single click or a double click; sliding upward can also be customized to slide in a certain direction, or slide to the first display area, etc.; this embodiment does not impose any restrictions on this.

[0196] In some embodiments, the mobile phone determines the touch point positions at the start and end of the user's sliding, and determines the user's sliding direction according to the touch point positions at the start and end of the sliding. It is understandable that the method of determining the sliding direction is not limited to determining the touch point positions at the start and end of the sliding. The mobile phone can also determine the sliding direction by recording the finger sliding trajectory, etc. This embodiment does not impose any limitation on this.

[0197] like Fig. 9B As shown, in response to the sixth operation of the user, the mobile phone starts the function of casting the screen to multiple devices, and casts the first application to the first electronic device 101, the second electronic device 102, and the third electronic device 103. In some embodiments, the multiple devices to which the mobile phone casts the screen can be all electronic devices with which the mobile phone points to establish a wireless communication connection relationship, or can be electronic devices corresponding to all electronic device icons displayed in the first display area in the multitasking interface to which the first application is cast.

[0198] Therefore, in a scenario where a single application needs to be projected to multiple devices, the user only needs to perform one step in the multitasking interface to project an application to multiple electronic devices. The multiple electronic devices can be electronic devices that the mobile phone points to and has established wireless communication connections. This makes projecting to multiple devices more convenient and faster while limiting the devices that need to be projected.

[0199] like Fig. 9C As shown, the user performs the seventh operation, which is specifically to press and hold the thumbnail 201 of the first application with two fingers and slide downward for a distance L, and the distance L must be greater than a predetermined length so that the electronic device can recognize and determine the direction of the user's sliding. The seventh operation is used to end the screen projection to multiple devices, and can simultaneously stop the screen projection of the application selected by the user to multiple devices.

[0200] like Fig.9D As shown, in response to the seventh operation of the user, the mobile phone instructs the first electronic device 101, the second electronic device 102, and the third electronic device 103 to stop playing the first application. Alternatively, the mobile phone can stop sending the display data generated when running the first application to the first electronic device 101, the second electronic device 102, and the third electronic device 103.

[0201] Therefore, in a scenario where you need to end screen projection of a single application to multiple devices, the user only needs to perform one step in the multitasking interface to stop multiple devices from projecting one application, making it more convenient and faster to end the projection of the application to multiple devices.

[0202] See also Fig.10 , Fig.10 The present application embodiment provides a flowchart of a multi-task interface projection method, which is applied to Figure 1 The electronic device 100 described above, below takes a mobile phone as an example of the electronic device 100; as shown in the figure, the multi-task interface projection method includes:

[0203] S401: In response to a first operation of a user, obtaining locations of a first electronic device and a second electronic device.

[0204] The first operation is used to open the multitasking interface of the mobile phone. For example, after the user quickly presses the Home button twice, the mobile phone displays the multitasking interface. For another example, after the user touches the multitasking button in the navigation bar, the mobile phone displays the multitasking interface. This embodiment does not limit the triggering method for the mobile phone to call up the multitasking interface, and the existing method can be referred to.

[0205] It is understandable that the wireless communication connection relationship established between the mobile phone and other electronic devices may be a wireless communication connection relationship established before responding to the first operation, and the wireless communication connection may be established through at least one of WIFI, Bluetooth, UWB ultra-wideband technology, infrared data transmission, wireless local area network, etc. After the user performs the first operation, the mobile phone may obtain the distance between the electronic device and the mobile phone through technologies such as antenna array or UWB ultra-wideband, and then determine the electronic device whose distance to the mobile phone is less than the distance threshold as a candidate electronic device, and display it in the first display area. The mobile phone may sort the discovered nearby devices according to a predetermined sorting rule and display the device information corresponding to each sorted nearby device in the first display area.

[0206] After the mobile phone finds the first electronic device and the second electronic device, it determines the relative positions of the above-mentioned nearby electronic devices and the mobile phone. For the specific process, please refer to Fig.17 Shown and described.

[0207] S402, determining the positions of the first electronic device icon and the second electronic device icon in the multi-tasking interface according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device, and the multi-tasking interface includes a first display area, and the first display area is used to display the first electronic device icon and the second electronic device icon.

[0208] Furthermore, the first electronic device icon and the second electronic device icon displayed in the first display area of ​​the multitasking interface correspond to the first electronic device and the second electronic device, which are the first electronic device and the second electronic device that have established a wireless communication connection relationship with the mobile phone, and the first electronic device and the second electronic device have a certain positional relationship with the mobile phone. They may be electronic devices with an established wireless communication connection relationship that are discovered in the direction indicated by the mobile phone. Thus, on the premise of establishing a wireless communication connection relationship between the devices, they can be further screened based on the positional relationship (such as direction, distance, etc.) between the mobile phone and the other devices, thereby reducing the number of electronic devices discovered by the mobile phone and improving the accuracy and efficiency of the user's selection of receiving devices.

[0209] Furthermore, if the indicated direction (posture) of the mobile phone changes, resulting in a change in the relative position of the nearby electronic device and the mobile phone, the electronic device in the first display area will also be updated accordingly. The user can control the indicated direction (posture) of the mobile phone, and the mobile phone can set different display effects for the device information corresponding to the nearby devices in the indicated direction among the nearby devices found in all directions, or only display the device information corresponding to the nearby electronic devices in the indicated direction.

[0210] It should be noted that the mobile phone may be pointing to multiple electronic devices with which wireless communication connections have been established. However, due to the limitations of the mobile phone display interface, only some electronic devices may be displayed in the multitasking interface. The user can trigger the mobile phone to display other electronic devices by sliding.

[0211] S403, obtaining a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the electronic device; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application, and the multitasking interface also includes a second display area, and the second display area is used to display a thumbnail of the first application and a thumbnail of the second application.

[0212] Furthermore, the application thumbnails displayed in the second display area of ​​the multitasking interface are formed by screenshots of the interface when the application is switched to the background.

[0213] It should be noted that there may be multiple applications running in the background, and due to the limitations of the mobile phone's multitasking interface, the multitasking interface may only display thumbnails of some applications. Users can trigger the phone to display thumbnails of other applications by sliding.

[0214] S404, displaying the multitasking interface.

[0215] Specifically, the multitasking interface includes a first display area and a second display area. The first display area displays a first electronic device icon and a second electronic device icon according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device. The second display area displays a thumbnail of a first application and a thumbnail of a second application; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application.

[0216] S405, in response to the user's second operation, projecting the multimedia data associated with the first application to the first electronic device.

[0217] The second operation is specifically to drag the thumbnail of the first application in the second display area to the first electronic device icon in the first display area, and then the mobile phone projects the multimedia data associated with the first application to the first electronic device.

[0218] As an extension of this embodiment, the second operation can be sliding up or down, or selecting an application thumbnail first and then performing a circle gesture in the second display area, or selecting an application thumbnail within a fixed time (such as 1 second) after shaking the electronic device, or a voice control operation, that is, the user can say the voice command "cast screen". The specific form of the first operation is not limited in this embodiment.

[0219] As a refinement of this embodiment, after the second operation is completed and before the screen projection function is started, the mobile phone may pop up a dialog box asking "Are you sure to project the first application to the first electronic device?", and the dialog box also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone starts the screen projection function; if the user selects "No", the mobile phone does not start the screen projection function.

[0220] It should be noted that in the screen projection process between the mobile phone and the electronic device in this embodiment, the data transmitted can support but is not limited to at least one of audio data, video data, picture data, web page data, and text data. For example, the mobile phone can send the display data output when running the WeChat application to the smart TV through the screen projection function, and the smart TV continues to display the display interface output by the WeChat application. For another example, if the WeChat program is installed in the smart TV, the mobile phone can send an instruction to open the WeChat application to the smart TV, so that the smart TV responds to the instruction to open the WeChat application installed by itself, thereby projecting the WeChat application on the mobile phone to the smart TV to continue running. Alternatively, when the mobile phone sends an instruction to open the WeChat application to the smart TV, it can also carry the running data of the current WeChat application in the instruction, for example, the running data is used to indicate that the current WeChat application is located in the chat interface of the contact Sam. In this way, after the smart TV opens the WeChat application installed by itself, it can automatically jump to the chat interface of the contact Sam according to the running data, so that the multimedia data can be seamlessly continued when projecting between multiple devices.

[0221] For example, after the mobile phone projects the multimedia data output by the first application from the mobile phone to the first electronic device for playback, the mobile phone can continue to run the first application to output the corresponding multimedia data. Alternatively, the mobile phone can also stop running the first application. Of course, after the mobile phone projects the first application to the first electronic device, the user can also perform various operations on the first application in the first electronic device, and the embodiments of the present application do not impose any restrictions on this.

[0222] As a refinement of this embodiment, in response to the user's second operation, the mobile phone starts the screen projection function, and the thumbnail of the first application dragged by the user to the first display area pops back to the second display area, and the text "Screen projection" is displayed. At the same time, the third display area of ​​the multitasking interface is updated to display the control ("Stop screen projection").

[0223] In this way, the user can select the application and the electronic device pointed to by the mobile phone through the above-mentioned operation method in the multitasking interface of the mobile phone, thereby completing the application screen projection to the electronic device, making the entire human-computer interaction process more natural and friendly, and improving the user experience.

[0224] See also Fig.11 , Fig.11 The present application embodiment provides a flowchart of a multi-task interface projection method, which is applied to Figure 1 The electronic device 100 described above, below takes a mobile phone as an example of the electronic device 100; as shown in the figure, the multi-task interface projection method includes:

[0225] S501: In response to a first operation of a user, obtain locations of a first electronic device and a second electronic device.

[0226] The first operation is used to open the multitasking interface of the mobile phone. For example, after the user quickly presses the Home button twice, the mobile phone displays the multitasking interface. For another example, after the user touches the multitasking button in the navigation bar, the mobile phone displays the multitasking interface. This embodiment does not limit the triggering method for the mobile phone to call up the multitasking interface, and the existing method can be referred to.

[0227] It is understandable that the wireless communication connection relationship established between the mobile phone and other electronic devices may be a wireless communication connection relationship established before responding to the first operation, and the wireless communication connection may be established through at least one of WIFI, Bluetooth, UWB ultra-wideband technology, infrared data transmission, wireless local area network, etc. After the user performs the first operation, the mobile phone may obtain the distance between the electronic device and the mobile phone through technologies such as antenna array or UWB ultra-wideband, and then determine the electronic device whose distance to the mobile phone is less than the distance threshold as a candidate electronic device, and display it in the first display area. The mobile phone may sort the discovered nearby devices according to a predetermined sorting rule and display the device information corresponding to each sorted nearby device in the first display area.

[0228] After the mobile phone finds the first electronic device and the second electronic device, it determines the relative positions of the above-mentioned nearby electronic devices and the mobile phone. For the specific process, please refer to Fig.17 Shown and described.

[0229] S502, determining the positions of the first electronic device icon and the second electronic device icon in the multitasking interface according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device, and the multitasking interface includes a first display area, and the first display area is used to display the first electronic device icon and the second electronic device icon.

[0230] Furthermore, the first electronic device icon and the second electronic device icon displayed in the first display area of ​​the multitasking interface correspond to the first electronic device and the second electronic device, which are the first electronic device and the second electronic device that have established a wireless communication connection relationship with the mobile phone, and the first electronic device and the second electronic device have a certain positional relationship with the mobile phone. They may be electronic devices with an established wireless communication connection relationship that are discovered in the direction indicated by the mobile phone. Thus, on the premise of establishing a wireless communication connection relationship between the devices, they can be further screened based on the positional relationship (such as direction, distance, etc.) between the mobile phone and the other devices, thereby reducing the number of electronic devices discovered by the mobile phone and improving the accuracy and efficiency of the user's selection of receiving devices.

[0231] Furthermore, if the indicated direction (posture) of the mobile phone changes, resulting in a change in the relative position of the nearby electronic device and the mobile phone, the electronic device in the first display area will also be updated accordingly. The user can control the indicated direction (posture) of the mobile phone, and the mobile phone can set different display effects for the device information corresponding to the nearby devices in the indicated direction among the nearby devices found in all directions, or only display the device information corresponding to the nearby electronic devices in the indicated direction.

[0232] It should be noted that the mobile phone may be pointing to multiple electronic devices with which wireless communication connections have been established. However, due to the limitations of the mobile phone display interface, only some electronic devices may be displayed in the multitasking interface. The user can trigger the mobile phone to display other electronic devices by sliding.

[0233] S503, obtaining a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the electronic device; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application, and the multitasking interface also includes a second display area, and the second display area is used to display a thumbnail of the first application and a thumbnail of the second application.

[0234] Furthermore, the application thumbnails displayed in the second display area of ​​the multitasking interface are formed by screenshots of the interface when the application is switched to the background.

[0235] It should be noted that there may be multiple applications running in the background, and due to the limitations of the mobile phone display interface, only thumbnails of some applications may be displayed in the multitasking interface. Users can trigger the phone to display thumbnails of other applications by sliding.

[0236] S504: Display the multi-tasking interface.

[0237] Specifically, the multitasking interface includes a first display area and a second display area. The first display area displays a first electronic device icon and a second electronic device icon according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device. The second display area displays a thumbnail of a first application and a thumbnail of a second application; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application.

[0238] S505, in response to a second operation of the user, projecting the multimedia data associated with the first application to the first electronic device.

[0239] The second operation is specifically to drag the thumbnail of the first application in the second display area to the first electronic device icon in the first display area, and then the mobile phone projects the multimedia data associated with the first application to the first electronic device.

[0240] As an extension of this embodiment, the second operation can be sliding up or down, or selecting an application thumbnail first and then performing a circle gesture in the second display area, or selecting an application thumbnail within a fixed time (such as 1 second) after shaking the electronic device, or a voice control operation, that is, the user can say the voice command "cast screen". The specific form of the first operation is not limited in this embodiment.

[0241] As a refinement of this embodiment, after the second operation is completed and before the screen projection function is started, the mobile phone may pop up a dialog box asking "Are you sure to project the first application to the first electronic device?", and the dialog box also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone starts the screen projection function; if the user selects "No", the mobile phone does not start the screen projection function.

[0242] It should be noted that in the screen projection process between the mobile phone and the electronic device in this embodiment, the data transmitted can support but is not limited to at least one of audio data, video data, picture data, web page data, and text data. For example, the mobile phone can send the display data output when running the WeChat application to the smart TV through the screen projection function, and the smart TV continues to display the display interface output by the WeChat application. For another example, if the WeChat program is installed in the smart TV, the mobile phone can send an instruction to open the WeChat application to the smart TV, so that the smart TV responds to the instruction to open the WeChat application installed by itself, thereby projecting the WeChat application on the mobile phone to the smart TV to continue running. Alternatively, when the mobile phone sends an instruction to open the WeChat application to the smart TV, it can also carry the running data of the current WeChat application in the instruction, for example, the running data is used to indicate that the current WeChat application is located in the chat interface of the contact Sam. In this way, after the smart TV opens the WeChat application installed by itself, it can automatically jump to the chat interface of the contact Sam according to the running data, so that the multimedia data can be seamlessly continued when projecting between multiple devices.

[0243] For example, after the mobile phone projects the multimedia data output by the first application from the mobile phone to the first electronic device for playback, the mobile phone can continue to run the first application to output the corresponding multimedia data. Alternatively, the mobile phone can also stop running the first application. Of course, after the mobile phone projects the first application to the first electronic device, the user can also perform various operations on the first application in the first electronic device, and the embodiments of the present application do not impose any restrictions on this.

[0244] As a refinement of this embodiment, in response to the user's second operation, the mobile phone starts the screen projection function, and the thumbnail of the first application dragged by the user to the first display area pops back to the second display area, and the text "Screen projection" is displayed. At the same time, the third display area of ​​the multitasking interface is updated to display the control ("Stop screen projection").

[0245] In this way, the user can select the application and the electronic device pointed to by the mobile phone through the above-mentioned operation method in the multitasking interface of the mobile phone, thereby completing the application screen projection to the electronic device, making the entire human-computer interaction process more natural and friendly, and improving the user experience.

[0246] S506, in response to a third operation of the user, stop projecting the multimedia data associated with the first application to the first electronic device, and project the multimedia data associated with the second application to the first electronic device.

[0247] Specifically, the third operation is to drag the thumbnail of the second application in the second display area to the first electronic device icon in the first display area, and then the mobile phone projects the multimedia data of the second application to the first electronic device.

[0248] As a refinement of this embodiment, after the third operation is completed and before the screen projection function is started, the mobile phone may pop up a dialog box asking "Are you sure to project the second application to the first electronic device?", and the dialog box also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone starts the screen projection function; if the user selects "No", the mobile phone does not start the screen projection function.

[0249] For example, after the mobile phone projects the multimedia data output by the second application from the mobile phone to the first electronic device for playback, the mobile phone can continue to run the second application to output the corresponding multimedia data. Alternatively, the mobile phone can also stop running the second application. Of course, after the mobile phone projects the second application to the first electronic device, the user can also perform various operations on the first application in the first electronic device, and the embodiments of the present application do not impose any restrictions on this.

[0250] As a refinement of this embodiment, in response to the user's third operation, the mobile phone starts the screen projection function, and the thumbnail of the second application dragged by the user to the first display area pops back to the second display area, and the text "Screen projection" is displayed. At the same time, the third display area of ​​the multitasking interface is updated to display the control ("Stop screen projection").

[0251] Therefore, users only need to perform one step in the multitasking interface to complete the switching of the projection application, without the need for multiple steps (for example, first open the first application, then choose to stop projection, and then open the second application to start projection). This makes it very convenient to change the projection content, protect user privacy from leakage, and improve user experience.

[0252] See also Fig.12 , Fig.12 The present application embodiment provides a flowchart of a multi-task interface projection method, which is applied to Figure 1 The electronic device 100 described above, below takes a mobile phone as an example of the electronic device 100; as shown in the figure, the multi-task interface projection method includes:

[0253] S601: In response to a first operation of a user, obtain locations of a first electronic device and a second electronic device.

[0254] The first operation is used to open the multitasking interface of the mobile phone. For example, after the user quickly presses the Home button twice, the mobile phone displays the multitasking interface. For another example, after the user touches the multitasking button in the navigation bar, the mobile phone displays the multitasking interface. This embodiment does not limit the triggering method for the mobile phone to call up the multitasking interface, and the existing method can be referred to.

[0255] It is understandable that the wireless communication connection relationship established between the mobile phone and other electronic devices may be a wireless communication connection relationship established before responding to the first operation, and the wireless communication connection may be established through at least one of WIFI, Bluetooth, UWB ultra-wideband technology, infrared data transmission, wireless local area network, etc. After the user performs the first operation, the mobile phone may obtain the distance between the electronic device and the mobile phone through technologies such as antenna array or UWB ultra-wideband, and then determine the electronic device whose distance to the mobile phone is less than the distance threshold as a candidate electronic device, and display it in the first display area. The mobile phone may sort the discovered nearby devices according to a predetermined sorting rule and display the device information corresponding to each sorted nearby device in the first display area.

[0256] After the mobile phone finds the first electronic device and the second electronic device, it determines the relative positions of the above-mentioned nearby electronic devices and the mobile phone. For the specific process, please refer to Fig.17 Shown and described.

[0257] S602, determining the positions of the first electronic device icon and the second electronic device icon in the multitasking interface according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device, and the multitasking interface includes a first display area, and the first display area is used to display the first electronic device icon and the second electronic device icon.

[0258] Furthermore, the first electronic device icon and the second electronic device icon displayed in the first display area of ​​the multitasking interface correspond to the first electronic device and the second electronic device, which are the first electronic device and the second electronic device that have established a wireless communication connection relationship with the mobile phone, and the first electronic device and the second electronic device have a certain positional relationship with the mobile phone. They may be electronic devices with an established wireless communication connection relationship that are discovered in the direction indicated by the mobile phone. Thus, on the premise of establishing a wireless communication connection relationship between the devices, they can be further screened based on the positional relationship (such as direction, distance, etc.) between the mobile phone and the other devices, thereby reducing the number of electronic devices discovered by the mobile phone and improving the accuracy and efficiency of the user's selection of receiving devices.

[0259] Furthermore, if the indicated direction (posture) of the mobile phone changes, resulting in a change in the relative position of the nearby electronic device and the mobile phone, the electronic device in the first display area will also be updated accordingly. The user can control the indicated direction (posture) of the mobile phone, and the mobile phone can set different display effects for the device information corresponding to the nearby devices in the indicated direction among the nearby devices found in all directions, or only display the device information corresponding to the nearby electronic devices in the indicated direction.

[0260] It should be noted that the mobile phone may be pointing to multiple electronic devices with which wireless communication connections have been established. However, due to the limitations of the mobile phone display interface, only some electronic devices may be displayed in the multitasking interface. The user can trigger the mobile phone to display other electronic devices by sliding.

[0261] S603, obtain a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the electronic device; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application, and the multitasking interface also includes a second display area, and the second display area is used to display a thumbnail of the first application and a thumbnail of the second application.

[0262] Furthermore, the application thumbnails displayed in the second display area of ​​the multitasking interface are formed by screenshots of the interface when the application is switched to the background.

[0263] It should be noted that there may be multiple applications running in the background, and due to the limitations of the mobile phone display interface, only thumbnails of some applications may be displayed in the multitasking interface. Users can trigger the phone to display thumbnails of other applications by sliding.

[0264] S604: Display the multi-tasking interface.

[0265] Specifically, the multitasking interface includes a first display area and a second display area. The first display area displays a first electronic device icon and a second electronic device icon according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device. The second display area displays a thumbnail of a first application and a thumbnail of a second application; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application.

[0266] S605: In response to a second operation of the user, the multimedia data associated with the first application is projected to the first electronic device.

[0267] The second operation is specifically to drag the thumbnail of the first application in the second display area to the first electronic device icon in the first display area, and then the mobile phone projects the multimedia data associated with the first application to the first electronic device.

[0268] As an extension of this embodiment, the second operation can be sliding up or down, or selecting an application thumbnail first and then performing a circle gesture in the second display area, or selecting an application thumbnail within a fixed time (such as 1 second) after shaking the electronic device, or a voice control operation, that is, the user can say the voice command "cast screen". The specific form of the first operation is not limited in this embodiment.

[0269] As a refinement of this embodiment, after the second operation is completed and before the screen projection function is started, the mobile phone may pop up a dialog box asking "Are you sure to project the first application to the first electronic device?", and the dialog box also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone starts the screen projection function; if the user selects "No", the mobile phone does not start the screen projection function.

[0270] It should be noted that in the screen projection process between the mobile phone and the electronic device in this embodiment, the data transmitted can support but is not limited to at least one of audio data, video data, picture data, web page data, and text data. For example, the mobile phone can send the display data output when running the WeChat application to the smart TV through the screen projection function, and the smart TV continues to display the display interface output by the WeChat application. For another example, if the WeChat program is installed in the smart TV, the mobile phone can send an instruction to open the WeChat application to the smart TV, so that the smart TV responds to the instruction to open the WeChat application installed by itself, thereby projecting the WeChat application on the mobile phone to the smart TV to continue running. Alternatively, when the mobile phone sends an instruction to open the WeChat application to the smart TV, it can also carry the running data of the current WeChat application in the instruction, for example, the running data is used to indicate that the current WeChat application is located in the chat interface of the contact Sam. In this way, after the smart TV opens the WeChat application installed by itself, it can automatically jump to the chat interface of the contact Sam according to the running data, so that the multimedia data can be seamlessly continued when projecting between multiple devices.

[0271] For example, after the mobile phone projects the multimedia data output by the first application from the mobile phone to the first electronic device for playback, the mobile phone can continue to run the first application to output the corresponding multimedia data. Alternatively, the mobile phone can also stop running the first application. Of course, after the mobile phone projects the first application to the first electronic device, the user can also perform various operations on the first application in the first electronic device, and the embodiments of the present application do not impose any restrictions on this.

[0272] As a refinement of this embodiment, in response to the user's second operation, the mobile phone starts the screen projection function, and the thumbnail of the first application dragged by the user to the first display area pops back to the second display area, and the text "Screen projection" is displayed. At the same time, the third display area of ​​the multitasking interface is updated to display the control ("Stop screen projection").

[0273] In this way, the user can select the application and the electronic device pointed to by the mobile phone through the above-mentioned operation method in the multitasking interface of the mobile phone, thereby completing the application screen projection to the electronic device, making the entire human-computer interaction process more natural and friendly, and improving the user experience.

[0274] S606: In response to a fourth operation of the user, the multimedia data associated with the second application is projected onto the second electronic device. Specifically, the fourth operation is dragging the thumbnail 202 of the second application onto the second electronic device icon 102'.

[0275] Thus, after the mobile phone has projected the first application to the first electronic device, the user can also project the second application on the mobile phone to the second electronic device through the fourth operation, realizing the convenient operation of projecting multiple applications to multiple electronic devices in the multitasking interface. At this time, the multimedia data output by the first application in the mobile phone is projected to the second device for playback, and the multimedia data output by the second application in the mobile phone is projected to the first application for playback. Users can enjoy the multimedia data played by different electronic devices at the same time, thereby improving the user experience.

[0276] See also Fig.13 , Fig.13 The present application embodiment provides a flowchart of a multi-task interface projection method, which is applied to Figure 1 The electronic device 100 described above, below takes a mobile phone as an example of the electronic device 100; as shown in the figure, the multi-task interface projection method includes:

[0277] S701: In response to a first operation of a user, obtain locations of a first electronic device and a second electronic device.

[0278] S702, determining the positions of the first electronic device icon and the second electronic device icon in the multi-tasking interface according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device, and the multi-tasking interface includes a first display area, and the first display area is used to display the first electronic device icon and the second electronic device icon.

[0279] S703, obtain a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the electronic device; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application, and the multitasking interface also includes a second display area, and the second display area is used to display a thumbnail of the first application and a thumbnail of the second application.

[0280] S704: Display the multi-tasking interface.

[0281] S705, in response to a second operation of the user, projecting the multimedia data associated with the first application to the first electronic device. The second operation is specifically dragging the thumbnail of the first application in the second display area to the first electronic device icon in the first display area, and then the mobile phone projects the multimedia data associated with the first application to the first electronic device.

[0282] As an extension of this embodiment, the second operation can be sliding up or down, or selecting an application thumbnail first and then performing a circle gesture in the second display area, or selecting an application thumbnail within a fixed time (such as 1 second) after shaking the electronic device, or a voice control operation, that is, the user can say the voice command "cast screen". The specific form of the first operation is not limited in this embodiment.

[0283] As a refinement of this embodiment, after the second operation is completed and before the screen projection function is started, the mobile phone may pop up a dialog box asking "Are you sure to project the first application to the first electronic device?", and the dialog box also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone starts the screen projection function; if the user selects "No", the mobile phone does not start the screen projection function.

[0284] For example, after the mobile phone projects the multimedia data output by the first application from the mobile phone to the first electronic device for playback, the mobile phone can continue to run the first application to output the corresponding multimedia data. Alternatively, the mobile phone can also stop running the first application. Of course, after the mobile phone projects the first application to the first electronic device, the user can also perform various operations on the first application in the first electronic device, and the embodiments of the present application do not impose any restrictions on this.

[0285] As a refinement of this embodiment, in response to the user's second operation, the mobile phone starts the screen projection function, and the thumbnail of the first application dragged by the user to the first display area pops back to the second display area, and the text "Screen projection" is displayed. At the same time, the third display area of ​​the multitasking interface is updated to display the control ("Stop screen projection").

[0286] In this way, the user can select the application and the electronic device pointed to by the mobile phone through the above-mentioned operation method in the multitasking interface of the mobile phone, thereby completing the application screen projection to the electronic device, making the entire human-computer interaction process more natural and friendly, and improving the user experience.

[0287] S706, in response to the user's fifth operation, stop projecting the multimedia data associated with the first application to the first electronic device. Specifically, in response to the user's second operation, the multitasking interface includes a control for stopping projection, and the fifth operation is specifically dragging the thumbnail of the first application to the control for stopping projection. Before the fifth operation, the first application is in a state of projection.

[0288] Furthermore, the mobile phone may stop sending the display data generated when the first application is running to the first electronic device, or the mobile phone may send a signal to the electronic device to end the screen projection. In addition, after the first electronic device stops running the first application, the mobile phone may switch the first application back to the mobile phone to continue displaying it, so that the first application can be seamlessly connected between the mobile phone and the first electronic device.

[0289] As an extension of this embodiment, the fifth operation can be sliding down or up, or selecting an application thumbnail first and then performing a circle gesture in the second display area, or selecting an application thumbnail within a fixed time (such as 1 second) after shaking the electronic device, or a voice control operation, that is, the user can say the voice command "stop screen projection". The specific form of the fifth operation is not limited in this embodiment.

[0290] As a refinement of this embodiment, after the fifth operation is completed and before the screen projection is stopped, the mobile phone may pop up a dialog box asking "Are you sure you want to stop the first application screen projection?", which also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone stops projecting the first application; if the user selects "No", the mobile phone continues to project the first application.

[0291] Therefore, the user only needs to perform one step in the multitasking interface to stop screen projection, without the need for multiple steps (such as opening the first application first and then choosing to stop screen projection), so that it can be ended very conveniently, improving the user experience.

[0292] See also Fig.14 , Fig.14 The present application embodiment provides a flowchart of a multi-task interface projection method, which is applied to Figure 1 The electronic device 100 described above, below takes a mobile phone as an example of the electronic device 100; as shown in the figure, the multi-task interface projection method includes:

[0293] S801, in response to a first operation of a user, obtaining locations of a first electronic device and a second electronic device.

[0294] S802, determining the positions of a first electronic device icon and a second electronic device icon in a multitasking interface according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device, and the multitasking interface includes a first display area, and the first display area is used to display the first electronic device icon and the second electronic device icon.

[0295] S803, obtain a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the electronic device; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application, and the multitasking interface also includes a second display area, and the second display area is used to display a thumbnail of the first application and a thumbnail of the second application.

[0296] S804: Display the multitasking interface.

[0297] S805, in response to a second operation of the user, projecting the multimedia data associated with the first application to the first electronic device. The second operation is specifically dragging the thumbnail of the first application in the second display area to the first electronic device icon in the first display area, and then the mobile phone projects the multimedia data associated with the first application to the first electronic device.

[0298] As an extension of this embodiment, the second operation can be sliding up or down, or selecting an application thumbnail first and then performing a circle gesture in the second display area, or selecting an application thumbnail within a fixed time (such as 1 second) after shaking the electronic device, or a voice control operation, that is, the user can say the voice command "cast screen". The specific form of the first operation is not limited in this embodiment.

[0299] As a refinement of this embodiment, after the second operation is completed and before the screen projection function is started, the mobile phone may pop up a dialog box asking "Are you sure to project the first application to the first electronic device?", and the dialog box also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone starts the screen projection function; if the user selects "No", the mobile phone does not start the screen projection function.

[0300] For example, after the mobile phone projects the multimedia data output by the first application from the mobile phone to the first electronic device for playback, the mobile phone can continue to run the first application to output the corresponding multimedia data. Alternatively, the mobile phone can also stop running the first application. Of course, after the mobile phone projects the first application to the first electronic device, the user can also perform various operations on the first application in the first electronic device, and the embodiments of the present application do not impose any restrictions on this.

[0301] As a refinement of this embodiment, in response to the user's second operation, the mobile phone starts the screen projection function, and the thumbnail of the first application dragged by the user to the first display area pops back to the second display area, and the text "Screen projection" is displayed. At the same time, the third display area of ​​the multitasking interface is updated to display the control ("Stop screen projection").

[0302] In this way, the user can select the application and the electronic device pointed to by the mobile phone through the above-mentioned operation method in the multitasking interface of the mobile phone, thereby completing the application screen projection to the electronic device, making the entire human-computer interaction process more natural and friendly, and improving the user experience.

[0303] S806, in response to the user's sixth operation, projecting the multimedia data associated with the first application to the first electronic device and the second electronic device. Specifically, the sixth operation may be sliding the thumbnail of the first application in a first direction, or long pressing the thumbnail 201 of the first application with two fingers and sliding it in the first direction for a distance L, and the distance L needs to be greater than a predetermined length so that the electronic device can recognize and determine the direction in which the user slides.

[0304] The first direction may be upward or other directions, and the sliding operation may be performed by sliding with two fingers or three fingers.

[0305] The sixth operation is used to start the function of projecting the mobile phone screen to multiple devices, which can project the application selected by the user to multiple electronic devices.

[0306] In some embodiments, the mobile phone determines the touch point positions at the start and end of the user's sliding, and determines the user's sliding direction according to the touch point positions at the start and end of the sliding. It is understandable that the method of determining the sliding direction is not limited to determining the touch point positions at the start and end of the sliding. The mobile phone can also determine the sliding direction by recording the finger sliding trajectory, etc. This embodiment does not impose any limitation on this.

[0307] That is to say, in a scenario where a single application needs to be projected to multiple devices, the user only needs to slide the application thumbnail in the first direction on the multitasking interface to project an application to multiple electronic devices. The multiple electronic devices can be electronic devices that the mobile phone is pointing to and have established wireless communication connections. This makes projecting to multiple devices more convenient and faster while limiting the devices that need to be projected.

[0308] See also Fig.15 , Fig.15 The present application embodiment provides a flowchart of a multi-task interface projection method, which is applied to Figure 1 The electronic device 100 described above, below takes a mobile phone as an example of the electronic device 100; as shown in the figure, the multi-task interface projection method includes:

[0309] S901: In response to a first operation of a user, obtain locations of a first electronic device and a second electronic device.

[0310] S902, determining the positions of the first electronic device icon and the second electronic device icon in the multitasking interface according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device, and the multitasking interface includes a first display area, and the first display area is used to display the first electronic device icon and the second electronic device icon.

[0311] S903, obtain a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the electronic device; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application, and the multitasking interface also includes a second display area, and the second display area is used to display a thumbnail of the first application and a thumbnail of the second application.

[0312] S904: Display the multitasking interface.

[0313] S905, in response to the user's second operation, projecting the multimedia data associated with the first application to the first electronic device. The second operation is specifically dragging the thumbnail of the first application in the second display area to the first electronic device icon in the first display area, and then the mobile phone projects the multimedia data associated with the first application to the first electronic device.

[0314] As an extension of this embodiment, the second operation can be sliding up or down, or selecting an application thumbnail first and then performing a circle gesture in the second display area, or selecting an application thumbnail within a fixed time (such as 1 second) after shaking the electronic device, or a voice control operation, that is, the user can say the voice command "cast screen". The specific form of the first operation is not limited in this embodiment.

[0315] As a refinement of this embodiment, after the second operation is completed and before the screen projection function is started, the mobile phone may pop up a dialog box asking "Are you sure to project the first application to the first electronic device?", and the dialog box also includes two buttons, "Yes" and "No". If the user selects "Yes", the mobile phone starts the screen projection function; if the user selects "No", the mobile phone does not start the screen projection function.

[0316] For example, after the mobile phone projects the multimedia data output by the first application from the mobile phone to the first electronic device for playback, the mobile phone can continue to run the first application to output the corresponding multimedia data. Alternatively, the mobile phone can also stop running the first application. Of course, after the mobile phone projects the first application to the first electronic device, the user can also perform various operations on the first application in the first electronic device, and the embodiments of the present application do not impose any restrictions on this.

[0317] As a refinement of this embodiment, in response to the user's second operation, the mobile phone starts the screen projection function, and the thumbnail of the first application dragged by the user to the first display area pops back to the second display area, and the text "Screen projection" is displayed. At the same time, the third display area of ​​the multitasking interface is updated to display the control ("Stop screen projection").

[0318] In this way, the user can select the application and the electronic device pointed to by the mobile phone through the above-mentioned operation method in the multitasking interface of the mobile phone, thereby completing the application screen projection to the electronic device, making the entire human-computer interaction process more natural and friendly, and improving the user experience.

[0319] S906, in response to the seventh operation of the user, stop projecting the multimedia data associated with the first application to the first electronic device and the second electronic device.

[0320] Specifically, the seventh operation can be sliding the thumbnail of the first application in the second direction, or long pressing the thumbnail 201 of the first application with two fingers and continuously sliding it in the second direction for a distance L, and the distance L needs to be greater than a predetermined length so that the electronic device can recognize and determine the direction in which the user slides.

[0321] The second direction may be downward or other directions, and the sliding operation may be performed by sliding with two fingers or three fingers.

[0322] The seventh operation is used to end screen projection to multiple devices, and can simultaneously stop the screen projection of the application selected by the user to multiple devices.

[0323] In some embodiments, the mobile phone determines the touch point positions at the start and end of the user's sliding, and determines the user's sliding direction according to the touch point positions at the start and end of the sliding. It is understandable that the method of determining the sliding direction is not limited to determining the touch point positions at the start and end of the sliding. The mobile phone can also determine the sliding direction by recording the finger sliding trajectory, etc. This embodiment does not impose any limitation on this.

[0324] Therefore, in a scenario where you need to end screen projection of a single application to multiple devices, the user only needs to perform one step in the multitasking interface to stop multiple devices from projecting one application, making it more convenient and faster to end the projection of the application to multiple devices.

[0325] In some embodiments, when the mobile phone starts the screen projection function, a communication connection is established with the electronic device receiving the screen projection, and after the communication connection is established, the data of the application selected for screen projection is transmitted. The transmitted data may support but is not limited to at least one of audio data, video data, picture data, web page data, and text data.

[0326] Fig.16 The exemplary embodiment shows the process of discovering all nearby devices and determining the relative positions of the nearby devices and the electronic device 100 after the electronic device opens the multitasking interface, and establishing a connection with the selected receiving device and sending data. Fig.16 As shown, when the electronic device 100 detects that the multitasking interface is turned on, it triggers the electronic device 100 to discover nearby devices. Figure 4A A first operation is performed in the graphical user interface shown to open a multitasking interface.

[0327] The electronic device 100 can discover nearby devices in the following ways:

[0328] Method 1: Use Wi-Fi Direct wireless communication technology to discover nearby devices

[0329] In some embodiments, the electronic device 100 may broadcast a probe request. After monitoring the probe request, the nearby device may respond with a probe response to notify its own existence. In other embodiments, the nearby device may periodically send a beacon frame. The electronic device 100 may discover nearby devices by monitoring the Beacon frames sent by the nearby devices.

[0330] That is to say, the electronic device 100 can actively discover nearby devices, or can passively discover nearby devices.

[0331] Method 2: Use Bluetooth wireless communication technology to discover nearby devices

[0332] In some embodiments, a Bluetooth device (such as a mobile phone, tablet computer, printer, etc. with a Bluetooth module) near the electronic device 100 can perform Bluetooth broadcasting. The electronic device 100 can perform Bluetooth scanning to scan the broadcast frames broadcast by nearby Bluetooth devices, thereby discovering nearby Bluetooth devices.

[0333] Method 3. Discover devices in the same Wi-Fi LAN

[0334] In some embodiments, the electronic device 100 can determine the IP address range of the LAN based on its own IP address and subnet mask in the LAN, and then discover the devices in the LAN by unicast polling. Not limited to this, the electronic device 100 can also discover the devices in the LAN by broadcasting messages or multicasting messages in the LAN.

[0335] The above-mentioned methods for electronic devices to discover nearby devices are not limited. In actual applications, electronic devices can also use other methods to discover nearby devices based on communication technologies such as Bluetooth, Wi-Fi Direct, Wi-Fi LAN, or millimeter wave, and this application does not impose any restrictions on this.

[0336] During the nearby device discovery process, the electronic device 100 obtains the identification information of the nearby device from the message received from the nearby device (such as a detection response, a beacon frame, or a Bluetooth broadcast frame, etc.), and then obtains the device information through the identification information. It can also obtain the device information of the nearby device from the message received from the nearby device.

[0337] After discovering the first electronic device 101, the second electronic device 102, and the third electronic device 103, the electronic device 100 determines the relative positions of the electronic devices 100 and the electronic device 100. The relative positions at least include information such as the angle of arrival (AoA). The specific process of the electronic device 100 determining the angle of arrival (AoA) of the nearby devices can be as follows: Fig.17 As shown. The electronic device 100 first sends a first request signal 1011 to the first electronic device 101, and the first request signal 1011 is used to request the electronic device 101 to send a first report signal 1012 to measure the angle of arrival (AoA). After receiving the first request signal 1011, the first electronic device 101 sends a first report signal 1012 to the electronic device 100, and the electronic device 100 uses an antenna array to receive the first report signal 1012, and estimates the angle of arrival (AoA) of the first electronic device 101 according to the phase difference of the first report signal 1012 received by different antennas. In some embodiments, the first report signal 1012 may specifically include a fixed frequency extension signal, which may be a data string containing a series of switching time slots and sampling time slots attached to the back of a normal data packet. When receiving the first report signal 1012, the receiving end switches different receiving antennas in a certain order, and determines the angle of arrival (AoA) according to the phase difference of the fixed frequency extension signals received by different antennas.

[0338] For example, if the millimeter wave antenna array of the electronic device 100 includes 8 millimeter wave antennas A1-A8, after sending the first request signal 1011, at time t1, only antenna A1 is used to receive the first report signal 1012 sent by the nearby device, and at time t2, only antenna A2 is used to receive the first report signal 1012 sent by the nearby device, and so on. After the 8 antennas respectively receive the first report signal 1012 sent by the nearby devices, the electronic device 100 calculates the angle of arrival (AoA) of the nearby device based on the phase difference of the first report signal 1012 received by the 8 antennas, that is, the relative angle between the nearby device and the electronic device 100.

[0339] In some other embodiments, the electronic device 100 and the first electronic device 101 may use an angle of departure (AoD) measurement method to determine the relative angle.

[0340] Next, the electronic device 100 uses the antenna array to respectively determine the angle of arrival (AoA) of the second electronic device 102 and the third electronic device 103 at different time intervals using the same steps as measuring the angle of arrival (AoA) of the electronic device 101. In some other embodiments, the electronic device 100 can simultaneously determine the angle of arrival (AoA) of multiple devices or all devices in the nearby devices.

[0341] In some other embodiments, the electronic device 100 can configure the antenna array to perform beamforming, receive broadcast messages sent by nearby devices only in one direction, or send a detection signal in that direction, and the nearby devices that receive the detection signal send a detection response to the electronic device 100 to discover nearby devices in that direction. The electronic device 100 obtains the device information of the nearby devices and corresponds the above-mentioned nearby devices to the direction. Next, the electronic device 100 configures the antenna array to perform beamforming to discover nearby devices in another direction. That is, by polling in various directions, the discovery of nearby devices is completed and the relative angles of nearby devices are obtained.

[0342] In some other embodiments, the relative position may also include distance information. The electronic device 100 may determine the distance between the electronic device 100 and the discovered nearby device by using the signal strength RSSI of the message received from the nearby device during the nearby device discovery process. For example, the electronic device 100 may determine the distance between the electronic device 100 and the discovered nearby device by using the signal strength RSSI of the broadcast message or response message received from the nearby device during the nearby device discovery process. Other ranging methods in the prior art, such as ToF / ToA / TDoA ranging, may also be used to determine the distance between the electronic device 100 and the discovered nearby device. This embodiment does not impose any restrictions on this.

[0343] After detecting the first operation of the user, the electronic device 100 determines the relative positions of the first electronic device 101, the second electronic device 102, and the third electronic device 103 and updates the multitasking interface, and the first display area in the multitasking interface includes the device information of the first electronic device 101, the second electronic device 102, and the third electronic device 103 and the relative positions of the above devices and the electronic device 100. The displayed multitasking interface can refer to Figure 4B or Figure 4C In addition, in Figure 4C In the multitasking interface shown in FIG. 1 , the closer the nearby device is to the electronic device, the closer its corresponding icon is to the center point 1111. Figure 4B or Figure 4C The multitasking interface shown monitors the user's selection of the application to be projected and the electronic device to receive the projection.

[0344] In some other embodiments, the electronic device 100 can determine the signal strength RSSI of the signal received from the nearby device during the nearby device discovery process. For example, the electronic device 100 can determine the signal strength RSSI of the broadcast message or response message received from the nearby device during the nearby device discovery process. After determining the relative angle between the nearby device and the electronic device 100 and detecting the user's operation of sharing data, the multitasking interface is updated and the device information, relative angle and signal strength of the nearby device are included in the multitasking interface. In addition, in the case of Figure 4C In the multitasking interface shown, the stronger the signal strength of a nearby device, the closer its corresponding icon is to the center point 1111.

[0345] like Fig.16 As shown, if the user selects the first electronic device 101 as the receiving device for projecting the first application through the second operation. Next, the electronic device 100 receives the second operation of the user, establishes a communication connection with the first electronic device 101 through wireless technology such as Bluetooth or WiFi, and starts sending data. The specific process of establishing the communication connection above may be: the electronic device 100 sends a connection establishment request message to the first electronic device 101, and after the first electronic device 101 receives the connection establishment request message, it replies to the electronic device 100 with a connection response message to establish a communication connection between the electronic device 100 and the first electronic device 101. Next, the electronic device 100 sends the data of the first application selected by the user to the first electronic device 101, such as images, videos, documents, application installers, application information, etc.

[0346] In some other embodiments, if the user indicates direction information when opening the multitasking interface, for example, the electronic device can set different display effects for the device information corresponding to the nearby devices in the indicated direction among the nearby devices found in all directions, or in the case of Figure 4B Or the multitasking interface shown in 4C only displays device information corresponding to nearby devices in the indicated direction.

[0347] It should be noted that in the above embodiments, a mobile phone is used as the electronic device 100, and the multitasking interface is set in the mobile phone for example. It is understandable that the above electronic device can also be any device in the device group such as a tablet computer, a notebook computer, etc., and the present application embodiment does not impose any limitation on this.

[0348] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0349] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A screen projection method, characterized in that: include: In response to a first operation by the user, the mobile phone obtains locations of the first electronic device and the second electronic device; The mobile phone determines the positions of the first electronic device icon and the second electronic device icon on the multitasking interface according to the positions of the first electronic device and the second electronic device; wherein the first electronic device icon is used to identify the first electronic device, and the second electronic device icon is used to identify the second electronic device, and the multitasking interface includes a first display area, and the first display area is used to display the first electronic device icon and the second electronic device icon; The mobile phone obtains a thumbnail of a first application and a thumbnail of a second application; the first application and the second application are applications currently running on the mobile phone; wherein the thumbnail of the first application is used to identify the first application, and the thumbnail of the second application is used to identify the second application, and the multitasking interface further includes a second display area, and the second display area is used to display the thumbnail of the first application and the thumbnail of the second application; The mobile phone displays the multitasking interface; In response to the user's second operation, the mobile phone projects the multimedia data associated with the first application to the first electronic device.

2. The method according to claim 1, characterized in that The mobile phone obtains the locations of the first electronic device and the second electronic device specifically as follows: The mobile phone obtains the positions of the first electronic device and the second electronic device through ultra-wideband positioning technology or antenna array technology.

3. The method according to claim 1, characterized in that: The first display area includes a coordinate system, and the mobile phone determines the positions of the first electronic device icon and the second electronic device icon on the multitasking interface according to the positions of the first electronic device and the second electronic device, specifically: The mobile phone determines the positions of the first electronic device icon and the second electronic device icon in the coordinate system in the multitasking interface according to the positions of the first electronic device and the second electronic device.

4. The method according to claim 1, characterized in that: The second operation is specifically dragging the thumbnail of the first application to the first electronic device icon.

5. The method according to any one of claims 1 to 4, characterized in that: After the mobile phone projects the multimedia data associated with the first application to the first electronic device, the method further includes: In response to a third operation by the user, the mobile phone stops projecting the multimedia data associated with the first application to the first electronic device, and projects the multimedia data associated with the second application to the first electronic device.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to the user's fourth operation, the mobile phone projects the multimedia data associated with the second application to the second electronic device.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to the user's fifth operation, the mobile phone stops projecting the multimedia data associated with the first application to the first electronic device.

8. The method according to claim 7, characterized in that After responding to the second operation of the user, the multitasking interface further includes a control for stopping screen projection; The fifth operation is specifically to drag the thumbnail of the first application to the control for stopping screen projection.

9. The method according to claim 1, characterized in that: The method further comprises: In response to a sixth operation by the user, the mobile phone projects the multimedia data associated with the first application to the first electronic device and the second electronic device.

10. The method according to claim 9, characterized in that The sixth operation is specifically sliding the thumbnail of the first application in the first direction.

11. The method according to claim 9, characterized in that The method further comprises: In response to the user's seventh operation, the mobile phone stops projecting the multimedia data associated with the first application to the first electronic device and the second electronic device.

12. The method according to claim 11, characterized in that The seventh operation is specifically sliding the thumbnail of the first application in the second direction.

13. A mobile phone, characterized in that: include: A touch screen, wherein the touch screen comprises a touch-sensitive surface and a display screen; one or more processors; one or more memories; And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, and when the instructions are executed by the mobile phone, the mobile phone implements the screen projection method as described in any one of claims 1-12.

14. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, characterized in that: When the instruction is executed on a mobile phone, the mobile phone executes the screen projection method as described in any one of claims 1 to 12.

Images