A method and an electronic device for screen mirroring

Abstract

The present application provides a screen mirroring method, characterized in that the method includes: a first electronic device plays multimedia content, and the camera of the first electronic device operates in a low-power mode; the camera of the first electronic device acquires at least one first image frame in the low-power operation mode; when it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal working mode and acquires at least one second image frame; when it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device and sends the multimedia content to the second electronic device, so that the second electronic device plays the multimedia content. The screen mirroring method in the embodiments of the present application helps to improve the user experience.

Description

Technical Field

[0001] This application relates to the field of electronic devices, and more particularly, to a method and an electronic device for screen mirroring. Background Art

[0002] With the increase in the number of user electronic devices, data sharing between devices has become increasingly frequent. A mobile phone can play videos, display documents, pictures, application interfaces, or web pages. Since the display screen of a mobile phone is small, when the content displayed on the mobile phone needs to be shown to others, the content displayed on the mobile phone can be projected onto other electronic devices (such as a TV, a computer, or another mobile phone) through screen mirroring technology. For example, a user can mirror a video on the mobile phone to the TV, or mirror a song to the speaker. Summary of the Invention

[0003] This application provides a method and an electronic device for screen mirroring, which helps to improve the user experience.

[0004] In a first aspect, a method for screen mirroring is provided. The method includes: a first electronic device plays multimedia content, and the camera of the first electronic device operates in a low-power mode; the camera of the first electronic device acquires at least one first image frame in the low-power operation mode; when it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal mode and acquires at least one second image frame; when it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device and sends the multimedia content to the second electronic device, so that the second electronic device plays the multimedia content.

[0005] The first electronic device uses a low-power camera to capture an image and detects that the image contains a preset electronic device, thereby triggering screen mirroring. When a user shares content between different electronic devices, the user can directly interact with the shared device, solving the problem that the user cannot find the shared device because they do not know the device name, making content sharing between devices more direct and natural. In addition, it solves the problem of one-step sharing regardless of whether it is close or far away, and content sharing can be achieved without walking in front of the shared device.

[0006] In combination with the first aspect, in some possible implementation manners of the first aspect, the camera is a front camera.

[0007] Considering that in most scenarios, the front camera of the user faces the user's face, using the front camera to capture images in the low-power mode can reuse the camera used for face recognition. On the other hand, if the rear camera is used, since the environmental images captured by the rear camera change too fast, the first electronic device has to continuously capture different images and detect whether a preset electronic device is included, which will cause too much computational pressure on the first electronic device. Therefore, using the front camera to capture images can improve the user experience and save power consumption.

[0008] Combined with the first aspect, in some possible implementation manners of the first aspect, the camera of the first electronic device operates in a normal working mode, including: the camera of the first electronic device captures image frames in a high-resolution mode.

[0009] After detecting that the captured image contains a preset electronic device, switching to the high-resolution mode to capture images can improve the success rate of image recognition.

[0010] Combined with the first aspect, in some possible implementation manners of the first aspect, the camera of the first electronic device captures at least one first image frame in the low-power operation mode, including: the frame rate of the camera capturing at least one first image frame in the low-power operation mode does not exceed 10 frames per second.

[0011] The frame rate of the camera of the first electronic device capturing at least one first image frame in the low-power operation mode not exceeding 10 frames per second can reduce power consumption.

[0012] Combined with the first aspect, in some possible implementation manners of the first aspect, detecting that the preset object is a preset second electronic device includes: performing image recognition on the captured image and recognizing that the object in the captured image is the preset second electronic device.

[0013] Detecting the captured image through an image recognition algorithm can improve the efficiency of image detection.

[0014] Combined with the first aspect, in some possible implementation manners of the first aspect, image recognition can be performed on the images captured by the camera through the neural network computing processor of the first electronic device.

[0015] Performing image recognition by quickly processing the images captured by the camera through the neural network computing processor of the local device can perform image recognition in real time and can also perform image recognition without a network, which can improve the recognition efficiency.

[0016] In combination with the first aspect, in some possible implementations of the first aspect, the collected image can be uploaded to the server for image recognition, where the server can perform image recognition on the image through a neural network computing processor.

[0017] By using the neural network computing processor of the server to perform image recognition on the images collected by the camera, the computing advantages of the server can be fully utilized, and the computing pressure on the local machine can be reduced.

[0018] In combination with the first aspect, in some possible implementations of the first aspect, when performing image recognition on the collected image and recognizing that the object in the collected image is a preset second electronic device, it may further include:

[0019] Matching the image recognition result with the electronic device mapping table and matching to the preset second electronic device

[0020] By establishing an electronic device mapping table and matching the image recognition result with the electronic device mapping table, the probability of misjudgment caused by image recognition errors can be reduced.

[0021] In combination with the first aspect, in some possible implementations of the first aspect, the electronic device mapping table includes:

[0022] The image of the preset second electronic device and the device information of the preset second electronic device, where the device information includes the name or device model of the second electronic device.

[0023] Recording the device information of the preset second electronic device in the electronic device mapping table can facilitate the establishment of a connection between the electronic device and the preset electronic device.

[0024] In combination with the first aspect, in some possible implementations of the first aspect, the electronic device mapping table can be pre-set by the user in advance, can be generated by the electronic device according to the user's screen mirroring usage habits, or can be set by the electronic device manufacturer before the electronic device leaves the factory.

[0025] In combination with the first aspect, in some possible implementations of the first aspect, before the camera of the first electronic device collects at least one first image frame in the low-power operation mode, the method further includes:

[0026] The second processor of the local machine sends a first message to the first processor of the local machine to control the camera to collect images.

[0027] The first electronic device processor may include a first processor and a second processor. The first processor may be an auxiliary operation chip, a coprocessor or an auxiliary processor. By using the coprocessor to execute different tasks, the burden on the application processor of the first electronic device can be reduced, the load on the main processor can be decreased, and thus the battery life can be extended. Images in the low-power mode can be received and recognized at any time with low power consumption.

[0028] In combination with the first aspect, in some possible implementation manners of the first aspect, before the first electronic device establishes a connection with the second electronic device, the method further includes: the first electronic device searches for and discovers the second electronic device.

[0029] In combination with the first aspect, in some possible implementation manners of the first aspect, the first electronic device searches for and discovers the second electronic device, including one or more combinations of the following methods: discovering a third electronic device through a local area network broadcast, and comparing the device information of the second electronic device with the device information of the third electronic device, or discovering the third electronic device through Bluetooth, and comparing the device information of the second electronic device with the device information of the third electronic device, or discovering the third electronic device through WIFI direct connection, and comparing the device information of the second electronic device with the device information of the third electronic device;

[0030] According to different scenarios, the first electronic device can dynamically select different methods to search for and discover a preset electronic device, which can enhance data transmission stability and improve search efficiency.

[0031] In combination with the first aspect, in some possible implementation manners of the first aspect, the first electronic device establishes a connection with the second electronic device, including: the first electronic device establishes a P2P connection with the second electronic device.

[0032] By establishing a P2P link and performing a screen mirroring service on the P2P channel, the anti-interference ability can be enhanced and the screen mirroring experience can be improved.

[0033] In combination with the first aspect, in some possible implementation manners of the first aspect, the multimedia content includes at least one of video, audio or picture;

[0034] In a second aspect, an electronic device is provided, including: a camera, where the camera includes a low-power mode and a normal working mode; at least one processor; a memory storing instructions, which when executed by the at least one processor, cause the electronic device to execute the screen mirroring method according to the first aspect and the possible implementation directions of the first aspect.

[0035] In a third aspect, a screen mirroring method is provided. The method includes: the camera of the first electronic device operates in a low-power mode; the camera of the first electronic device captures at least one first image frame in the low-power operation mode; when it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal mode and captures at least one second image frame; when it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device; the first electronic device receives a multimedia file sent by the second electronic device, where the content of the multimedia file is the multimedia content being played by the second electronic device; the first electronic device plays the multimedia file.

[0036] In a fourth aspect, an electronic device is provided, including: a camera, where the camera includes a low-power mode and a normal operation mode; at least one processor; a memory storing instructions, when the instructions are executed by the at least one processor, enabling the electronic device to execute the screen mirroring method in the third aspect above.

[0037] In a fifth aspect, a screen mirroring system is provided, including a first electronic device and a second electronic device; the camera of the first electronic device operates in a low-power mode; the camera of the first electronic device captures at least one first image frame in the low-power operation mode; when it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal mode and captures at least one second image frame; when it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device; the second electronic device sends the multimedia content being played to the first electronic device; the first electronic device plays the multimedia content.

[0038] In a sixth aspect, a computer storage medium is provided, including computer instructions, when the computer instructions run on an electronic device, enabling the electronic device to execute the screen mirroring method in any possible design in any of the above aspects.

[0039] In a seventh aspect, a computer program product is provided, when the computer program product runs on a computer, enabling the computer to enable the electronic device to execute the screen mirroring method in any possible design in any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic structural diagram of the electronic device provided by an embodiment of the present application.

[0041] Figure 2 is a software structure block diagram of the electronic device provided by an embodiment of the present application.

[0042] Figure 3 is a schematic system block diagram provided by an embodiment of the present application

[0043] Figure 4A is a schematic diagram of a graphical user interface provided by an embodiment of the present application.

[0044] Figure 4B is a schematic diagram of a graphical user interface provided by an embodiment of the present application.

[0045] Figure 4C is a schematic diagram of a graphical user interface provided by an embodiment of the present application.

[0046] Figure 5 is a schematic diagram of a graphical user interface provided by an embodiment of the present application.

[0047] Figure 6 is a schematic diagram of a graphical user interface provided by an embodiment of the present application.

[0048] Figure 7A - 7C is a schematic diagram of a screen mirroring method provided by an embodiment of the present application.

[0049] Figure 8 is a schematic diagram of a screen mirroring method provided by an embodiment of the present application.

[0050] Figure 9 is a schematic diagram of a screen mirroring method provided by an embodiment of the present application.

[0051] Figures 10 - 11 is a schematic diagram of a graphical user interface provided by an embodiment of the present application.

[0052] Figures 12 - 14 is a schematic flowchart provided by an embodiment of the present application. Detailed implementation manners

[0053] Next, the technical solutions in the embodiments of the present application will be described in conjunction with the accompanying drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may represent A or B; herein, "and / or" is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0054] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this embodiment, unless otherwise specified, "a plurality of" means two or more.

[0055] The terms used in the description of the invention herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the invention.

[0056] Figure 1 Schematically shows a structural diagram of an electronic device 100 having a display screen and at least one camera (e.g., a front camera and / or a rear camera).

[0057] The electronic device 100 may include at least one of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, or a smart city device. The specific type of the electronic device 100 is not particularly limited in the embodiments of the present application.

[0058] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) connector 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0059] It can be understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0060] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. In some embodiments, the processor may include a first processor 1101 (such as a cooperative processor) and a second processor 1102 (such as an application processor).

[0061] The processor may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0062] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory may store instructions or data that have been used by or are frequently used by the processor 110. If the processor 110 needs to use such instructions or data, it can directly call them from this memory. This avoids repeated accesses and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0063] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 may be connected to modules such as a touch sensor, an audio module, a wireless communication module, a display, a camera, etc. through at least one of the above interfaces.

[0064] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present application are only illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0065] The USB connector 130 is an interface that complies with the USB standard specifications and can be used to connect the electronic device 100 and peripheral devices. Specifically, it can be a Mini USB connector, a Micro USB connector, a USB Type C connector, etc. The USB connector 130 can be used to connect a charger to charge the electronic device 100, or to connect other electronic devices to transfer data between the electronic device 100 and other electronic devices. It can also be used to connect headphones to output the audio stored in the electronic device through the headphones. This connector can also be used to connect other electronic devices, such as VR devices, etc. In some embodiments, the standard specifications of the Universal Serial Bus can be USB1.x, USB2.0, USB3.x, and USB4.

[0066] The charging management module 140 is used to receive the charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input from the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0067] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the input from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0068] The wireless communication function of the electronic device 100 can be implemented through antenna 1, antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0069] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0070] The mobile communication module 150 may provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be disposed in the same device.

[0071] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and disposed in the same device as the mobile communication module 150 or other functional modules.

[0072] The wireless communication module 160 may provide wireless communication solutions applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0073] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other electronic devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0074] Electronic device 100 may implement a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0075] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or more display screens 194.

[0076] The electronic device 100 can implement the camera function through the camera module 193, ISP, video codec, GPU, display screen 194, application processor AP, neural network processor NPU, etc.

[0077] The camera module 193 can be used to collect color image data and depth data of the photographed object. The ISP can be used to process the color image data collected by the camera module 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera sensor. The optical signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be disposed in the camera module 193.

[0078] In some embodiments, the camera module 193 can be composed of a color camera module and a 3D sensing module.

[0079] In some embodiments, the photosensitive element of the camera of the color camera module can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats.

[0080] In some embodiments, the 3D sensing module may be a (time of flight, TOF) 3D sensing module or a structured light 3D sensing module. Among them, structured light 3D sensing is an active depth sensing technology. The basic components of a structured light 3D sensing module may include an infrared (Infrared) emitter, an IR camera module, etc. The working principle of a structured light 3D sensing module is to first emit a light spot with a specific pattern to the object to be photographed, then receive the light coding of the light spot pattern on the surface of the object, and then compare the similarities and differences with the original projected light spot, and use the triangulation principle to calculate the three-dimensional coordinates of the object. The three-dimensional coordinates include the distance between the electronic device 100 and the object to be photographed. Among them, TOF 3D sensing can be an active depth sensing technology. The basic components of a TOF 3D sensing module may include an infrared (Infrared) emitter, an IR camera module, etc. The working principle of a TOF 3D sensing module is to calculate the distance (i.e., depth) between the TOF 3D sensing module and the object to be photographed by the time of infrared return, so as to obtain a 3D depth of field map.

[0081] The structured light 3D sensing module can also be applied to fields such as face recognition, motion sensing game consoles, and industrial machine vision detection. The TOF 3D sensing module can also be applied to fields such as game consoles, augmented reality (AR) / virtual reality (VR), etc.

[0082] In some other embodiments, the camera module 193 can also be composed of two or more cameras. These two or more cameras may include a color camera, and the color camera can be used to collect color image data of the object to be photographed. These two or more cameras can use stereo vision technology to collect depth data of the object to be photographed. Stereo vision technology is based on the principle of human eye parallax. Under natural light, images of the same object are taken from different angles through two or more cameras, and then operations such as triangulation are performed to obtain the distance information between the electronic device 100 and the object to be photographed, that is, depth information.

[0083] In some embodiments, the electronic device 100 may include one or more camera modules 193. Specifically, the electronic device 100 may include one front camera module 193 and one rear camera module 193. Among them, the front camera module 193 is usually used to collect color image data and depth data of the photographer facing the display screen 194, and the rear camera module can be used to collect color image data and depth data of the subject (such as a person, a landscape, etc.) that the photographer is facing.

[0084] In some embodiments, the CPU, GPU, or NPU in the processor 110 may process the color image data and depth data collected by the camera module 193. In some embodiments, the NPU may identify the color image data collected by the camera module 193 (specifically, the color camera module) through a neural network algorithm based on the skeleton point recognition technology, such as a convolutional neural network algorithm (CNN), to determine the skeleton points of the photographed person. The CPU or GPU may also run the neural network algorithm to determine the skeleton points of the photographed person based on the color image data. In some embodiments, the CPU, GPU, or NPU may also be used to confirm the figure of the photographed person (such as body proportions and the fatness or thinness of the body parts between the skeleton points) according to the depth data collected by the camera module 193 (which may be a 3D sensing module) and the identified skeleton points, and may further determine the body beautification parameters for the photographed person. Finally, the captured image of the photographed person is processed according to the body beautification parameters, so that the figure of the photographed person in the captured image is beautified. How to perform body beautification processing on the image of the photographed person based on the color image data and depth data collected by the camera module 193 will be introduced in detail in subsequent embodiments and will not be elaborated here.

[0085] The digital signal processor is used to process digital signals and can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0086] The video codec is used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0087] The NPU is a neural-network (NN) computing processor. By learning from the biological neural network structure, such as learning from the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0088] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card. Or files such as music and videos are transferred from the electronic device to the external memory card.

[0089] The internal memory 121 can be used to store computer-executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional methods or data processing of the electronic device 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor.

[0090] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0091] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0092] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music through the speaker 170A, or output an audio signal for hands-free calls.

[0093] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by placing the receiver 170B close to the human ear.

[0094] The microphone 170C, also known as a "microphone" or "transmitter", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with their mouth to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0095] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0096] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0097] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and controls the lens to move in the opposite direction to cancel the shaking of the electronic device 100, thereby achieving anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenarios.

[0098] The barometric pressure sensor 180C is used to measure barometric pressure. In some embodiments, the electronic device 100 calculates the altitude according to the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0099] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. When the electronic device is a foldable electronic device, the magnetic sensor 180D can be used to detect the folding or unfolding of the electronic device, or the folding angle. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the flip, features such as automatic flip unlocking can be set.

[0100] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0101] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance through infrared or laser. In some embodiments, in a shooting scenario, the electronic device 100 can use the distance sensor 180F to measure distance to achieve rapid focusing.

[0102] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode may be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from a nearby object. When the intensity of the detected reflected light is greater than a threshold value, it can be determined that there is an object near the electronic device 100. When the intensity of the detected reflected light is less than the threshold value, the electronic device 100 may determine that there is no object near the electronic device 100. The electronic device 100 may use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear during a call, so as to automatically turn off the screen to achieve the purpose of power saving. The proximity light sensor 180G can also be used in the holster mode and the pocket mode to automatically unlock and lock the screen.

[0103] The ambient light sensor 180L can be used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is blocked, for example, the electronic device is in the pocket. When it is detected that the electronic device is blocked or in the pocket, some functions (such as the touch function) can be disabled to prevent misoperation.

[0104] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access the application lock, fingerprint taking pictures, fingerprint answering calls, etc.

[0105] The temperature sensor 180J is used to detect the temperature. In some embodiments, the electronic device 100 executes a temperature processing strategy using the temperature detected by the temperature sensor 180J. For example, when the temperature detected by the temperature sensor 180J exceeds a threshold value, the electronic device 100 reduces the performance of the processor to reduce the power consumption of the electronic device to implement thermal protection. In other embodiments, when the temperature detected by the temperature sensor 180J is lower than another threshold value, the electronic device 100 heats the battery 142. In still other embodiments, when the temperature is lower than yet another threshold value, the electronic device 100 can boost the output voltage of the battery 142.

[0106] The touch sensor 180K, also known as the "touch control device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as the "touch control screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from where the display screen 194 is located.

[0107] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals of the vibrating bone mass of the human vocal part. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulsation signals. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone to form a bone conduction earphone. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bone mass of the human vocal part acquired by the bone conduction sensor 180M to implement the voice function. The application processor can parse out heart rate information based on the blood pressure pulsation signals acquired by the bone conduction sensor 180M to implement the heart rate detection function.

[0108] The button 190 can include a power-on button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The electronic device 100 can receive button inputs and generate key signal inputs related to the user settings and function control of the electronic device 100.

[0109] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations acting on different regions of the display screen 194 can also correspond to different vibration feedback effects for the motor 191. Different application scenarios (such as time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0110] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0111] The SIM card interface 195 is used to connect to a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact with and separation from the electronic device 100. The electronic device 100 can support one or more SIM card interfaces. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The types of multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0112] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device 100.

[0113] Figure 2 It is a software structure block diagram of the electronic device 100 in the embodiments of this application.

[0114] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely the application layer, the application framework layer, Android runtime (ART) and native C / C++ libraries, the Hardware Abstract Layer (HAL), and the kernel layer.

[0115] The application layer can include a series of application packages.

[0116] As Figure 2 shown, the application packages can include applications such as cameras, galleries, calendars, calls, maps, navigation, WLAN, Bluetooth, music, videos, and text messages.

[0117] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0118] As Figure 2As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, etc.

[0119] The window manager provides the Window Manager Service (WMS). The WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.

[0120] The content provider is used to store and obtain data, and make this data accessible to applications. The data can include videos, images, audio, incoming and outgoing calls, browsing history and bookmarks, phone books, etc.

[0121] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures.

[0122] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0123] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify that the download is complete, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background-running application, or a notification that appears in the form of a dialogue window on the screen. For example, it can prompt text information in the status bar, emit a prompt sound, vibrate the electronic device, blink the indicator light, etc.

[0124] The activity manager can provide the Activity Manager Service (AMS). The AMS can be used for the startup, switching, scheduling of system components (such as activities, services, content providers, broadcast receivers), and the management and scheduling of application processes.

[0125] The input manager can provide the Input Manager Service (IMS). The IMS can be used to manage system inputs, such as touch screen input, key input, sensor input, etc. The IMS retrieves events from input device nodes and distributes the events to appropriate windows through interaction with the WMS.

[0126] The Android Runtime includes the core libraries and the Android Runtime. The Android Runtime is responsible for converting source code into machine code. The Android Runtime mainly includes the Ahead-of-Time (AOT) compilation technology and the Just-in-Time (JIT) compilation technology.

[0127] The core libraries are mainly used to provide the functions of the basic Java class libraries, such as libraries for basic data structures, mathematics, IO, tools, databases, networks, etc. The core libraries provide APIs for users to develop Android applications.

[0128] The native C / C++ libraries can include multiple functional modules. For example: Surface Manager, Media Framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0129] Among them, the Surface Manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The Media Framework supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides the drawing and operation of 2D and 3D graphics in applications. SQLite provides a lightweight relational database for the applications of the electronic device 100.

[0130] The Hardware Abstraction Layer runs in the user space, encapsulates the kernel layer drivers, and provides call interfaces to the upper layer.

[0131] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0132] Next, in combination with the scenario of capturing a photo with the front camera, the working processes of the software and hardware of the electronic device 100 will be exemplarily described.

[0133] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation and the control corresponding to the click operation as the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a static image or video through the camera 193.

[0134] With the increase in the number of user electronic devices, data sharing among electronic devices has become increasingly frequent. For example, users can cast a video on their mobile phone to a TV, or cast audio to a speaker, thus obtaining different user experiences.

[0135] Before explaining the embodiments, relevant terms related to screen mirroring are introduced first.

[0136] Source side and Sink side: For example, when casting resources on a mobile phone to a TV, two electronic devices are involved, one is the mobile phone and the other is the TV. In this process, the initiating side of the casting (the mobile phone) can be called the Source side, and the TV can be called the Sink side.

[0137] Same-source screen mirroring: A screen mirroring method in which the interfaces of the Source side and the Sink side are exactly the same.

[0138] Different-source screen mirroring: A screen mirroring method in which the interfaces of the Source side and the Sink side are different.

[0139] Mirror projection: A method of same-source projection that projects the screen of the Source side onto the Sink side like a mirror. The specific method can include the Source side performing screen recording, then sending the recording result to the Sink side in real time, and then playing it on the Sink side.

[0140] Resource projection: A method of different-source projection. The Source side (e.g., a mobile phone) transmits the resources of the local device (such as audio, video, picture and other media resources) to the Sink side in the form of files or streams, and then plays them through the player of the Sink side. It can be divided into online resource projection and local resource projection. Specifically, for online resource projection, the Source side sends the URL of the online resource to the Sink side, and the Sink side obtains the content of the media resource according to the URL and plays it on the Sink side; for local resource projection, the Source side sends the local resource to the Sink side, and the Sink side plays it. During the projection process, the Source side can control and adjust the Sink side, such as play / pause, volume adjustment, brightness adjustment, language adjustment, etc. Playback control: Playback control refers to controlling the playback state of the Sink side through the Source side (e.g., a mobile phone) in the way of resource projection.

[0141] Reverse control: In the scenario of mirror screen mirroring, when the screen of the Source side (e.g., a mobile phone) is projected onto the Sink side, the projected Source side (e.g., a mobile phone) screen can be directly operated on the Sink side through a mouse, keyboard, or touch screen, achieving the effect of operating the Source side.

[0142] LAN connection: A connection method for screen mirroring. The Source and Sink ends need to be connected to the same local area network, which can be a wired local area network or a wireless local area network.

[0143] P2P (Peer-To-Peer) connection: Also known as device direct connection, it is a connection method where the Source and Sink directly establish a WIFI (Wireless Fidelity) direct connection for data exchange without passing through a router.

[0144] Common screen mirroring protocols include Miracast, DLNA, Cast+, AirPlay, Chromecast, etc.

[0145] It should be understood that the screen mirroring method described in this application can be applied to large screen mirroring (for example, a mobile phone projecting to a TV, a projector, a set-top box), PC (Personal computer) collaboration (for example, after a mobile phone discovers a device through near-field communication, it establishes a P2P connection with the PC, and then projects the mobile phone screen mirror image, and also supports reverse control, that is, controlling the mobile phone through the PC), Pad (Portable android device) collaboration, in-vehicle screen mirroring, etc.

[0146] Figure 3 Shown is an exemplary system diagram of some embodiments. System 30 may include electronic device 100, electronic device 102, electronic device 103, and electronic device 104. It should be understood that in some embodiments, System 30 may further include more or fewer electronic devices. It should be understood that in some embodiments, electronic device 104 may be a wireless router, or a customer premise equipment (CPE) that can provide wireless network access, or any other wireless access point (WAP) or wired network access point. In some embodiments, electronic device 100, electronic device 102, and electronic device 103 may be connected to the same local area network (wired local area network or wireless local area network) through electronic device 104.

[0147] The screen mirroring method provided by the embodiments of this application can be applied to electronic devices such as mobile phones, tablets, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs) that include cameras. The embodiments of this application do not impose any restrictions on the specific types of electronic devices.

[0148] Figure 4A 、 Figure 4B 、 Figure 4C And Figure 5 Shown is an exemplary graphical user interface where the user manually clicks on the screen mirroring control to trigger screen mirroring.

[0149] A graphical user interface (GUI) is displayed on the screen of the electronic device 100, and the graphical user interface includes a screen mirroring control. It should be understood that the graphical user interface can be the application interface of a certain application (for example, Huawei Music as shown in Figure 4B , or Huawei Video as shown in Figure 4C ), or it can be the drop-down menu bar of the electronic device 100 as shown in Figure 4A (it should be understood that the term "drop-down menu bar" here is illustrative and does not represent a specific limitation on the name of this graphical user page). Schematically, in Figure 4C , the content displayed on GUI4002 includes video 402 (the video can be in a paused or playing state). It should be understood that the content displayed on GUI4002 can also include controls 404 (such as volume adjustment controls, play / pause controls, fast forward controls, rewind controls), video title 406, video list 408, etc. It should be understood that the content on GUI 4002 is illustrative.

[0150] The electronic device 100 detects that the screen mirroring control is activated (for example, the user clicks on the screen mirroring control). The electronic device 100 detects that it is in a network and searches for screen-mirrorable electronic devices in the local area network in the local area network. It should be understood that the screen mirroring control can be different on different graphical user interfaces (for example, the screen mirroring control can be the "Multi-screen Interaction" control 412 as shown in Figure 4B , the "Wireless Screen Mirroring" as shown in Figure 4A , or as shown in Figure 4CThe control shown (406). Schematically, the electronic device 100, the electronic device 102, and the electronic device 103 are within the same local area network. The electronic device 100 can search for the electronic device 102 and the electronic device 103 within the same local area network, and obtain the device information of the electronic device 102 and the electronic device 103 for display on the electronic device 100 (for example, Figure 4B In the GUI4004 in, the device information of the electronic devices 102 and 103 in the local area network is displayed in a list form on the electronic device 100), where the device information may include the name of the device.

[0151] When the electronic device 100 detects that the user selects the electronic device 102, the electronic device 100 establishes a connection with the electronic device 102, and the electronic device 100 transmits the content displayed on the local machine (at least one of video, text, or picture) to the electronic device 102 for display, or the electronic device 100 transmits the audio played on the local machine to the electronic device 102 for playback.

[0152] In some embodiments, before receiving the content displayed on the local machine of the electronic device 100, the electronic device 102 may receive a connection request sent by the electronic device 100, and the electronic device 102 may pop up a window to request whether to agree to establish a connection.

[0153] Optionally, after the electronic device 100 establishes a connection with the electronic device 102, the electronic device 102 may only display a part of the content displayed on the electronic device 100 (for example, the screen mirroring method may be a resource projection method). Schematically, as Figure 5 shown, the image user interface displayed on the electronic device 100 includes a video 402 (the video may be in a paused or playing state). After the electronic device 100 establishes a connection with the electronic device 102, the electronic device 102 may only display the video 402 (the video may be in a paused or playing state). Further, after the electronic device 100 establishes a connection with the electronic device 102, the electronic device 102 may only display the video 402, and the electronic device 100 stops displaying the video 402. Further, after the electronic device 100 stops displaying the video 402, the user can control the video 402 played on the electronic device 102 on the electronic device 100.

[0154] Optionally, after the electronic device 100 transmits the audio played on the local machine to the electronic device 102 for playback, the electronic device 100 may stop playing the audio. Further, after the electronic device 100 stops playing the audio, the user can control the audio played on the electronic device 102 on the electronic device 100.

[0155] Optionally, the electronic device 100 may mirror and transmit the graphical user interface displayed on the local device to the electronic device 102, and the graphical user interface displayed on the electronic device 102 is the same as that on the electronic device 100 (for example, the screen mirroring method may be the mirror projection method). Further, after the user issues a command to switch pages on the electronic device 100, the electronic device 100 switches to display the second graphical user interface, and the electronic device 102 receives the data sent by the electronic device 100 and switches to display the third graphical user interface, where the second graphical user interface is the same as the third graphical user interface. For example, when the electronic device 100 switches to display the local desktop, the electronic device 102 also switches to display the desktop of the electronic device 100.

[0156] However, triggering screen mirroring by clicking on a control and manually selecting another electronic device for screen mirroring is inconvenient for the user, and the user needs to perform multiple steps to share the content on the electronic device 100 to other electronic devices. For example, the user needs to first click on the screen mirroring icon, and then needs to search for the electronic device to be screen mirrored in the list of available screen mirroring devices. A very troublesome problem for the user is that since there are many available screen mirroring devices, it is difficult for the user to find the device they want to screen mirror in the list of available screen mirroring devices, because it is difficult to distinguish different electronic devices only by name. In other words, when there are many similar devices, the user may not know which device name needs to be screen mirrored.

[0157] Secondly, for screen mirroring on some application program user interfaces, since additional button icons need to be set on the application interface, and in the limited screen space, the content is often closely arranged. The screen mirroring icon set in such a limited space is difficult for the user to notice, which will impose a burden on the user. In addition, setting too many icons on the application program user interface is not suitable for immersive pages and affects the user experience.

[0158] As Figure 6 shown is a schematic diagram of the electronic device 100 and other electronic devices triggering screen mirroring through near-field connection. The electronic device 100 may establish a near-field connection with the near-field tag of the electronic device 102 (for example, the electronic device 102 is a TV, and the near-field tag is placed inside the remote control) to start screen mirroring.

[0159] After the electronic device 100 establishes a connection with the electronic device 102, the electronic device 100 transmits the content (videos, texts, pictures, etc.) displayed on the local device or the audio played on the local device to the electronic device 102. In some embodiments, before receiving the content, the electronic device 102 may receive a connection request sent by the electronic device 100, and the electronic device 102 may pop up a window to request whether to agree to establish a connection.

[0160] After the electronic device 100 detects a near-field connection and transmits the content (videos, audios, texts, pictures, etc.) displayed on the local device to the electronic device 102, the display mode of the electronic device 100 can refer to Figure 4A , Figure 4B , Figure 4C and Figure 5 the content of the described embodiment.

[0161] However, near-field interaction is more suitable for short-distance application scenarios. For example, it is suitable for a mobile phone to project its screen to a laptop, but it is not very suitable for scenarios where a mobile phone interacts with electronic devices such as a TV or a speaker at a long distance. For example, if you want to project the screen to a TV, you need to find the TV remote control first or walk in front of the TV to establish a near-field connection. Similarly, if you want to synchronize the audio on the mobile phone to the speaker for playback, the user also needs to walk in front of the speaker to establish a near-field connection.

[0162] Figure 7A - 7C Figures 15A-15C are a set of schematic diagrams in which the electronic device 100 captures an image through a camera (e.g., a front camera or a rear camera), and performs image recognition on the captured image, and triggers screen projection after recognizing a preset electronic device.

[0163] Image Recognition is a technology that, based on deep learning and big data, uses a computer to analyze and understand images in order to identify various different patterns of targets and objects. Based on deep learning technology, it can accurately recognize the visual content in images, can provide various object, scene, and concept labels, has capabilities such as object detection and attribute recognition, accurately recognizes and understands image content, constructs an intelligent system, and improves business efficiency.

[0164] It should be understood that the preset electronic device described herein can be an electronic device in the electronic device mapping table pre-established by the user. In some embodiments, the preset electronic device can also be an electronic device in the electronic device mapping table preset when the electronic device 100 leaves the factory. The electronic device mapping table can also be statistically generated by the electronic device 100 or the server according to the user's screen projection usage habits.

[0165] When the user shares content through screen projection, the user will consider whether privacy will be leaked. For example, the user is more willing to project the screen to share content at home or in the office area, but in other public places such as shopping malls and stations, the user does not want to project the screen to share.

[0166] Therefore, the user can customize the electronic devices that the user wants to project the screen to. For example, the user can create an electronic device mapping table for electronic devices such as TV A, computer B, speaker C at home, and computer D in the office. Then, the electronic devices in the electronic device mapping table are all preset electronic devices.

[0167] Schematically, the user can establish an electronic device mapping table by taking pictures of the electronic device. The user can take pictures of different electronic devices and save the taken pictures in the local device. In some embodiments, the user can also input the model number of the electronic device, download the picture of the electronic device from the server, and establish the electronic device mapping table. In some other embodiments, the electronic device mapping table can be obtained through training by a neural network model.

[0168] When the electronic device 100 turns on the camera (front camera or rear camera) and aims the camera at an object, the electronic device 100, through image recognition technology, recognizes that the object belongs to a preset electronic device in the electronic device mapping table pre-established by the user, then the electronic device 100 starts further verification (for example, detecting whether the electronic device 100 and the recognized electronic device are in the same local area network). It should be noted that when the camera of the electronic device 100 is aimed at an object, it can continuously collect pictures without the user taking a picture of the object. At this time, the electronic device 100 can extract one or more image frames containing the object from the cache and use the image frame for image recognition.

[0169] When the electronic device 100 fails to recognize an electronic device through the images collected by the camera, or the recognized electronic device does not belong to the preset electronic device, the electronic device 100 does not start verification. Taking the electronic device 100 as an example for schematic illustration. The electronic device 100 can use the camera of the local device to collect images of objects in the environment (including electronic devices, tables, chairs, human faces, etc.).

[0170] Step 1: The electronic device 100 collects images through the camera.

[0171] In some embodiments, the electronic device 100 can use the front camera of the local device to continuously (for example, periodically) collect images. Optionally, the front camera can be an ultra-low power consumption camera carried by the electronic device 100 (the camera operates in a low power consumption mode), where the ultra-low power consumption means that its power consumption is much lower than that of the conventional front camera and rear camera on the current electronic device (such as a mobile phone) (such as a 1-megapixel, 5-megapixel, or 10-megapixel camera), and usually can collect image frames at a very low resolution to save power. Since the resolution of the ultra-low power consumption camera is very low, it will not affect the user's privacy. It should be understood that the ultra-low power consumption camera can switch between a low power consumption mode (for example, the collected images are of low resolution) and a normal mode (for example, the collected images are of high resolution).

[0172] As Figure 1The hardware structure diagram of the electronic device 100 shown. The processor 110 of the electronic device 100 may include a first processor 1101 and a second processor 1102. The first processor 1101 may be an auxiliary operation chip, which can be a coprocessor or an auxiliary processor, used to relieve the burden of the application processor of the electronic device 100, execute preset processing tasks, such as processing image or video data, or sensing and measuring motion data, etc., to reduce the load of the main processor, thereby extending the battery life. The coprocessor in the embodiment of the present invention can receive and identify images in the low-power mode at any time with lower power consumption. It can be understood that this is only an illustrative description here. According to the specific type of the electronic device 100, in fact, both the first processor 1101 and the second processor 1102 can be ARM architecture processors or both X86 architecture processors, or processors of other architectures. Further, the first processor 1101 and the second processor 1102 can also be processing units with different performances and functions integrated on the same processor component, that is, the combination of different types of processors is not specifically limited in the embodiment of the present invention. The second processor 1102 may be an application processor (which can integrate an image information processor or include an image information processing unit), and serve as the main processor of the electronic device 100 to be responsible for the display of the screen 193, the playback of audio and video, various applications, voice calls, and data transmission, etc. When the user presses the power button to start the terminal, the second processor 1102 starts to run so that the electronic device 100 can be used normally; when the user turns off the power button, in order to reduce the power consumption of the second processor 1102, at this time the second processor 1102 will enter the sleep state, and in the embodiment of the present invention, only the first processor 1101 with lower power consumption can work normally at this time.

[0173] The screen mirroring method can be applied to the electronic device 100 including a first processor, a second processor, and a front camera 1931. Among them, the first processor 1101 and the second processor 1102 are respectively connected to the front camera 1931 through the front camera interface, and the first processor 1101 and the second processor 1102 are electrically connected to each other and to the screen 193 respectively.

[0174] Optionally, when the second processor detects that the front camera of the electronic device 100 is in the low-power mode, it can send a first message to the first processor.

[0175] It should be understood that the low-power mode described in the embodiments of the present application refers to a mode in which the front camera is not in the normal photo-taking or shooting mode, that is, the user does not enter the normal shooting mode of the front camera through a photo-taking related application at this time. The non-low-power mode may refer to a mode in which the user enables the normal shooting function of the front camera through a related photo-taking application, or the front camera uses a high-resolution mode for image acquisition. It can be understood that when the electronic device 100 is turned on from the shutdown state, obviously the front camera is still in the low-power mode at this time (because it must be turned on first, and then it is possible to turn on the normal shooting function of the camera).

[0176] Therefore, when the second processor detects the state from shutdown to startup and knows that the front camera is in the low-power mode; or when the second processor detects that the front camera exits from the non-low-power mode (normal working mode) to the low-power mode (not in the normal working mode), the second processor will send a first message to the first processor indicating that the front camera is in the low-power mode. It can also be understood that the first processor may be started together with the second processor when turning on from the shutdown state, or may be started by the second processor when sending the first message to the first processor.

[0177] Optionally, when the electronic device 100 displays the application interface of certain applications (such as browser applications, video call applications, game applications, etc.), the second processor may send a first message to the first processor. For example, when the electronic device 100 detects that the displayed application interface of the local device is the video chat interface of an instant messaging application, the second processor may send a first message to the first processor.

[0178] Optionally, when the electronic device 100 plays a multimedia file (such as a video, audio, picture, etc.), and the front camera of the electronic device 100 is in the low-power mode, the second processor may send a first message to the first processor.

[0179] Optionally, when the electronic device 100 is in the running state, the second processor may send a first message to the first processor.

[0180] After receiving the first message, the first processor controls the front camera to continuously collect images.

[0181] It should be understood that the second processor may periodically send a first message to the first processor to control the front camera to continuously collect images.

[0182] For example, in Figure 7AIn this case, there is a video being played on the application interface displayed by the electronic device 100, and the electronic device 100 is playing the video in the GUI 4002 shown in Figure 4. The second processor may send a first message to the first processor. After receiving the first message, the first processor controls the front camera to continuously capture images. The user turns the front camera of the electronic device 100 towards the electronic device 102, and the image captured by the front camera includes the image of the electronic device 102.

[0183] Optionally, the front camera may capture images at a certain frame rate (for example, not greater than 10 frames per second (fps)). Capturing images at a low frame rate by the front camera can reduce power consumption because the purpose of the coprocessor receiving the low-power mode images is to determine whether the currently captured images contain a preset electronic device based on the images captured in the low-power mode. Therefore, there is no need for clearer or higher-frame-rate images, and only the ability to determine whether there is preset feature information is required, thus avoiding wasting image resources and resulting in wasted system power consumption. When the front camera is an ultra-low-power camera (the camera operates in the low-power mode), when an object similar to an electronic device is detected in the image frames captured by the ultra-low-power camera, the electronic device 100 can enable the high-resolution mode of the ultra-low-power camera to capture images, or can also enable the conventional camera to capture high-resolution images, thereby facilitating image recognition and analysis by the electronic device.

[0184] In some embodiments, the electronic device 100 may only be equipped with a conventional front camera and a rear camera. The electronic device 100 can determine the posture of the electronic device 100 through the gyroscope and / or accelerometer in the device itself. When it is detected that the posture of the electronic device 100 presents a specific posture, such as when the electronic device 100 presents an almost vertical posture (such as Figure 7A ), it is speculated that the electronic device may be about to perform screen mirroring. At this time, the electronic device 100 determines whether to turn on the front camera or the rear camera according to the posture of the device itself. Optionally, the electronic device 100 may use the front camera to capture image frames.

[0185] In some embodiments, the electronic device 100 may also display the application interface of the camera application, detect a predefined gesture input by the user (for example, the user activates the "take a photo" control to take a photo), and capture an image through the camera (front camera or rear camera).

[0186] In some embodiments, the electronic device 100 may provide a control for selecting to turn on or off the intelligent sensing screen mirroring function on the setting page. As Figure 10 shown, the user can choose to click on the control 711 to turn on or off the intelligent sensing screen mirroring function of the electronic device. It should be understood that the "intelligent sensing screen mirroring" refers to Figures 7 to Figure 9A schematic name for the screen mirroring method in the above embodiment. Further, the electronic device 100 can also provide different ways of image acquisition. Schematically, the user can click Figure 10 the control 712 in Figure 11 to enter the setting page of the image acquisition method, as shown in Figure 11 . In Figure 10 , the user can turn on or off the low-power camera image acquisition mode by clicking the control 811, or turn on or off the camera application photo-taking mode by clicking the control 812. It should be understood that Figure 11 the elements on the graphical user interface are schematic. In other embodiments, Figure 10 and Figure 11 the graphical user interface may include more or fewer elements.

[0187] Step 2: The electronic device 100 performs image recognition through an identification algorithm

[0188] After the electronic device 100 captures an image through the camera (for example, an image captured by the front camera or an image taken after the user turns on the camera application), it can perform image recognition through an identification algorithm.

[0189] Optionally, the electronic device 100 can quickly process the image captured by the camera through the neural-network (NN) computing processor of the device itself to perform image recognition.

[0190] Optionally, in some embodiments, the electronic device 100 can upload the captured image to the server for image recognition, where the server can recognize the image through the neural-network (NN) computing processor.

[0191] Optionally, after the user opens the camera application of the electronic device 100 and captures an image of the electronic device 102 in the environment, the electronic device 100 can send a request to ask whether to perform image recognition.

[0192] Because the purpose of the user opening the camera application to take a photo is not necessarily for screen mirroring. By setting the step of "the electronic device 100 can send a request to ask whether to perform image recognition", it can be combined with the user's needs to judge whether to perform screen mirroring, which can improve the user experience and increase the user's trust. Optionally, the user can set a control in the electronic device 100 (for example, Figure 11 the control 812 shown), and when the control is in the on state, after the electronic device 100 detects that the user opens the camera application to take a photo, it automatically performs image recognition.

[0193] Step 3: The electronic device 100 determines whether the electronic device in the captured image is a preset electronic device according to the image recognition result. For example, if an electronic device is recognized, the recognition result may include the device information of the recognized electronic device (including device name, model, etc.).

[0194] In some embodiments, the electronic device 100 may first determine whether the recognition result is an electronic device according to the image recognition result. If so, it further determines whether the electronic device in the recognition result is a preset electronic device.

[0195] If the electronic device 100 recognizes that the captured image does not contain an electronic device, it may prompt "No electronic device recognized";

[0196] After the electronic device 100 recognizes an electronic device in the image, it can be matched with the preset electronic device mapping table in the database. It should be understood that the preset electronic device mapping relationship in the database may be pre - set in advance, or may be generated by the electronic device 100 or the server according to the user's screen mirroring usage habits. It should be understood that the database may contain the images and device information of the preset electronic devices (for example, the name, model, etc. of the electronic devices).

[0197] Optionally, the electronic device 100 may upload the image to the server for matching with the database in the server, and the electronic device 100 receives the matching result from the server.

[0198] Optionally, the electronic device 100 may also match the image with the database in the local device to determine whether the electronic device in the image is a preset electronic device.

[0199] After database matching, when a preset electronic device is found in the preset electronic device mapping table, the device information of the preset electronic device (for example, the name of the electronic device) can be obtained.

[0200] After database matching, when a preset electronic device is not found in the preset electronic device mapping table, it may request whether to start the screen mirroring sharing operation, leaving the option to the user, avoiding the inability to start screen mirroring caused by incorrect image recognition, enhancing user trust, and improving the user experience.

[0201] For example, Figure 7A In, the electronic device 100 is playing a video. The user points the front - facing camera of the electronic device 100 at the electronic device 102. After the front - facing camera of the electronic device 100 captures an image of the electronic device 102, the image is recognized through an identification algorithm to determine whether the electronic device 102 is a preset electronic device.

[0202] In some other embodiments, the electronic device 100 may identify whether the object in the captured image is a preset electronic device. For example, the electronic device 100 identifies, through image recognition, that the object in the image is a preset electronic device and outputs an identification result, which may include device information (including device name, model, etc.) of the identified electronic device. Instead of first determining whether the object in the image is an electronic device and then matching it with the database.

[0203] The electronic device 100 determines, through image recognition, whether the object in the captured image is a preset electronic device without matching it with the database, and then determines whether to trigger the next screen mirroring operation, which can improve the recognition efficiency.

[0204] Step 3: The electronic device 100 searches for and discovers the preset electronic device.

[0205] The electronic device 100 determines, based on the image recognition result, that the electronic device in the captured image is a preset electronic device.

[0206] In some embodiments, the electronic device 100 may discover an electronic device that can accept screen mirroring through a transmission protocol and search for the preset electronic device among the electronic devices that can accept screen mirroring. The transmission protocol includes, but is not limited to, the Alljoyn protocol, the DLNA protocol (Digital Living Network Alliance), the Airplay protocol, the miracst protocol, HTTP (Hypertext transfer protocol), the ChromeCast protocol, the Cast+ protocol, etc.

[0207] Optionally, the electronic device 100 may discover an electronic device that can accept screen mirroring through a local area network broadcast. For example, in some embodiments, the electronic device 100 may be referred to as the Source end, and the electronic device that can accept screen mirroring may be referred to as the Sink end. After the Sink end device is started, it registers its device information with the soft bus. After the Source end starts device search, it sends a broadcast message to the local area network through a fixed port (for example, 5684). After the devices in the local area network receive the broadcast message, if they meet the requirements, they will reply to the Source end through a unicast message. After the Source end receives the message, it parses the message and displays the Sink end electronic device on the interface of the Source end device.

[0208] Optionally, the electronic device 100 may discover an electronic device that can accept screen mirroring through Bluetooth. For example, through the monitoring of the electronic device that can accept screen mirroring, the electronic device 100 sends a broadcast, and the electronic device that can accept screen mirroring replies with a message to complete device discovery.

[0209] Optionally, the electronic device 100 can discover an acceptable screen mirroring electronic device through WIFI (Wireless Fidelity) direct connection.

[0210] Optionally, the electronic device 100 can dynamically select WIFI (Wireless Fidelity) direct connection, Bluetooth, or local area network to discover an acceptable screen mirroring electronic device.

[0211] The electronic device 100 searches for a preset electronic device in the list of acceptable screen mirroring electronic devices.

[0212] Optionally, the electronic device 100 can search for a preset electronic device in the list of acceptable screen mirroring electronic devices by using electronic device information (such as the electronic device name).

[0213] If the electronic device 100 searches and discovers the preset electronic device, the electronic device 100 can establish a connection with the preset electronic device; if the electronic device 100 fails to search and discover the preset electronic device, it can prompt the user that the search has failed.

[0214] Step 4: The electronic device 100 establishes a connection with the preset electronic device.

[0215] Optionally, the electronic device 100 can establish a connection with the preset electronic device through a local area network.

[0216] Optionally, the electronic device 100 can establish a connection with the preset electronic device through P2P.

[0217] By establishing a P2P link and performing a screen mirroring service on the P2P channel, the anti-interference ability can be enhanced and the screen mirroring experience can be improved.

[0218] Step 5: The electronic device 100 and the preset electronic device perform data transmission.

[0219] After the electronic device 100 establishes a connection with the preset electronic device, it performs data transmission.

[0220] Optionally, the electronic device 100 can transmit all the content of the screen displayed on the local device to the preset electronic device for display (for example, the screen mirroring method can be the mirroring screen method). Schematically, as Figure 7A shown, after the electronic device 100 establishes a connection with the preset electronic device 102, the graphical user interface displayed on the electronic device 102 is the same as the graphical user interface displayed on the display screen of the electronic device 100. Further, after the user issues an instruction to switch pages on the electronic device 100, the electronic device 100 switches pages, and at the same time, the electronic device 102 also switches the displayed screen.

[0221] Optionally, the electronic device 100 may transmit partial content of the screen displayed on the local device (e.g., video, picture) to a preset electronic device (e.g., the screen mirroring method is the resource projection method). Schematically, when the electronic device 100 is playing a video, the electronic device 100 may only transmit the video displayed on the local device to the electronic device 102.

[0222] Optionally, after the electronic device 100 establishes a connection with the preset electronic device, the electronic device 100 may detect whether the local device is playing a video, playing audio, or displaying a picture, and the screen mirroring method may automatically switch between the resource projection method and the mirror projection method.

[0223] Optionally, after the electronic device 100 establishes a connection with the preset electronic device, the preset electronic device may display partial content of the graphical user interface displayed on the electronic device 100 (e.g., the screen mirroring method may be the resource projection method). Schematically, when the electronic device 100 displays the GUI 4002, after the electronic device 100 establishes a connection with the electronic device 102, the electronic device 102 only displays the video played on the GUI 4002 of the electronic device 100, as Figure 7B shown.

[0224] Optionally, after the electronic device 100 establishes a connection with the preset electronic device, when the preset electronic device displays partial content (e.g., the screen mirroring method may be the resource projection method) or all content (e.g., the screen mirroring method may be the mirror screen mirroring method) of the graphical user interface of the electronic device 100, the electronic device 100 may enter the remote control mode. It should be understood that the remote control mode means that the user can control the electronic device 102 through the electronic device 100, as Figure 7C shown. For example, the user can control the play / pause of the video played on the electronic device 102 through the controls (e.g., play / pause) on the virtual remote control of the electronic device 100.

[0225] Figure 8 Shown is an exemplary diagram of the electronic device 100 capturing an image, identifying the electronic device 102, and playing the audio played on the electronic device 100 on the electronic device 102.

[0226] The process of the electronic device 100 capturing an image and identifying the electronic device 102 may refer to the technical solution of the embodiment Figure 7A - 7C shown, and the process of the electronic device 100 establishing a connection with the electronic device 102 may also refer to the technical solution of the embodiment Figure 7A - 7C shown, which will not be elaborated here.

[0227] Optionally, after establishing a connection with the electronic device 102, the electronic device 100 can transmit audio to the electronic device 102. After the electronic device 100 transmits the audio to the electronic device 102, the electronic device 100 can stop playing the audio to save power consumption, as Figure 8 shown. In some embodiments, the electronic device 100 can display audio controls (e.g., fast forward control, rewind control, play / pause control, volume adjustment control) on the screen, and the user can control operations such as play / pause of the audio on the electronic device 102 through the electronic device 100.

[0228] By using the camera of the electronic device to capture an image and identify the electronic device, thereby triggering screen mirroring, when the user shares content between different electronic devices, the user can directly interact with the shared device, solving the problem that the user cannot find the shared device because they do not know the device name, making the content sharing between devices more direct and natural. In addition, it solves the problem of being able to share content in one step regardless of whether it is close or far away, and the content can be shared without having to walk in front of the shared device.

[0229] Figure 9 The illustrated embodiment is a schematic diagram of the electronic device 100 receiving screen mirroring from the electronic device 102 by capturing an image of the electronic device 102 through a camera (e.g., a rear camera) and identifying it.

[0230] For example, the electronic device 102 is playing a video. The electronic device 100 can capture an image of the electronic device 102 through a camera (e.g., a rear camera) and identify it, search for and discover the electronic device 102 and establish a connection, and project the content of the electronic device 102 onto the local device for display.

[0231] It should be understood that the method for identifying the electronic device 102, the method for searching for the electronic device 102, and the method for establishing a connection with the electronic device 102 can refer to Figure 7A - 7C the relevant content, which will not be elaborated here.

[0232] It should also be understood that after the content of the electronic device 102 is projected onto the electronic device 100, the screen mirroring method can refer to Figure 7A - 7C the relevant content, which will not be elaborated here.

[0233] Figure 12 Shown is a schematic flowchart of a screen mirroring method in some embodiments.

[0234] 1201, the first electronic device plays multimedia content, and the camera of the first electronic device operates in a low-power mode;

[0235] Exemplarily, as Figure 7A shown, the mobile phone is playing a video, and the ultra-low-power camera equipped on the mobile phone operates in a low-power mode.

[0236] Optionally, the video being played on the mobile phone can be a locally stored video file or an online video.

[0237] Optionally, the camera is a front camera.

[0238] Optionally, the camera can periodically capture images.

[0239] Optionally, the frame rate of the images captured by the camera is no more than 10 frames.

[0240] 1202. The camera of the first electronic device captures at least one first image frame in the low-power operation mode;

[0241] Exemplarily, as Figure 7A shown, the camera of the mobile phone can capture low-resolution images in the low-power mode. For example, when the user is holding the mobile phone to watch a video, the first image frame captured at this time can include a face.

[0242] 1203. When it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in the normal mode and captures at least one second image frame;

[0243] For example, as Figure 7A shown, the mobile phone captures images of the TV through the camera. The mobile phone can perform image recognition on the captured images to determine whether the captured images are preset objects (for example, the user can set the preset object as an electronic device). When the mobile phone detects that the image contains a preset object, it turns on the normal mode to capture the second image frame.

[0244] Optionally, the normal mode can be that the camera of the mobile phone captures images at a high frame rate, where the high frame rate is greater than the image sampling frame rate in the low-power mode.

[0245] Optionally, the normal mode can be the high-resolution image capture mode of the mobile phone camera. The camera of the mobile phone can turn on the high-resolution mode to capture at least one second image frame, where the resolution of the second image frame is higher than that of the first image frame.

[0246] 1204. When it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device;

[0247] Exemplarily, as Figure 7A shown, the mobile phone detects that the TV in the image is a preset electronic device and establishes a connection with the TV.

[0248] Optionally, the mobile phone can identify that the TV set in the image is a preset electronic device through an image recognition algorithm.

[0249] Optionally, the mobile phone can establish a connection with the TV set through protocols such as Miracast, DLNA, Cast+, AirPlay, Chromecast, etc.

[0250] Optionally, before establishing a connection with the TV set, the mobile phone can search for and discover electronic devices that can accept screen mirroring, and then search for the TV set among the electronic devices that can accept screen mirroring.

[0251] Optionally, before establishing a connection with the TV set, the mobile phone can search for and discover electronic devices that can accept screen mirroring, and then search for the TV set among the electronic devices that can accept screen mirroring through device information (such as device name, device identifier, etc.).

[0252] Optionally, the mobile phone can be connected to the TV set through a local area network;

[0253] Optionally, the mobile phone can be connected to the TV set through WiFi direct connection;

[0254] Optionally, the mobile phone can be connected to the TV set through Bluetooth;

[0255] Optionally, before connecting the mobile phone to the TV set, a connection request can be sent to the TV set;

[0256] At 12:05, send the multimedia file to the second electronic device so that the second electronic device plays the multimedia content.

[0257] Exemplarily, as Figure 7A - 7C shown, after the mobile phone establishes a connection with the TV set, the mobile phone sends the video played on the local machine to the TV set, and the TV set plays the video.

[0258] Optionally, after the mobile phone establishes a connection with the TV set, the screen mirroring method can be mirror screen mirroring. For example, after the mobile phone sends the video played on the local machine to the TV set, the pictures displayed on the mobile phone and the TV set are the same, and the TV set plays the video.

[0259] Optionally, after the mobile phone establishes a connection with the TV set, the screen mirroring method can be resource screen mirroring. For example, after the mobile phone establishes a connection with the TV set and sends the video played on the local machine to the TV set, the TV set plays the video, the mobile phone does not play the video, and the mobile phone is in the remote control mode. The user can control the pause of the video played on the TV through the mobile phone.

[0260] Optionally, after the mobile phone establishes a connection with the TV set, the screen mirroring method can be dynamically switched between resource screen mirroring and mirror screen mirroring.

[0261] Figure 13 Shown is a schematic flowchart of another screen mirroring method in some embodiments.

[0262] 1301. The camera of the first electronic device operates in a low-power mode;

[0263] The camera of the first electronic device operates in a low-power mode, and the content of 1201 in Figure 12 can be cited and will not be elaborated here.

[0264] 1302. The camera of the first electronic device acquires at least one first image frame in the low-power operation mode;

[0265] For specific content, the content of 1202 in the Figure 12 embodiment can be cited and will not be elaborated here.

[0266] 1303. When it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal mode and acquires at least one second image frame;

[0267] For specific content, the content of 1203 in the Figure 12 embodiment can be cited and will not be elaborated here.

[0268] 1304. When it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device;

[0269] For specific content, the content of 1204 in the Figure 12 embodiment can be cited and will not be elaborated here.

[0270] 1305. The first electronic device receives a multimedia file sent by the second electronic device, where the content of the multimedia file is the multimedia content being played by the second electronic device;

[0271] Exemplarily, a mobile phone can send a request to a television to request the transmission of the multimedia content being played by the television. After receiving the request, the television can send the multimedia content being played to the mobile phone.

[0272] 1306. The first electronic device plays the multimedia file.

[0273] Exemplarily, after receiving the multimedia file sent by the television, the mobile phone plays the multimedia file;

[0274] Optionally, the mobile phone can play the multimedia file frame while receiving the multimedia file frame sent by the television;

[0275] Figure 14The following is a schematic flowchart of a screen mirroring system in some embodiments.

[0276] 1401, the camera of the first electronic device operates in a low-power mode;

[0277] For specific content, reference can be made to Figure 12 the content of 1201 in the embodiment, which will not be elaborated here.

[0278] 1402, the camera of the first electronic device acquires at least one first image frame in the low-power operation mode;

[0279] For specific content, reference can be made to Figure 12 the content of 1202 in the embodiment, which will not be elaborated here.

[0280] 1403, when it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal mode and acquires at least one second image frame;

[0281] For specific content, reference can be made to Figure 12 the content of 1203 in the embodiment, which will not be elaborated here.

[0282] 1404, when it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device;

[0283] For specific content, reference can be made to Figure 12 the content of 1204 in the embodiment, which will not be elaborated here.

[0284] 1405, the second electronic device sends the multimedia content being played to the first electronic device;

[0285] For specific content, reference can be made to Figure 12 the content of 1205 in the embodiment, which will not be elaborated here.

[0286] 1406, the first electronic device plays the multimedia content.

[0287] For specific content, reference can be made to Figure 12 the content of 1206 in the embodiment, which will not be elaborated here.

[0288] An embodiment of the present invention also provides a computer storage medium, in which computer instructions are stored. When the computer instructions run on an electronic device, the electronic device executes the above relevant method steps to implement the screen mirroring method in the above embodiments.

[0289] An embodiment of the present invention also provides a computer program product. When the computer program product runs on a computer, it causes the computer to execute the above relevant method steps to implement the screen mirroring method in the above embodiments.

[0290] In addition, an embodiment of the present invention further provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer-executable instructions. When the device runs, the processor may execute the computer-executable instructions stored in the memory, so that the chip executes the screen mirroring method in each of the above method embodiments.

[0291] Among them, the electronic device, computer storage medium, computer program product or chip provided by the embodiments of the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0292] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0293] In several embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces. The indirect coupling or communication connection of the device or unit may be in an electrical, mechanical or other form.

[0294] The unit described as a separated component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0295] In addition, in each embodiment of the present invention, each functional unit may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0296] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0297] The above content is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for screen mirroring, characterized in that, The method includes: The first electronic device plays multimedia content, and the camera of the first electronic device operates in a low-power mode; The camera of the first electronic device acquires at least one first image frame in the low-power operating mode; When it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal working mode and acquires at least one second image frame; When it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device and sends the multimedia content to the second electronic device so that the second electronic device plays the multimedia content; wherein, the second electronic device is an electronic device in an electronic device mapping table, and the electronic device mapping table includes images and device information of preset electronic devices; the device information of the second electronic device included in the electronic device mapping table is used for the first electronic device to establish a connection with the second electronic device.

2. The method according to claim 1, characterized in that, The camera of the first electronic device includes: A front camera.

3. The method according to claim 1 or 2, characterized in that, The camera of the first electronic device operates in a normal working mode, including: The camera of the first electronic device acquires an image frame in a high-resolution mode.

4. The method according to any one of claims 1 to 3, characterized in that, The camera of the first electronic device acquires at least one first image frame in the low-power operating mode, including: The frame rate of the camera acquiring at least one first image frame in the low-power operating mode does not exceed 10 frames per second.

5. The method according to any one of claims 1 - 4, characterized in that, The detection that the preset object is a preset second electronic device includes: Performing image recognition on the acquired image, and recognizing that the object in the acquired image is a preset second electronic device.

6. The method according to claim 5, characterized in that, The performing image recognition on the acquired image and recognizing that the object in the acquired image is a preset second electronic device may further include: Matching the image recognition result with the electronic device mapping table and matching to a preset second electronic device.

7. The method according to any one of claims 1 - 6, characterized in that, The device information of the second electronic device includes the name of the second electronic device.

8. The method according to any one of claims 1 - 7, characterized in that, Before the camera of the first electronic device acquires at least one first image frame in the low-power operating mode, the method further includes: The second processor of the local device sends a first message to the first processor of the local device to control the camera to acquire an image.

9. The method according to any one of claims 1 - 8, characterized in that, Before the first electronic device establishes a connection with the second electronic device, the method further includes: The first electronic device searches and discovers the second electronic device.

10. The method according to claim 9, characterized in that, The first electronic device searches and discovers the second electronic device, including one or a combination of the following methods: Discovering a third electronic device through a local area network broadcast and comparing the device information of the second electronic device with that of the third electronic device, or Discovering the third electronic device through Bluetooth and comparing the device information of the second electronic device with that of the third electronic device, or Discovering the third electronic device through WIFI direct connection and comparing the device information of the second electronic device with that of the third electronic device.

11. The method according to any one of claims 1 - 10, characterized in that, The first electronic device establishes a connection with the second electronic device, including: The first electronic device establishes a P2P connection with the second electronic device.

12. The method according to any one of claims 1 - 11, characterized in that, The multimedia content includes: At least one of video, audio, or pictures.

13. An electronic device supporting screen mirroring, characterized in that The electronic device includes: A camera, where the camera includes a low-power mode and a normal operating mode; At least one processor; A memory storing instructions that, when executed by the at least one processor, cause the electronic device to perform the following operations: Play multimedia content; Control the camera to capture at least one first image frame in the low-power operating mode; When it is detected that the at least one first image frame includes a preset object, control the camera to operate in the normal mode and capture at least one second image frame; When it is detected that the preset object is a preset device, establish a connection with the preset device and send the multimedia content to the preset device so that the preset device plays the multimedia content; where the preset device is an electronic device in an electronic device mapping table, and the electronic device mapping table includes images and device information of preset electronic devices; the device information of the preset device included in the electronic device mapping table is used for the electronic device to establish a connection with the preset device.

14. The electronic device according to claim 13, characterized in that The camera is a front camera of the electronic device.

15. The electronic device according to claim 13 or 14, characterized in that The electronic device controls the camera to operate in the normal operating mode, including the electronic device performing the following steps: The electronic device controls the camera to capture an image frame in the high-resolution mode.

16. The electronic device according to any one of claims 13 to 15, characterized in that The electronic device controls the camera to capture at least one first image frame in the low-power operating mode, including the electronic device performing the following steps: The electronic device controls the camera to capture at least one first image frame in the low-power operating mode at a frame rate not exceeding 10 frames per second.

17. The electronic device according to any one of claims 13 - 16, characterized in that The electronic device detects that the preset object is a preset device, including the electronic device performing the following steps: The electronic device performs image recognition on the captured image and recognizes that the object in the captured image is a preset device.

18. The electronic device according to claim 17, characterized in that The electronic device performs image recognition on the captured image and recognizes that the object in the captured image is a preset device. The electronic device further performs the following steps: The electronic device matches the image recognition result with the electronic device mapping table and matches a preset device.

19. The electronic device according to any one of claims 13 - 18, characterized in that The device information of the preset device includes the name of the preset device.

20. The electronic device according to any one of claims 13 - 19, characterized in that Before the electronic device controls the camera to capture at least one first image frame in the low-power operating mode, the electronic device performs the following steps: A second processor of the electronic device can send a first message to a first processor of the local machine to control the camera to capture an image.

21. The electronic device according to any one of claims 13 - 20, characterized in that Before the electronic device establishes a connection with the preset device, the electronic device further performs the following steps: The electronic device searches and discovers the preset device.

22. The electronic device according to any one of claims 13 - 21, characterized in that The electronic device establishes a connection with the preset device, including the electronic device performing the following steps: The electronic device establishes a P2P connection with the preset device.

23. The electronic device according to any one of claims 13 - 22, characterized in that The electronic device plays multimedia content includes: The electronic device plays at least one of video, audio, or pictures.

24. A method for screen mirroring, characterized in that The method includes: The camera of the first electronic device operates in the low-power mode; The camera of the first electronic device captures at least one first image frame in the low-power operating mode; When it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal mode and captures at least one second image frame; When it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device; wherein, the second electronic device is an electronic device in the electronic device mapping table, and the electronic device mapping table includes images and device information of preset electronic devices; the device information of the second electronic device included in the electronic device mapping table is used for the first electronic device to establish a connection with the second electronic device; The first electronic device receives a multimedia file sent by the second electronic device, where the content of the multimedia file is the multimedia content being played by the second electronic device; The first electronic device plays the multimedia file.

25. A screen mirroring system, characterized in that, It includes a first electronic device and a second electronic device; The camera of the first electronic device operates in a low-power mode; The camera of the first electronic device captures at least one first image frame in the low-power operation mode; When it is detected that the at least one first image frame includes a preset object, the camera of the first electronic device operates in a normal mode and captures at least one second image frame; When it is detected that the preset object is a preset second electronic device, the first electronic device establishes a connection with the second electronic device; wherein, the second electronic device is an electronic device in the electronic device mapping table, and the electronic device mapping table includes images and device information of preset electronic devices; the device information of the second electronic device included in the electronic device mapping table is used for the first electronic device to establish a connection with the second electronic device; The second electronic device sends the multimedia content being played to the first electronic device; The first electronic device plays the multimedia content.

26. A computer storage medium, characterized in that, It includes computer instructions, and when the computer instructions run on an electronic device, the electronic device is caused to execute the screen mirroring method according to any one of claims 1-12 or 24.

27. A computer program product, characterized in that, When the computer program product runs on a computer, the computer is caused to execute the screen mirroring method according to any one of claims 1-12 or 24.

Images