Screen projection method, screen projection source, screen projection destination, screen projection system and storage medium

Abstract

This article provides a screen projection method, a screen projection source end, a screen projection destination end, a screen projection system and a storage medium. The screen projection method is applied to a screen projection source end with a video call function, and the method includes: in a video call state, receiving a user's first call image sent by the screen projection destination end; generating a video call interface for display on the screen projection source end, the video call interface includes the user's first call image and interface elements; sending image data of the interface elements to the screen projection destination end, so that the screen projection destination end synthesizes the interface elements and the user's second call image to generate an interface for display on the screen projection destination end; wherein the user's second call image is an image captured by the camera of the screen projection destination end when the screen projection destination end receives the image data of the interface element. The present application can reduce the user-perceived delay during a video call.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a screen projection method, a screen projection source end, a screen projection destination end, a screen projection system, and a computer-readable storage medium. Background Art

[0002] With the popularization of terminal devices, users often have multiple terminal devices (for example, wearable devices, mobile phones, tablet computers, etc.), and multi-screen interactive technology can bring users a good user experience. Multi-screen interactive technology is a technology that shares screen display content between different terminal devices. Among them, multi-screen sharing of video call (for example, video chat, multi-person video conferencing, etc.) interface is an important application of multi-screen interactive technology.

[0003] In the prior art, when a user makes a video call through multi-screen interactive technology, the user may feel that the call delay displayed on the video call interface is relatively serious, which affects the user experience. Summary of the invention

[0004] Some embodiments of the present application provide a screen projection method, a screen projection source, a screen projection destination and a screen projection system. The present application is introduced from multiple aspects below, and the implementation methods and beneficial effects of the following aspects can be referenced to each other.

[0005] In the first aspect, an embodiment of the present application provides a screen projection method, which is applied to a screen projection source end with a video call function, and the screen projection method includes: in a video call state, receiving a user's first call image sent by a screen projection destination end; generating a video call interface for display on the screen projection source end, the video call interface including the user's first call image and interface elements; sending image data of the interface elements to the screen projection destination end, so that the screen projection destination end synthesizes the interface elements and the user's second call image to generate an interface for display on the screen projection destination end; wherein the user's second call image is an image captured by the camera of the screen projection destination end when the screen projection destination end receives the image data of the interface element.

[0006] According to the implementation mode of the present application, the screen projection destination end synthesizes the interface elements of the video call interface of the screen projection source end with the user call image collected in real time by the local camera to generate a video call interface for display on the screen projection destination end. In this way, on the video call interface displayed on the screen projection destination end, the acquisition time and display time of the user call image are only separated by the time it takes for the screen projection destination end to process the image data, thereby significantly reducing the user-perceived delay.

[0007] In some embodiments, the interface elements of the video call interface include at least one of the following: controls for controlling the call status and / or interface display status, call status information, user identification, and call images of other users participating in the video call.

[0008] In some implementations, the image data of the interface element includes: image information of each element in the interface element, and position information of each element on the video call interface. According to the implementation of the present application, the amount of image data of the interface element can be reduced.

[0009] In some embodiments, the starting conditions of the screen projection method are: the communication delay between the screen projection source and the screen projection destination exceeds a set threshold; and / or the data buffer amount in the communication channel between the screen projection source and the screen projection destination exceeds a set threshold.

[0010] In the second aspect, an embodiment of the present application provides a screen projection method, which is applied to a screen projection destination end. The screen projection method includes: when a screen projection source end is in a video call state, sending a user's first call image to the screen projection source end, so that the screen projection source end generates a video call interface for display on the screen projection source end, and the video call interface includes the user's first call image and interface elements; receiving image data of the interface elements sent by the screen projection source end; synthesizing the interface elements and the user's second call image to generate an interface for display on the screen projection destination end; wherein the user's second call image is an image captured by the camera of the screen projection destination end when the screen projection destination end receives the image data of the interface element.

[0011] According to the implementation mode of the present application, the screen projection destination end synthesizes the interface elements of the video call interface of the screen projection source end with the user call image collected in real time by the local camera to generate a video call interface for display on the screen projection destination end. In this way, on the video call interface displayed on the screen projection destination end, the acquisition time and display time of the user call image are only separated by the time it takes for the screen projection destination end to process the image data, thereby significantly reducing the user-perceived delay.

[0012] In some embodiments, the interface elements of the video call interface include at least one of the following elements: controls for controlling the call status and / or interface display status, call status information, user identification, and call images of other users participating in the video call.

[0013] In some implementations, the image data of the interface element includes: image information of each element in the interface element, and position information of each element on the video call interface. According to the implementation of the present application, the amount of image data of the interface element can be reduced.

[0014] In some embodiments, the starting conditions of the screen projection method are: the communication delay between the screen projection source and the screen projection destination exceeds a set threshold; and / or the data buffer amount in the communication channel between the screen projection source and the screen projection destination exceeds a set threshold.

[0015] In a third aspect, an embodiment of the present application provides a screen projection source end, the screen projection source end has a video call function, and the screen projection source end includes: a memory for storing instructions executed by one or more processors of the screen projection source end; a processor, when the processor executes the instructions in the memory, the screen projection source end can execute the screen projection method provided by any embodiment of the first aspect of the present application. The beneficial effects that can be achieved in the third aspect can refer to the beneficial effects of the method provided by any embodiment of the first aspect, and will not be repeated here.

[0016] In a fourth aspect, an embodiment of the present application provides a screen projection destination terminal, which has a video call function, and the screen projection destination terminal includes: a memory for storing instructions executed by one or more processors of the screen projection destination terminal; a processor, when the processor executes the instructions in the memory, the screen projection destination terminal can execute the screen projection method provided by any embodiment of the second aspect of the present application. The beneficial effects that can be achieved in the fourth aspect can refer to the beneficial effects of the method provided by any embodiment of the second aspect, and will not be repeated here.

[0017] In a fifth aspect, an embodiment of the present application provides a screen projection system, the screen projection system includes a screen projection source end and a screen projection destination end, the screen projection source end has a video call function, wherein the screen projection source end is used to execute the screen projection method provided by any embodiment of the first aspect of the present application, and the screen projection destination end is used to execute the screen projection method provided by any embodiment of the second aspect of the present application. The beneficial effects that can be achieved in the fifth aspect can refer to the beneficial effects of the method provided by any embodiment of the first aspect or any embodiment of the second aspect, and will not be repeated here.

[0018] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored, and when the instructions are executed on a computer, the computer can execute the screen projection method provided by any embodiment of the first aspect of the present application, or the computer can execute the screen projection method provided by any embodiment of the second aspect of the present application. The beneficial effects that can be achieved in the sixth aspect can refer to the beneficial effects of the method provided by any embodiment of the first aspect or any embodiment of the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An exemplary application scenario of the screen projection method provided in the embodiments of the present application;

[0020] Figure 2 A schematic diagram of the image data transmission process provided in an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of the image data transmission process in the prior art;

[0022] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0023] Figure 5a A software architecture diagram of the screen projection source end provided for the implementation mode of this application;

[0024] Figure 5b A software architecture diagram of the screen projection destination provided for the implementation mode of this application;

[0025] Figure 6a Schematic diagram of a multi-screen interactive scenario (screen projection source end) provided for an implementation manner of the present application;

[0026] Figure 6b Schematic diagram of a multi-screen interactive scenario provided for an implementation method of the present application (screen projection destination);

[0027] Figure 7 A flowchart of a screen projection method provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the mobile phone interface provided for the implementation of the present application;

[0029] Figure 9a A schematic diagram of image data of interface elements of a mobile phone side interface provided in an embodiment of the present application;

[0030] Figure 9b Another schematic diagram of the interface element image data of the mobile phone side interface provided in the implementation manner of the present application;

[0031] Fig.9c A schematic diagram of a tablet side interface provided for an embodiment of the present application;

[0032] Fig.10 An exemplary structural diagram of a screen projection system provided in an embodiment of the present application;

[0033] Fig.11 A block diagram of an electronic device provided in an embodiment of the present application is shown;

[0034] Fig.12 A schematic diagram of the structure of a system on chip (SoC) provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] The implementation of this embodiment will be described in detail below with reference to the accompanying drawings.

[0036] Figure 1 The diagram shows a scenario in which user A and user B have a video call.

[0037] User A has device A1 and device A2, wherein device A1 has a video call application (e.g., WeChat, Skype, etc.) installed on it. Thus, during a video call, device A1 can generate a video call interface through the video call application. Using multi-screen interactive technology, during a video call, device A1 can transmit the interface image of the video call interface to device A2 (e.g., through screen projection technology, or based on a distributed system), so that device A2 can display the video call interface generated by device A1. Typically, the video call interface includes the call image of user A, the call image of user B, interface controls, call status information, etc.

[0038] Sometimes, during a video call, device A2 is the device currently being operated by user A (for example, user A is using an application on device A2 (for example, a picture processing application), or user A needs to make a video call through device A2 (for example, a car computer) while driving a vehicle, etc.). At this time, the call image of user A on the video call interface is captured by the camera of device A2. After capturing the call image of user A, the camera of device A2 transmits the call image to device A1, and device A1 generates a video call interface based on the call image transmitted from device A2. Afterwards, device A1 transmits the interface image of the generated video call interface to device A2, so that user A, who is operating device A2, can see the video call interface.

[0039] The present application embodiment is used to provide an application Figure 1 A screen projection method for the scenario shown is used to reduce user-perceived latency. In summary, in the implementation mode of the present application, device A2 does not directly display the video call interface on device A1, but synthesizes the interface element images in the video call interface of A1 with the user call image captured in real time by the local camera to generate a video call interface for display on device A2. In this way, in the video call interface displayed by device A2, the user call image is a real-time image of user A, which is basically consistent with the current status of user A, thereby significantly reducing the user-perceived latency and improving the user experience.

[0040] In the implementation manner of the present application, device A1 is the screen projection source, and device A2 is the screen projection destination. For the sake of clarity, in the following, a mobile phone is used as an example of a screen projection source, and a tablet is used as an example of a screen projection destination. Therefore, the video call interface generated by the mobile phone is called the "mobile phone side interface", and the video call interface generated by the tablet is called the "tablet side interface".

[0041] However, it is understood that the present application is not limited thereto. In other examples, the screen projection source may be other devices with video call functions, such as tablets, laptops, etc.; the screen projection destination may be other devices with display functions, such as TVs, car computers, etc. It should be noted that the screen projection source and the screen projection destination may be the same type of device (for example, both the screen projection source and the screen projection destination are mobile phones), or different types of devices (for example, the screen projection source is a mobile phone, and the screen projection destination is a car computer).

[0042] In addition, in the implementation manner of the present application, there is no limit on the number of users participating in the video call, and it can be a one-to-one video call, for example Figure 1 The video call scenario shown is between user A and user B; it can also be a many-to-many video call, for example, a multi-party video conferencing scenario, an online teaching scenario, etc.

[0043] Figure 2 The transmission process of image data during a video call in an embodiment of the present application is shown. Specifically, at time T1, device A2 collects and sends the call image P1 of user A to device A1; at time T2, device A1 receives the call image P1 of user A sent by device A2, and the call image of user B sent by device B; at time T3, device A1 sends the interface element image of the video call interface (i.e., the mobile phone side interface) generated by device A1 to device A2 (which does not include the call image P1 of user A); at time T4', device A2 receives the interface element image sent by device A1, and collects the call image P2 of user A in real time through the camera of device A2, and then synthesizes the interface element image and the call image P2 of user A to generate a video call interface (i.e., the tablet side interface) for display by device A2; at time T5', device A2 displays the tablet side interface, which includes the call image P2 of user A.

[0044] Figure 3 The existing technical solutions in the prior art are shown. Figure 3 and Figure 2 The main difference is that at time T3, device A1 generates a video call interface (i.e., mobile phone side interface) based on the call image P1 of user A and the call image of user B, and sends the interface image of the mobile phone side interface to device A2; at time T4, device A2 receives the interface image of the mobile phone side interface; at time T5, device A2 displays the mobile phone side interface, which includes the call image P1 of user A captured by device A2 at time T1.

[0045] Assume that at time T1 (i.e., the time when device A2 collects the call image of user A), user A is raising his hand. After a time interval ΔT=T5-T1, i.e., at time T5, the hand-raising action will be displayed on device A2. When the time when device A2 displays the video call interface has a longer delay than the time when device A2 collects the call image of user A, i.e., when the time interval ΔT is longer, user A will perceive that the content displayed by device A2 is significantly behind the current state of user A, i.e., user A will perceive a more serious call delay (referred to as "user perceived delay" in this article), which will affect user A's user experience.

[0046] The main reasons for user-perceived delay are the transmission delay of image data in the communication network and the processing delay of image data by the terminal device. The former is the main reason for user-perceived delay. When the network quality is average, the user-perceived delay may exceed 300ms. When the network is congested, the user-perceived delay may exceed 500ms, which will reduce the user experience.

[0047] refer to Figure 2 In the implementation manner of the present application, because device A2 synthesizes the interface elements of the video call interface of device A1 with the user call image captured in real time by the local camera to generate a video call interface for display by device A2, that is, on the video call interface displayed by device A2, the call image P2 of user A is the image captured by the camera of device A2 at time T4'. Since the acquisition time (that is, T4') and the display time (that is, T5') of the call image P2 are only separated by the time it takes device A2 to process the image data, the user perceived delay can be significantly reduced.

[0048] The following is Figure 1 The technical solution of the present application is introduced by taking the video call scenario between user A and user B as an example.

[0049] Figure 4 1 shows a schematic diagram of the structure of an electronic device 100. The electronic device 100 can be implemented as a mobile phone or a tablet.

[0050] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) connector 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0051] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0053] The processor can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0054] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0055] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0056] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 may be coupled to the camera 193 via the I2C bus interface.

[0057] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It transmits data in Serial Communication and Parallel communication In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function.

[0058] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0059] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, etc. The GPIO interface can also be configured as an I2C interface, a UART interface, a MIPI interface, etc.

[0060] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0061] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0062] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0063] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0064] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0065] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0066] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0067] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0068] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0069] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0070] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0071] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0072] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0073] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0074] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0075] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0076] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0077] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0078] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The internal memory 121 and / or a memory provided in the processor stores instructions. When the electronic device 100 is implemented as a mobile phone, when the instructions stored in the internal memory 121 and / or the memory set in the processor are executed by the processor 110, the mobile phone can execute the steps performed by the mobile phone in the screen projection method provided in the implementation mode of the present application; when the electronic device 100 is implemented as a tablet, when the instructions stored in the internal memory 121 and / or the memory set in the processor are executed by the processor 110, the tablet can execute the steps performed by the tablet in the screen projection method provided in the implementation mode of the present application.

[0079] It should be understood that in practical applications, the electronic device 100 may include Figure 4 The embodiments of the present application are not limited to the more or fewer components shown. The illustrated electronic device 100 is only an example, and the device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0080] The following introduces the software system of the mobile phone. The software system of the mobile phone can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the mobile phone system architecture.

[0081] Figure 5a The system architecture of the mobile phone is shown. The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

[0082] The application layer can include a series of application packages. Figure 5aAs shown, the application package on the mobile phone may include video call applications, mobile screen projection applications, cameras, galleries, calendars, calls, maps, navigation, WLAN, Bluetooth, music, videos, short messages and other applications.

[0083] The video call application is used to generate a mobile phone side interface during a video call. Specifically, during a video call, the video call application obtains the call image of user A and the call image of user B, and generates call controls (for example, controls for controlling the call status, controls for controlling the interface display status), user identifiers, etc. Afterwards, the video call application synthesizes the call image of user A, the call image of user B, the call controls, the user identifiers, etc. to form a mobile phone side interface. In the implementation manner of the present application, other elements on the mobile phone side interface except the call image of user A (for example, the call image of user B, call controls, etc.) are referred to as interface elements.

[0084] The video call application also transmits the image data of the interface elements of the mobile phone side interface to the mobile phone screen projection application. After receiving the image data of the interface elements, the mobile phone screen projection application processes the image data of the interface elements, such as encoding and packaging, and then the mobile phone sends the processed image data to the tablet.

[0085] like Figure 5a As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0086] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0087] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0088] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0089] The phone manager is used to provide the communication functions of the mobile phone, such as the management of call status (including answering, hanging up, etc.).

[0090] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0091] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0092] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0093] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0094] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0095] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0096] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0097] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0098] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0099] A 2D graphics engine is a drawing engine for 2D drawings.

[0100] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, sensor driver, etc.

[0101] Figure 5bThe system architecture of a tablet is shown. The system architecture of a tablet is basically the same as that of a mobile phone. The following only introduces the parts of the tablet system architecture that are different from the mobile phone system architecture, and the same parts of the two are not repeated.

[0102] The application layer of the tablet includes the tablet screen projection application and the mobile phone clone application.

[0103] After the tablet receives the interface element image data of the mobile phone side interface sent by the mobile phone, the tablet screen projection application decodes the received image data to restore the interface element image of the mobile phone side interface. The tablet side screen projection can also obtain the user call image P2 captured by the tablet camera, and synthesize the interface element image of the tablet side interface with the user call image P2 to generate an overlay image.

[0104] The mobile phone clone application can obtain the overlay image generated by the tablet screen projection application and process the overlay image, for example, adjust the resolution of the overlay image, add the identification information of the screen projection source (i.e., mobile phone) to the overlay image, etc., to generate the tablet side interface.

[0105] The following combination Figure 1 The multi-screen interactive scenario shown introduces the specific embodiments of the present application.

[0106] Before the method of this embodiment is started, the mobile phone and tablet of user A are in a multi-screen interactive state, that is, the mobile phone shares its screen display content to the tablet through the multi-screen interactive technology, and the tablet displays the content of the mobile phone screen through the mobile phone clone application installed on it. Figure 6a and Figure 6b , the mobile phone clone application on the tablet displays the application list on the mobile phone. In this state, one party of the video call initiates a video call request to the other party.

[0107] For example, user A initiates a video call request to user B. Figure 6b After user A clicks the icon of the video call application on the tablet, the tablet sends user A's operation information (e.g., operation location, operation type, etc.) to the mobile phone. The mobile phone determines that the user's instruction is to open the video call application based on the operation information sent by the tablet. In response to the received user instruction, the mobile phone opens the video call application and displays a contact interface for establishing a video call connection.

[0108] The tablet simultaneously displays the contact interface of the video call application. After user A clicks on user B's avatar, the mobile phone can send a video call request to user B's device (i.e., device B). After user B sends an instruction to its video call device to accept the video call request, a video call connection is established between user A's mobile phone and user B's device.

[0109] To reduce the user perceived delay in video calls, refer to Figure 7 This embodiment is used to provide a screen projection method in a video call state, including the following steps:

[0110] S10: In the video call state, the mobile phone obtains the call image P1 of user A (as the first call image of user) and the call image of user B, wherein the call image of user A is the image captured by the tablet camera, and the call image of user B is the image captured by the device of user B.

[0111] After the mobile phone establishes a video call connection with the device of user B, the mobile phone sends a call image acquisition request to the tablet (for example, the mobile phone sends a signal to the tablet that the video call connection is established). In response to the call image acquisition request sent by the mobile phone, the tablet starts the camera and captures the call image of user A at a certain frame rate (for example, 40fps). In the current step, the tablet sends the call image P1 of user A captured by the camera to the mobile phone.

[0112] At the same time, the device of user B collects the call image of user B and sends the call image of user B to the mobile phone.

[0113] It should be noted that in this embodiment, the user's call image is the image captured by the device's camera, and is not necessarily the user's own image. For example, when user A is teaching online, the tablet's camera is facing the demonstration board. At this time, user A's call image is the screen of the demonstration board.

[0114] In addition, the tablet can also collect the voice signal of user A and send the voice signal of user A to the mobile phone. The mobile phone sends the voice signal of user A to the device of user B through the video call application, so that user B can hear the voice of user A. Similarly, the device of user B collects the voice signal of user B and sends the voice signal of user B to the mobile phone. The mobile phone sends the voice signal of user B back to the tablet, so that user A can hear the voice of user B.

[0115] S20: The mobile phone generates a mobile phone side interface (as a video call interface for displaying the screen projection source end). In this embodiment, the mobile phone generates the mobile phone side interface through a video call application. Specifically, the mobile phone video application determines the call controls, call information, etc. on the video call interface for display on the mobile phone, and synthesizes the call controls, call information, etc. with the call image P1 of user A and the call image of user B to generate the mobile phone side interface. In this application, the elements on the mobile phone side interface other than the call image of user A are referred to as "interface elements" of the mobile phone side interface.

[0116] refer to Figure 8In this embodiment, the interface elements of the mobile phone side interface include one or more of the following: controls for controlling the call status, such as a hang-up button 10; a voice call / video call switching button 20, etc.; controls for controlling the interface display status, such as a front camera / rear camera switching button 30, a call image switching button (used to switch the display position of the call image of user A and the call image of user B), etc.; call status information, such as call duration 40, etc.; user B's call image 50 (as a call image of other users participating in the video call), etc. It should be understood that in other embodiments, the interface elements may include more or fewer elements than in this embodiment. For example, in some embodiments, the interface elements may also include user identifiers (such as avatars and nicknames of user A and / or user B), interface borders, etc.

[0117] In this embodiment, the call image P1 of user A is displayed in the upper right corner of the mobile phone interface, and the call image of user B is distributed on the entire mobile phone interface. It can be understood that in other embodiments, the call images of user A and user B can be distributed in different ways, for example, the call images of user A and user B are distributed in different ways. Figure 8 The positions of the calls can be swapped by the settings, or the call image of user A and the call image of user B are displayed side by side on the mobile phone interface.

[0118] S30: The mobile phone sends the image data of the interface element to the tablet. In this embodiment, after the video call application of the mobile phone determines the interface element of the mobile phone side interface, it transmits the image data of the interface element to the mobile phone screen projection application, and the mobile phone screen projection application processes the image data of the interface element and then sends the processed image data to the tablet.

[0119] In this embodiment, the image data of the interface elements includes the image information of each interface element and the position information of each interface element on the mobile phone side interface. Figure 9a For the convenience of description, the interface elements on the video call interface are numbered. Specifically, the interface elements are numbered as element 1, element 2, ..., element 6. Taking element 2 (i.e., the switch control for controlling the on / off state of the video call) as an example, the image information of element 2 is the pixel value of each pixel forming element 2, and the position information of element 2 is the coordinate of element 2 on the mobile phone side interface. Figure 9a In the coordinate system xOy established in the manner shown, the position information of element 2 is the coordinates of its center point (35 mm, 10 mm).

[0120] In this embodiment, the image data of the interface element includes the image information and position information of each interface element, so the amount of image data of the interface element is relatively small. However, the present application is not limited thereto. For example, in other embodiments, reference Figure 9b,The image data of the interface element is the pixel value of each pixel in the virtual frame area.

[0121] After the mobile phone video call application determines the image data of the interface element, it sends the image data to the mobile phone screen projection application. The mobile phone screen projection application processes the image data, for example, encodes the image data, packages it according to the screen projection protocol (for example, miracast protocol, airplay mirroring protocol, etc.), etc., and then sends the processed interface element image data to the tablet.

[0122] S40: The tablet generates a superimposed image of the call image P2 of user A (as the second call image of the user) and the interface element image. In this embodiment, step S40 specifically includes:

[0123] S41: The tablet screen projection application restores the interface element images. After receiving the image data of the interface elements sent by the mobile phone, the tablet unpacks and decodes the image data through the tablet screen projection application, restores the image data of each interface element, and restores the interface element images (i.e., images of other parts of the mobile phone side interface except the user call image P1) based on the image data of the interface elements.

[0124] S42: The tablet screen projection application obtains the call image P2 of user A. After the tablet receives the image data of the interface element sent by the mobile phone, the tablet camera collects the call image P2 of user A in real time, and the tablet screen projection application obtains the call image P2 of user A through the tablet underlying system.

[0125] S43: The tablet screen projection application superimposes the call image P2 of user A with the interface element image to generate a superimposed image. In this embodiment, the tablet determines the size and position of the call image P2 of user A on the superimposed image based on the size and position of the call image P1 of user A on the mobile phone side interface (this information may be sent by the mobile phone to the tablet), so that the superimposed image is as close to the mobile phone side interface as possible. However, the present application is not limited to this. For example, in other embodiments, the size of the call image P2 of user A and its position on the superimposed image are set by the user.

[0126] S50: The tablet generates a tablet side interface based on the overlay image (as an interface for display on the screen projection destination). In this embodiment, the tablet screen projection application transmits the generated overlay image to the mobile phone clone application on the tablet, and the mobile phone clone application processes the overlay image, for example, adjusting the resolution of the overlay image to match the area of ​​the tablet side interface with the area of ​​the tablet display; and / or, adding identification information of the screen projection source to the overlay image (for example, adding text "Screen projection from Mate40" to the overlay image), etc., to generate the tablet side interface. Afterwards, the mobile phone clone application transmits the interface image of the tablet side interface to the underlying system of the tablet, so that the underlying system of the tablet controls the tablet display to display the tablet side interface (such as Fig.9c shown).

[0127] It should be noted that the technical solution of this embodiment is only an exemplary description of the implementation method of the present application, and the present application may have other variations.

[0128] For example, the applications in this embodiment may be merged, split or recombined. For example, the video call application and the mobile screen projection application on the mobile phone may be merged into one application; or some functions of the tablet screen projection application (for example, the image overlay function in step S43) may be merged into the mobile phone clone application on the tablet.

[0129] For another example, some steps in this embodiment may be omitted, or steps may be added on the basis of this embodiment. For example, the step of processing the image data of the interface element by the mobile phone screen projection module in step S30 is omitted, and the mobile phone directly sends the image data of the interface element determined by the video call application to the tablet; or the step of processing the superimposed image by the mobile phone avatar application in step S50 is omitted, and the tablet directly displays the superimposed image generated by the tablet screen projection application as the tablet side interface, etc.

[0130] In addition, the screen projection method provided in this application can also be set with start conditions. When the communication network where the mobile phone and the tablet are located is unobstructed, the method in the prior art can be used (for example, Figure 3 When the communication network is congested, the screen projection method provided by this application can be activated to reduce the user-perceived delay.

[0131] In some embodiments, whether to start the screen projection method provided by the present application is determined based on the communication delay between the tablet and the mobile phone. Exemplarily, the communication delay can be determined based on the image transmission time between the tablet and the mobile phone. For example, referring to Figure 2The call image P1 of user A sent by the tablet to the mobile phone carries a timestamp. The mobile phone can obtain the sending time T1 of the call image P1 by reading the timestamp. When the mobile phone receives the call image P2, it obtains the system time T2 on the mobile phone. If the difference between T2 and T1 is less than the set value (for example, 100ms), it continues to Figure 3 The method shown performs multi-screen interaction with the tablet, otherwise, the mobile phone starts the screen projection method provided in this application and sends an instruction to the tablet to start the screen projection method of this application.

[0132] In other embodiments, whether to start the screen projection method of the present application is determined based on the amount of data buffered in the communication channel between the mobile phone and the tablet. Exemplarily, the amount of data buffered in the communication channel can be the amount of data buffered in the wireless communication module of the image data sending end (for example, Figure 4 For example, continue to refer to Figure 3 When the tablet sends the call image P1 of user A to the mobile phone, the data buffer in the wireless communication module of the tablet is detected. If the buffer is less than the set value, the data buffer is continuously sent to the mobile phone. Figure 2 The method shown performs multi-screen interaction with the tablet, otherwise, the tablet starts the screen projection method provided in this application and sends an instruction to the mobile phone to start the screen projection method of this application.

[0133] In some other embodiments, whether to start the screen projection method of the present application is determined comprehensively based on the communication delay between the tablet and the mobile phone, and the amount of data buffered in the communication channel between the tablet and the mobile phone. For example, when the communication delay between the tablet and the mobile phone exceeds a set value, or the amount of data buffered in the communication channel between the mobile phone and the tablet exceeds a set value, the screen projection method provided by the present application is started.

[0134] The following combination Fig.10 The following is a schematic diagram of the structure of the screen projection system 200 provided according to an embodiment of the present application. Fig.10 The screen projection system 200 includes a screen projection source end 210 (eg, a mobile phone) and a screen projection destination end 220 (eg, a tablet).

[0135] The screen projection source 210 includes:

[0136] (1) Device virtualization module 211: The device virtualization module 211 includes a Display submodule 211a.

[0137] The display submodule 211a is used to receive the user's first call image (for example, the call image P1 in step S10) sent by the screen projection destination 220, and transmit the received user's first call image to the video call application module 212;

[0138] The display submodule 211a is also used to obtain the image data of the interface elements from the video call application module 212; and send the image data of the interface elements to the screen projection destination 220. The interface elements include controls for controlling the call status and / or the interface display status (for example, Figure 9a Element 1, Element 2, Element 3), call status information (for example, Figure 9a 4), user identification, and / or call images of other users participating in the video call (e.g., Figure 9a Element 5 in the screen projection source end 210). The image data of the interface elements includes image information of each interface element and position information of each interface element on the video call interface displayed by the screen projection source end 210.

[0139] The device virtualization module 211 can also determine whether to start the screen projection method provided in the first embodiment of the present application according to the communication status between the screen projection source terminal 210 and the screen projection destination terminal 220. For example, the device virtualization module 211 determines whether to start the screen projection method provided in the present application according to the communication delay between the screen projection source terminal 210 and the screen projection destination terminal 220, and / or the amount of data buffer in the communication channel between the screen projection source terminal 210 and the screen projection destination terminal 220. Specifically, when the communication delay between the screen projection source terminal 210 and the screen projection destination terminal 220 is greater than the set threshold, or the amount of data buffer in the communication channel between the screen projection source terminal 210 and the screen projection destination terminal 220 is greater than the set threshold, the device virtualization module 211 determines to start the screen projection method provided in the present application, and sends an instruction to start the screen projection method provided in the present application to the screen projection destination terminal 220.

[0140] (2) Video call application module 212, including one or more video call applications, such as WeChat TM , Facetime TM , system phone applications, etc. The video call application module 212 is used to receive the user's first call image from the screen projection destination terminal 220 from the Display submodule 211a; and determine the interface elements of the video call interface displayed by the screen projection source terminal 210 (for example, Figure 9a ; and, generating a video call interface for display on the screen projection source end 210 (for example, the mobile phone side interface described in step S20) according to the received user first call image and the determined interface elements;

[0141] (3) A display module 213 , used to display the video call interface generated by the video call application module 212 .

[0142] The screen projection destination 220 includes:

[0143] (1) The device virtualization module 221 includes a Display submodule 221a.

[0144] The display submodule 221a sends the user's first call image (for example, the call image P1 in step S10) acquired by the image acquisition module 222 to the screen projection source end 220;

[0145] The Display submodule 221a is also used to receive image data of interface elements sent by the projection source end 210; and, to receive the user's second call image captured in real time by the image acquisition module 222 (for example, the call image P2 in step S40); and, based on the received interface elements and the user's second call image captured in real time by the image acquisition module 222, to generate a video call interface for display on the projection destination end 220 (for example, the tablet side interface described in step S50).

[0146] The device virtualization module 221 can also be used to obtain the current working status of the image acquisition module 222 and the display module 223 of the screen projection destination 220, for example, whether the camera in the image acquisition module 222 is in use, and whether the current working state of the display module 223 is the screen projection state;

[0147] (2) an image acquisition module 222, including a camera, for acquiring an image of the user's first call; and, when the screen projection destination 220 receives image data of the interface element from the screen projection source 210, acquiring an image of the user's second call in real time;

[0148] (3) Display module 223, including a display screen, for displaying the video call interface generated by the Display submodule 221a.

[0149] Reference now Fig.11 , shown is a block diagram of an electronic device 400 according to one embodiment of the present application. The electronic device 400 may include one or more processors 401 coupled to a controller hub 403. For at least one embodiment, the controller hub 403 communicates with the processor 401 via a multi-drop bus such as a front side bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or a similar connection 406. The processor 401 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 403 includes, but is not limited to, a graphics & memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which can be on a separate chip) (not shown), wherein the GMCH includes a memory and a graphics controller and is coupled to the IOH.

[0150] The electronic device 400 may also include a coprocessor 402 and a memory 404 coupled to a controller hub 403. Alternatively, one or both of the memory and the GMCH may be integrated within the processor (as described in this application), with the memory 404 and the coprocessor 402 being directly coupled to the processor 401 and the controller hub 403, with the controller hub 403 being in a single chip with the IOH.

[0151] The memory 404 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of the two. The memory 404 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, specifically, temporary and permanent copies of the instructions.

[0152] like Fig.11 The electronic device 400 shown can be implemented as a screen projection source and a screen projection destination, respectively. When the electronic device 400 is implemented as a screen projection source, the instructions stored in the memory 404 may include: when executed by at least one of the processors, causing the screen projection source to implement the following Figure 7 When the electronic device 400 is implemented as a screen projection destination, the instructions stored in the memory 404 may include: when executed by at least one of the processors, causing the screen projection destination to implement the following steps: Figure 7 Instructions for the steps implemented by the screen projection destination in the method shown.

[0153] In one embodiment, the coprocessor 402 is a special-purpose processor, such as, for example, a high-throughput Many Integrated Core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on graphics processing units (GPGPU), or an embedded processor. The optional nature of the coprocessor 402 is indicated by a dashed line in FIG. Fig.11 middle.

[0154] In one embodiment, the electronic device 400 may further include a network interface (Network Interface Controller, NIC) 406. The network interface 406 may include a transceiver for providing a radio interface for the electronic device 400, and then communicating with any other suitable device (such as a front-end module, an antenna, etc.). In various embodiments, the network interface 406 may be integrated with other components of the electronic device 400. The network interface 406 may implement the functions of the communication unit in the above-mentioned embodiments.

[0155] The electronic device 400 may further include an input / output (I / O) device 405. The I / O 405 may include: a user interface designed to enable a user to interact with the electronic device 400; a peripheral component interface designed to enable peripheral components to interact with the electronic device 400; and / or a sensor designed to determine environmental conditions and / or location information related to the electronic device 400.

[0156] It is worth noting that Fig.11 This is for illustrative purposes only. Fig.11 It is shown that the electronic device 400 includes multiple devices such as a processor 401, a controller hub 403, a memory 404, etc.; however, in actual applications, the device using the methods of the present application may include only a portion of the devices in the electronic device 400, for example, it may include only the processor 401 and the network interface 406. Fig.11 The properties of optional devices are shown with dotted lines.

[0157] Reference now Fig.12 , which is a block diagram of a system on chip (SoC) 500 according to an embodiment of the present application. Fig.12 In the , similar components have the same reference numerals. In addition, the dashed boxes are optional features of more advanced SoCs. Fig.12In the embodiment, SoC 500 includes: an interconnect unit 550 coupled to a processor 510; a system agent unit 580; a bus controller unit 590; an integrated memory controller unit 540; a group or one or more coprocessors 520, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 530; and a direct memory access (DMA) unit 560. In one embodiment, the coprocessor 520 includes a special-purpose processor, such as, for example, a network or communication processor, a compression engine, a general-purpose computing on graphics processing units (GPGPU), a high-throughput MIC processor, or an embedded processor.

[0158] Static random access memory (SRAM) unit 530 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, and more specifically, stores temporary and permanent copies of the instructions.

[0159] like Fig.12 The SoC shown can be set in the screen projection destination and the screen projection source respectively. When the SoC is set in the screen projection destination, the static random access memory (SRAM) unit 530 stores instructions, which may include: when executed by at least one of the processors, causing the wearable device to implement Figure 7 When the SoC is set in the screen projection source, the static random access memory (SRAM) unit 530 stores instructions, which may include: when executed by at least one of the processors, causing the screen projection source to implement the following steps: Figure 7 Instructions for the steps implemented by the screen projection source in the method shown.

[0160] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0161] Each method implementation mode of the present application can be implemented in the form of software, magnetic components, firmware, etc.

[0162] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0163] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described herein is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0164] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a computer-readable storage medium, which represent various logics in a processor, and when the instructions are read by a machine, the machine makes logic for performing the techniques described herein. These representations, known as "Intellectual Property (IP) cores," may be stored on tangible computer-readable storage media and provided to multiple customers or production facilities to be loaded into a manufacturing machine that actually manufactures the logic or processor.

[0165] In some cases, the instruction converter can be used to convert instructions from a source instruction set to a target instruction set. For example, the instruction converter can transform (e.g., using static binary transformation, dynamic binary transformation including dynamic compilation), deform, simulate, or otherwise convert instructions into one or more other instructions to be processed by the core. The instruction converter can be implemented in software, hardware, firmware, or a combination thereof. The instruction converter can be on the processor, outside the processor, or partially on the processor and partially outside the processor.

Claims

1. A screen projection method, characterized in that: Applied to a screen projection source end having a video call function, the screen projection method includes: In the video call state, receive the user's first call image sent by the screen projection destination; Generate a video call interface for display on the screen projection source end, wherein the video call interface includes the user's first call image and interface elements; Send the image data of the interface element to the projection destination so that the projection destination synthesizes the interface element and the user's second call image to generate an interface for display on the projection destination; wherein the user's second call image is an image captured by the camera of the projection destination when the projection destination receives the image data of the interface element.

2. The screen projection method according to claim 1, characterized in that: The interface elements of the video call interface include at least one of the following: controls for controlling the call status and / or interface display status, call status information, user identification, and call images of other users participating in the video call.

3. The screen projection method according to claim 2, characterized in that: The image data of the interface elements includes: image information of each element in the interface elements, and position information of each element on the video call interface.

4. The screen projection method according to claim 1, characterized in that: The starting conditions of the screen projection method are: The communication delay between the screen projection source and the screen projection destination exceeds a set threshold; and / or, The data buffer amount in the communication channel between the screen projection source and the screen projection destination exceeds a set threshold.

5. A screen projection method, characterized in that: Applied to the screen projection destination, the screen projection method includes: When the screen projection source end is in a video call state, sending the user's first call image to the screen projection source end, so that the screen projection source end generates a video call interface for display by the screen projection source end, and the video call interface includes the user's first call image and interface elements; Receiving image data of the interface element sent by the screen projection source end; The interface element and the user's second call image are synthesized to generate an interface for display on the screen projection destination; wherein the user's second call image is an image captured by the camera of the screen projection destination when the screen projection destination receives the image data of the interface element.

6. The screen projection method according to claim 5, characterized in that: The interface elements of the video call interface include at least one of the following elements: controls for controlling the call status and / or interface display status, call status information, user identification, and call images of other users participating in the video call.

7. The screen projection method according to claim 6, characterized in that: The image data of the interface elements includes: image information of each element in the interface elements, and position information of each element on the video call interface.

8. The screen projection method according to claim 5, characterized in that: The starting conditions of the screen projection method are: The communication delay between the screen projection source and the screen projection destination exceeds a set threshold; and / or, The data buffer amount in the communication channel between the screen projection source and the screen projection destination exceeds a set threshold.

9. A screen projection source terminal, characterized in that: The screen projection source end has a video call function, and the screen projection source end includes: A memory, used to store instructions executed by one or more processors of the screen projection source end; The processor, when the processor executes the instructions in the memory, can enable the screen projection source end to execute the screen projection method described in any one of claims 1 to 4.

10. A screen projection destination terminal, characterized in that: include: A memory, used to store instructions executed by one or more processors of the screen projection destination; The processor, when the processor executes the instructions in the memory, can enable the screen projection destination to execute the screen projection method described in any one of claims 5 to 8.

11. A screen projection system, characterized in that: The screen projection system includes a screen projection source end and a screen projection destination end, the screen projection source end has a video call function, wherein the screen projection source end is used to execute the screen projection method as described in any one of claims 1 to 4, and the screen projection destination end is used to execute the screen projection method as described in any one of claims 5 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, can enable the computer to execute the screen projection method described in any one of claims 1 to 4, or enable the computer to execute the screen projection method described in any one of claims 5 to 8.

Images