Electronic device and method performed by electronic device for providing application screen on external device display

Abstract

According to various embodiments, an electronic device includes: a display; a communication interface; a memory; and a processor operatively connected to the display, the communication interface, and the memory, wherein the processor is configured to: determine a connection with an external device through the communication interface; receive a request to run a first task from the external device; in response to the request to run the first task, run the first task based on an application stored in the memory; generate a first virtual screen including a running screen of the first task; send the first virtual screen to the external device; while running the first task, receive a request to run a second task from the external device; in response to the request to run the second task, run the second task based on the application stored in the memory; generate a second virtual screen including the running screen of the second task; and send the second virtual screen to the external device.

Description

TECHNICAL FIELD

[0001] The present document relates to an electronic device, and for example, to a method in which an electronic device can provide an application screen on a display of an external device by executing an application. BACKGROUND

[0002] As wireless communication technology and processor technology have developed, portable electronic devices, represented by smart phones (hereinafter, referred to as electronic devices), are equipped with various functions in addition to a call function. For example, an electronic device can provide an application screen via a display of the electronic device by executing various applications.

[0003] In order to facilitate user portability, an electronic device can have a display of a relatively small size. Accordingly, the electronic device can have a mirroring function of transmitting an application screen executed in the electronic device to an external device having a larger display, thereby providing the application screen on the display of the external device. In addition to simply transmitting a running screen of an application to the display of the external device, a user interface for an application of the electronic device is provided in a personal computer (PC) environment, and a service for controlling an operation such as execution of an application of the electronic device by user input in the PC environment is also being developed. SUMMARY

[0004] TECHNICAL PROBLEM

[0005] A conventional screen sharing service can allow a screen being displayed on an electronic device to be displayed as is on a display of an external device, or can provide an application screen of the electronic device on one window provided by the external device.

[0006] Various embodiments of the present document will simultaneously provide a plurality of tasks or application screens executed in an electronic device on a display of an external device.

[0007] SOLUTION TO PROBLEM

[0008] An electronic device according to various embodiments can include a display, a communication interface, a memory, and a processor operatively connected to the display, the communication interface, and the memory, wherein the processor is configured to: determine a connection with an external device via the communication interface; receive a request to execute a first task from the external device; execute the first task based on an application stored in the memory in response to the request to execute the first task, generate a first virtual display including a running screen of the first task, and transmit the first virtual display to the external device; during execution of the first task, receive a request to execute a second task from the external device; and execute the second task based on an application stored in the memory in response to the request to execute the second task, generate a second virtual display including a running screen of the second task, and transmit the second virtual display to the external device.

[0009] According to various embodiments, a method for an electronic device to provide a screen on a display of an external device may include: an operation of determining a connection with an external device; an operation of receiving a request to run a first task from the external device; an operation of running the first task based on an application stored in a memory in response to the request to run the first task, generating a first virtual display including a running screen of the first task, and sending the first virtual display to the external device; an operation of receiving a request to run a second task from the external device during the running of the first task; and an operation of running the second task based on an application stored in the memory in response to the request to run the second task, generating a second virtual display including a running screen of the second task, and sending the second virtual display to the external device.

[0010] According to various embodiments, an electronic device may include a display, a communication interface, and a processor operatively connected to the display and the communication interface, wherein the processor is configured to: determine a connection with an external device via the communication interface; in response to a first user input to run a first task that can be run in the external device, send a request to the external device to run the first task; receive a first virtual display including a running screen of the first task from the external device; in response to a second user input to run a second task that can be run in the external device, send a request to the external device to run the second task; receive a second virtual display including a running screen of the second task from the external device; and display the first virtual display and the second virtual display on different windows on the display.

[0011] Advantageous Effects of the Invention

[0012] According to various embodiments of this document, a plurality of tasks or application screens running in an electronic device may be simultaneously provided on a display of an external device.

[0013] To provide multiple task or application screens on an external device, various embodiments of this document may provide methods for generating and controlling screens, managing running applications, processing user input, and / or managing resources of an electronic device.

[0014] Furthermore, various technical effects identified directly or indirectly via this document may be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an electronic device in a network environment according to various embodiments;

[0016] Figure 2 shows examples of a first electronic device and a second electronic device according to various embodiments;

[0017] Figure 3a and Figure 3bElements of a first electronic device and a second electronic device for generating a virtual display during screen sharing are shown according to various embodiments;

[0018] Figure 4 is a block diagram of a first electronic device according to various embodiments;

[0019] Figure 5 is a block diagram of a second electronic device according to various embodiments;

[0020] Figure 6 is a block diagram of a first electronic device for providing a screen sharing service according to various embodiments;

[0021] Figure 7 is a block diagram of a second electronic device for providing a screen sharing service according to various embodiments;

[0022] Figure 8 shows examples of containers configured in a first electronic device and a second electronic device according to various embodiments;

[0023] Figure 9 is a flowchart of a method for providing a screen sharing service by a first electronic device and a second electronic device according to various embodiments;

[0024] Figure 10 is a flowchart of a method for processing events and user input by a first electronic device and a second electronic device according to various embodiments;

[0025] Figure 11 is a flowchart of a method performed by a first electronic device and a second electronic device when an application is terminated according to various embodiments;

[0026] Figure 12a and Figure 12b shows a screen provided on a display of a second electronic device according to various embodiments;

[0027] Figure 13 is a flowchart of a method of processing an execution screen by a first electronic device when a new task is executed during a screen sharing service according to various embodiments;

[0028] Figure 14a and Figure 14b illustrating screens provided by a first electronic device and a second electronic device when a new task is executed according to various embodiments;

[0029] Figure 15 is a flowchart of a method for processing a task on a new virtual display by a first electronic device according to various embodiments; and

[0030] Figure 16is a flowchart of a method for a first electronic device to process a task when the task is terminated immediately after being executed according to various embodiments. DETAILED DESCRIPTION

[0031] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Referring to Figure 1 , the electronic device 101 in the network environment 100 can communicate with an electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or an electronic device 104 or a server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 can communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 can include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one (e.g., the connection terminal 178) of the above components can be omitted from the electronic device 101, or one or more other components can be added in the electronic device 101. In some embodiments, some of the above components (e.g., the sensor module 176, the camera module 180, or the antenna module 197) can be implemented as a single integrated component (e.g., the display module 160).

[0032] The processor 120 can execute, for example, software (e.g., a program 140) to control at least one other component (e.g., a hardware or software component) of the electronic device 101 coupled with the processor 120 and can perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor 120 can store a command or data received from another component (e.g., the sensor module 176 or the communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store resulting data in the non-volatile memory 134. According to an embodiment, the processor 120 can include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)), or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor 121. For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 can be adapted to consume less power than the main processor 121, or to be specialized in a specific function. The auxiliary processor 123 can be implemented as separate from, or as part of, the main processor 121.

[0033] The auxiliary processor 123, rather than the main processor 121, can control at least some of the functions or status related to at least one component (for example, the display module 160, the sensor module 176, or the communication module 190) among the components of the electronic device 101 while the main processor 121 is in an inactive (for example, sleep) state, or the auxiliary processor 123 can control at least some of the functions or status related to at least one component (for example, the display module 160, the sensor module 176, or the communication module 190) among the components of the electronic device 101 together with the main processor 121 while the main processor 121 is in an active state (for example, executing an application). According to an embodiment, the auxiliary processor 123 (for example, an image signal processor or a communication processor) can be implemented as a part of another component (for example, the camera module 180 or the communication module 190) functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (for example, a neural processing unit) can include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model can be generated through machine learning. For example, such learning can be performed by the electronic device 101 where artificial intelligence is performed or via a separate server (for example, the server 108). The learning algorithm can include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model can include multiple artificial neural network layers. The artificial neural network can be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, or a combination of two or more thereof, but is not limited thereto. Additionally or alternatively, the artificial intelligence model can include a software structure other than the hardware structure.

[0034] The memory 130 can store various data used by at least one component (for example, the processor 120 or the sensor module 176) of the electronic device 101. The various data can include, for example, software (for example, a program 140) and input data or output data for commands related thereto. The memory 130 can include the volatile memory 132 or the non-volatile memory 134.

[0035] The program 140 can be stored in the memory 130 as software, and can include, for example, an operating system (OS) 142, middleware 144, or an application 146.

[0036] The input module 150 can receive a command or data, which is to be used by other components (for example, the processor 120) of the electronic device 101, from the outside (for example, a user) of the electronic device 101. The input module 150 can include, for example, a microphone, a mouse, a keyboard, a key (for example, a button), or a digital pen (for example, a stylus pen).

[0037] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module 160 can visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. Depending on the embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of the force caused by the touch.

[0039] The audio module 170 can convert sound into an electrical signal, and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0040] The sensor module 176 can detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. Depending on the embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0041] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). Depending on the embodiment, the interface 177 may include, for example, a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0042] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. Depending on the embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] The haptic module 179 can convert electrical signal into a mechanical stimulus (e.g., a vibration or movement) or electrical stimulus that can be recognized by a user via his tactile sensation or kinesthetic sensation. According to an embodiment of the present disclosure, the haptic module 179 can include, for example, a motor, a piezoelectric element, or an electrical stimuluser.

[0044] The camera module 180 can capture still images or moving images. According to an embodiment of the present disclosure, the camera module 180 can include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module 188 can manage power supplied to the electronic device 101. According to an embodiment of the present disclosure, the power management module 188 can be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0046] The battery 189 can supply power to at least one component of the electronic device 101. According to an embodiment of the present disclosure, the battery 189 can include, for example, a primary cell, a secondary cell, or a fuel cell.

[0047] The communication module 190 can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108) and performing communication between the electronic devices 101 and the external electronic device via the established communication channel. The communication module 190 can include one or more communication processors that are operable independently from the processor 120 (e.g., an application processor (AP)) and supports a direct (e.g., wired) communication or wireless communication. According to an embodiment of the present disclosure, the communication module 190 can include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can perform communication by using at least one of the first network 198 (e.g., a short-range wireless communication network, such as Bluetooth, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network 199 (e.g., a long-range wireless communication network, such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). The various types of communication modules can be implemented as a single component (e.g., a single chip) or can be implemented as separate components (e.g., separate chips) from each other. The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network, such as the first network 198 or the second network 199, using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module 196.

[0048] The wireless communication module 192 can support 5G networks and next-generation communication technologies (e.g., new radio (NR) access technology) after 4G networks. The NR access technology can support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable low-latency communications (URLLC). The wireless communication module 192 can support a high frequency band (e.g., a millimeter wave band) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for securing performance on a high frequency band, such as, for example, beamforming, massive multiple input multiple output (massive MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large scale antenna. The wireless communication module 192 can support various requirements designated in the electronic device 101, an external electronic device (e.g., the electronic device 104), or a network system (e.g., the second network 199). According to an embodiment, the wireless communication module 192 can support a peak data rate of eMBB (e.g., 20 Gbps or more) for implementation, a loss coverage (e.g., 164 dB or less) for mMTC implementation, or a U-plane latency (e.g., 0.5 ms or less for each of downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC implementation.

[0049] The antenna module 197 can transmit or receive a signal or power to or from the outside (e.g., an external electronic device) of the electronic device 101. According to an embodiment, the antenna module 197 can include an antenna including a radiating element composed of a conductive material or a conductive pattern formed in or on a base (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 can include a plurality of antennas (e.g., array antennas). In this case, at least one antenna suitable for a communication scheme used in a communication network, such as the first network 198 or the second network 199, can be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). Then, a signal or power can be transmitted or received between the communication module 190 and an external electronic device via the selected at least one antenna. According to an embodiment, in addition to the radiating element, another component (e.g., a radio frequency integrated circuit (RFIC)) can be additionally formed as part of the antenna module 197.

[0050] According to various embodiments, the antenna module 197 can form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module can include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., a bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high frequency band (e.g., a millimeter wave band), and the plurality of antennas is disposed on a second surface (e.g., a top surface or a side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving a signal of the designated high frequency band.

[0051] At least some of the above-described components can be connected to each other by an inter-chip communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicate information (e.g., commands or data) between them.

[0052] According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type than the electronic device 101. According to an embodiment, all or some operations to be executed on the electronic device 101 may be executed on one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute at least part of the function or service requested, or execute another function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request, with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0053] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those described above.

[0054] It should be understood that various embodiments of the present disclosure and the terms used therein are not intended to limit technically described features to particular embodiments and include various changes, equivalents, or replacements for a corresponding embodiment. With regard to the description of the drawings, like reference numerals can be used to refer to like or similar elements. It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. As used herein, each of the phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can include all possible combinations of the items listed in the corresponding one of the phrases. As used herein, the terms such as "1st" and "2nd," or "first" and "second" can be used to simply distinguish a corresponding component from another, and do not limit the components in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being "operatively or communicatively connected" to or with another element (e.g., a second element), or as being "connected" to or with the other element without using the term "operatively" or "communicatively," it means that the element can be connected to the other element directly (e.g., with a wired line) or wirelessly, or connected to the other element via a third element.

[0055] As used in connection with various embodiments of the present disclosure, the term "module" can include a unit implemented in hardware, software, or firmware, and can interchangeably be used with other terms such as "logic," "logic block," "part," or "circuitry." A module can be a single integral component, or a minimum unit or part thereof, adapted to perform one or more functions. For example, a module can be implemented in a form of an application-specific integrated circuit (ASIC).

[0056] Various embodiments as set forth herein can be implemented as software (e.g., the program 140) including one or more instructions that are stored in a storage medium (e.g., internal memory 136 or external memory 138) that are readable by a machine (e.g., electronic device 101). For example, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke at least one of the one or more instructions stored in the storage medium, and execute it, with or without using one or more other components under the control of the processor. This allows the machine to be operated as a special purpose machine to perform at least one function. The one or more instructions can include a code generated by a compiler or a code that forms at least a part of a language as provided in a high-level programming language. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. The term "non-transitory" simply means that the storage medium is tangible, but does not include a signal (e.g., an electromagnetic wave). The term "non-transitory" does not distinguish between where data is semi-permanently stored in the storage medium and where the data is temporarily stored in the storage medium.

[0057] According to an embodiment, a method according to various embodiments of the disclosure can be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed online via an application store (e.g., Play StoreTM). If distributed online, at least part of the computer program product can be temporarily stored in a storage medium such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0058] According to various embodiments, each component (for example, module or program) in the above-mentioned components may include a single entity or multiple entities, and some entities in the multiple entities may be separably arranged in different components. According to various embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (for example, module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components before integration. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in different orders, or one or more other operations may be added.

[0059] In the following text, a device that runs an application and sends an application screen is defined as a first electronic device or source device, and a device that receives an application screen and outputs it on its own display or another connected display device is defined as a second electronic device or sink device. Depending on the actual implementation, the same device can operate as either the first electronic device or the second electronic device in the text.

[0060] Figure 2 Examples of a first electronic device and a second electronic device according to various embodiments are shown.

[0061] In various embodiments of this document, the first electronic device 200 (eg, Figure 1 The first electronic device 101) may be implemented as a portable electronic device, such as a smart phone or a tablet PC, and the second electronic device 250 may be implemented as a computing device, such as a desktop PC or a laptop PC.

[0062] According to various embodiments, the first electronic device 200 may store an application in a memory and execute the application. For example, the first electronic device 200 may execute various applications such as an Internet browser, a gallery, a message, and a game, and may provide an execution screen of the application on the display of the first electronic device 200.

[0063] According to various embodiments, the second electronic device 250 may run an application via a processor, memory, and operating system independent of the first electronic device 200. The second electronic device 250 may output an application screen via a separate display (e.g., monitor) as in a desktop PC, or may be provided integrally with a display as in a laptop PC.

[0064] According to various embodiments, the first and second electronic devices 200 and 250 can be connected to each other via wired or wireless communication. For example, the first and second electronic devices 200 and 250 can be connected to each other via a wired interface such as a high-definition multimedia interface (HDMI) or a universal serial bus (USB), and can be connected to each other via a short-range wireless communication network such as Wi-Fi direct or Bluetooth.

[0065] According to various embodiments, if the first and second electronic devices 200 and 250 are connected via wired or wireless communication, a screen sharing service can be provided. The screen sharing service described in the file can be a service or function that is a scheme in which the first electronic device 200 runs an application of the first electronic device 200 in the first electronic device 200, in which the second electronic device 250 displays the application of the first electronic device 200 on a display of the second electronic device 250 by using a virtualization technology, and provides user input to the application to the first electronic device 200. For example, the screen sharing service can be a service or function that allows an application of a portable electronic device such as a desktop experience (DeX) and MS Connect to Windows to be used in a PC environment.

[0066] According to various embodiments, if the first and second electronic devices 200 and 250 are connected via a wired or wireless interface, the first electronic device 200 can transmit an application execution screen generated by a processor and a memory of the first electronic device 200 to the second electronic device 250. The second electronic device 250 can display the application execution screen 255 received from the first electronic device 200, and can transmit user input to the application to the first electronic device 200. For example, when the first and second electronic devices 200 and 250 are connected, the second electronic device 250 can run an application for a screen sharing service, and can receive application list information from the first electronic device 200, thereby displaying icons corresponding to the applications of the first electronic device 200 on a display of the second electronic device 250 (or a display connected to the second electronic device 250). The second electronic device 250 can transmit user input to the icons to the first electronic device 200, the first electronic device 200 can run the corresponding applications according to the user input, thereby transmitting an application execution screen to the second electronic device 250. While the application screen 255 is displayed on the display of the second electronic device 250, the second electronic device 250 can transmit user input in real time on a window providing the application screen to the first electronic device 200, and the first electronic device 200 can control the application based on the received user input.

[0067] According to various embodiments, if a plurality of applications or tasks are run in the first electronic device 200, an execution screen of each of the applications or tasks can be provided via a separate window on a display of the second electronic device 250.

[0068] Figure 3a Elements of a first electronic device for generating a virtual display during screen sharing according to various embodiments are illustrated.

[0069] According to various embodiments, the first electronic device 300 can configure a virtual display 360 (virtual display) to transmit a screen generated by an application to the second electronic device 350. Referring to Figure 3a , the first electronic device 300 can include a surface flinger 375, a video encoder 364, an input surface 362, and a display manager 370 as software components for configuring the virtual display 360.

[0070] According to various embodiments, the display manager 370 can be used to configure data generated by an application for a screen sharing service into the virtual display 360. The virtual display 360 is a virtual display device, and can have one surface on which screen information can be drawn. The video encoder 364 can compress data generated by an application in order to improve transmission efficiency.

[0071] According to various embodiments, when a screen sharing service is turned on, the display manager 370 can generate the virtual display 360. The display manager 370 can generate the video encoder 364, and can designate the input surface 362 of the video encoder 364 as a surface of the virtual display 360. Here, the surface of the virtual display 360 is the input surface 362 of the video encoder 364, such that if the content on the surface is updated, the data can be compressed into a video form and output through an output buffer. The first electronic device 300 can transmit the video-encoded data as described above to the second electronic device 350.

[0072] Figure 3b Elements of a second electronic device for receiving and outputting a virtual display during screen sharing according to various embodiments are illustrated.

[0073] Referring to Figure 3a , the second electronic device 350 can include a surface view 380, a video decoder 384, and an output surface 382 as software components for outputting the virtual display 360 received from the first electronic device 300.

[0074] According to various embodiments, when screen sharing data is transmitted from the first electronic device 300, the second electronic device 350 can generate a surface view 380 for content reproduction. The data transmitted from the first electronic device 300 is compressed data, such that the second electronic device 350 can generate a video decoder 384 to change the data back to original data, and can acquire an output surface 382 from the video decoder 384.

[0075] When the data transmitted from the first electronic device 300 is input to the input buffer of the video decoder 384, the screen data can be drawn on the surface view 380 via the configured pipe, and thus, the application screen transmitted from the first electronic device 300 can be output on the second electronic device 350.

[0076] According to various embodiments, in order to independently provide the execution screens of a plurality of applications and tasks, the first electronic device 300 can configure the video encoder 364 for configuring the virtual display 360 generated by each application or task, the input surface 362, and the virtual display 360 as one container. In response thereto, the second electronic device 350 can configure the video decoder 384, the output surface 382, and the surface view 380 as one container. This will be described with reference to Figure 8 The feature of configuring an independent container for each application or task in the first electronic device 300 and the second electronic device 350 will be described in more detail.

[0077] Figure 4 is a block diagram of a first electronic device according to various embodiments.

[0078] Referring to Figure 4 , the first electronic device 400 (e.g., Figure 2 the first electronic device 200 of FIG. 1) can include a first display 410, a first communication interface 420, a first memory 430, and a first processor 450, and can implement various embodiments of the present document even though at least some of the illustrated elements are omitted and / or replaced. The first electronic device 400 can further include at least some elements and / or functions of the electronic device 101 of Figure 1

[0079] Figure 4 Some of the illustrated elements (e.g., the first processor 450, the memory 430, and the first communication interface 420) and / or other elements of the first electronic device 400 not shown can be disposed inside a housing (not shown) of the first electronic device 400, and at least a part of some elements (e.g., the first display 410 and the first communication interface 420) can be exposed to the outside of the housing.

[0080] ​According to various embodiments, the first display 410 displays an image, and can be implemented as one of a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical system (MEMS) display, or an electronic paper display 310, but is not limited thereto. According to various embodiments, the first display 410 can be a foldable display (e.g., an inward foldable display or an outward foldable display) at least a portion of which can be folded, or can be a rollable display at least a portion of which can be rolled. Alternatively, the first electronic device 400 can include two or more displays independent of each other. The first display 410 can include at least some elements and / or functions of the display module 160. Figure 1

[0081] According to various embodiments, the first communication interface 420 can support wired or wireless communication connection with an external device (e.g., the second electronic device 250), and the first electronic device 400 can include only one of a wired communication interface or a wireless communication interface. Figure 2

[0082] For example, the first communication interface 420 is an example of a wired communication interface (e.g., the interface 177), and can include a high-definition multimedia interface (HDMI) and a universal serial bus (USB) interface. The first communication interface 420 can include a wireless communication module (e.g., the wireless communication module 192) supporting a short distance communication module with an external device. The wireless communication module can support various short distance wireless communication schemes (e.g., Wi-Fi, Bluetooth, and Bluetooth low energy (BLE)), and can include independent hardware and / or software elements to support each wireless communication scheme. Figure 1 Figure 1

[0083] According to various embodiments, the first storage 430 is used to temporarily or permanently store unrestricted digital data, and can include at least some elements and / or functions of the storage 130. Figure 1 The first storage 430 can include a non-volatile memory such as one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, flash ROM, a flash memory (e.g., NAND flash or NOR flash), a hard drive, or a solid state drive (SSD), and a volatile memory such as dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM).

[0084] ​​​​The first memory 430 can store various instructions executable by the first processor 450. The instructions can include various control commands, including arithmetic and logical operations, data movement, and / or input / output recognizable by the first processor 450.

[0085] According to various embodiments, the first processor 450 is an element (e.g., an application processor) capable of performing operations or data processing related to control and / or communication of each element of the first electronic device 400, and thus can include at least some elements and / or functions of the processor 120 of FIG. 1. Figure 1 The first processor 450 can be operatively, functionally, and / or electrically connected to each element of the first electronic device 400, including the first display 410, the first communication interface 420, and the first memory 430.

[0086] According to various embodiments, although there is no limitation on the arithmetic and data processing functions that the first processor 450 can implement on the first electronic device 400, this document provides a description of various embodiments related to generation and control of a running screen of a plurality of applications or tasks, management of a running application, processing of a user input, and resource management of the first electronic device 400 in a screen sharing service. The operations of the first processor 450, which will be described later, can be performed by running instructions stored in the first memory 430.

[0087] According to various embodiments, if connected to the second electronic device via the first communication interface 420, the first processor 450 can transmit, to the second electronic device, a list of applications installed in the first electronic device 400.

[0088] According to various embodiments, the first processor 450 can receive, from the second electronic device via the first communication interface 420, a request for running a first task. For example, the second electronic device can display icons corresponding to each application on a display of the second electronic device based on the received list of applications, and can transmit, to the first electronic device 400, a request for running a corresponding application or task according to a user input for the icons.

[0089] According to various embodiments, in response to the request for running, the first processor 450 can run the first task based on an application stored in the memory.

[0090] According to various embodiments, the first processor 450 can generate a first virtual display including a running screen of a first task. The first processor 450 can configure a first encoder that compresses first image information generated in the first task, a first input surface that records the first image information, and the first virtual display as a first container (e.g., an application running container). Here, the container refers to an object (or class or instance) on a program that manages each virtual display, and can be defined by different names. For example, the first container and the second container can be defined by different names such that the first container can be defined by a first virtual display management group, the second container can be defined by a second virtual display management group, and the like.

[0091] According to various embodiments, the first processor 450 can generate a video encoder in response to a request for running a first task, and can designate an input surface of the video encoder as a surface of the first virtual display. The first processor 450 can manage components (e.g., a virtual display, an input surface, and a video encoder) that run a task or an application to be run as a container, and can generate a separate container for each application or task. Each container can have an ID, and can be mapped to a container of the second electronic device via the ID. Each container can further include at least some of an ID, a display ID, an application package name, display-related information, a task ID, task termination reason information, and timestamp information. The container configured by the first processor 450 will be described in more detail with reference to Figure 6

[0092] According to various embodiments, the first electronic device 400 and the second electronic device can configure independent containers in units of tasks or applications.

[0093] According to various embodiments, the first electronic device 400 and the second electronic device can respectively configure corresponding containers. For example, if an application running container 1 is generated according to the running of a task (or an application) in the first electronic device 400, and related information is transmitted to the second electronic device, the second electronic device can generate a corresponding remote application container 1. Thereafter, if an application running container 2 is generated according to the running of a new task (or an application) in the second electronic device, the second electronic device can generate a corresponding remote application container 2. According to various embodiments, containers corresponding to each other can form one mirroring session, which can be independent of mirroring sessions formed between other containers. For example, a mirroring session formed between the application running container 1 and the remote application container 1 and a mirroring session formed between the application running container 2 and the remote application container 2 can be independent of each other. The first electronic device and the second electronic device can classify each mirroring session according to an ID of a container or a virtual display.

[0094] ​According to various embodiments, the mirror session formed between the corresponding containers can be connected via mutually independent communication channels. For example, if the first electronic device 400 and the second electronic device are connected via a USB interface, the mirror session formed by the respective containers can communicate via mutually independent USB channels (logical channels), and if the first electronic device 400 and the second electronic device are connected via Wi-Fi (or WFD), the respective mirror sessions can communicate by forming mutually independent Wi-Fi channels.

[0095] According to various embodiments, the generated first virtual display can be transmitted to the external device. For example, the first container including the first virtual display can be transmitted to the external device. The second electronic device can generate a container corresponding to the first container to receive the virtual display, and the container of the first electronic device 400 and the corresponding container of the second electronic device can configure a mirror session.

[0096] According to various embodiments, while the first task is executed, the first processor 450 can receive a request for executing a second task from the external device. Here, the second task can be the same application as the first task or a task executed by a different application.

[0097] According to various embodiments, in response to the request for executing the second task, the first processor 450 can execute the second task and generate a second virtual display including an execution screen of the second task. For example, the first processor 450 can configure components constituting the second virtual display as a second container. The second container can have the same configuration as that of the first container.

[0098] According to various embodiments, if an input event of the virtual display is received from the second electronic device, the first processor 450 can process the input event in the mapped first task or the second task by using identification information included in the received input event. For example, if a user input (e.g., a mouse click or a touch input) occurs on the first virtual display in the second electronic device, the second electronic device can transmit input event information and an ID of the first virtual display (or an ID of the container) to the first electronic device 400. The first processor 450 can identify the virtual display mapped to the received ID, and can transmit the input event to an application configuring the corresponding virtual display.

[0099] According to various embodiments, upon receiving an intention to run a third task on the first virtual display, the first processor 450 can be configured to identify a virtual display corresponding to the third task. If a virtual display corresponding to the third task is not generated, the first processor 450 can generate a third virtual display including a running screen of the third task and transmit the third virtual display to the external device. If a virtual display corresponding to the third task is generated, the first processor 450 can cause the third task to operate in the foreground.

[0100] According to various embodiments, the first processor 450 can be configured to, when a running screen of the third task is displayed on the first display 410, stop displaying the running screen of the third task on the first display 410, generate a third virtual display including the running screen of the third task, and transmit the third virtual display to the external device or transmit the running screen of the third task being displayed on the first display 410 to the external device.

[0101] In this case, the first processor 450 can display a dialog for determining whether to move a running location of a currently running task to a display of the second electronic device. If the user allows the movement of the running location of the task on the dialog, the first processor 450 can terminate an application running on the first display 410 and can run the third task on the newly generated third virtual display.

[0102] According to various embodiments, when a request to run a new task is received from the external device, the first processor 450 can identify a number of currently generated virtual displays, and if the number of virtual displays exceeds a maximum number, can terminate at least one generated virtual display. The first processor 450 can terminate a virtual display (or container) that is generated earliest or a virtual display (or container) that is deactivated earliest.

[0103] According to an embodiment, a maximum number of virtual displays that can be simultaneously run can be pre-allocated according to a processor and / or a memory included in the first electronic device 400. For example, the maximum number of virtual displays can be determined as a pre-defined number of each chip set (e.g., the first processor 450 and / or the first memory 430) of the electronic device. For example, the maximum number of virtual displays can be determined as a number defined for each type (e.g., a processor name, an identification (ID), and / or a clock) of the first processor 450 and / or a size of the first memory 430.

[0104] According to an embodiment, the first processor 450 can determine the maximum number of virtual displays based on available resources of the first processor 450 and / or the first memory 430. The available resources can be determined according to the size (or area) of the encoder resources allocated to the first processor 450 and / or the first memory 430 and the size of resources required to transmit the current virtual display. For example, the first processor 450 can determine a value obtained by dividing the area of the encoder surface that can be implemented on the first memory 430 by the maximum resolution provided to the virtual display as the maximum number of virtual displays.

[0105] According to an embodiment, the first processor 450 can sense the temperature of the first electronic device 400 and can determine the maximum number of virtual displays according to the sensed temperature. For example, the first electronic device 400 can measure the temperature of one or more regions inside the first electronic device 400 via an element such as a thermistor and can determine the maximum number of virtual displays according to the measured temperature interval.

[0106] According to an embodiment, the first processor 450 can determine the maximum number of virtual displays based on the transmission rate of data transmitted to the second external device via the first communication interface 420. For example, the first processor 450 can determine the maximum number of virtual displays according to the interval of the average data transmission rate.

[0107] According to various embodiments, when a specific task is terminated, the first processor 450 can transmit task termination reason information (for example, terminated because it has been running for a long time, or terminated due to the running of a new task) to the second electronic device. When receiving the task termination reason information, the second electronic device can close the corresponding container in response to the task termination reason information, and can remove the window of the virtual display displaying the corresponding container on the display.

[0108] Figure 5 is a block diagram of a second electronic device according to various embodiments.

[0109] Referring to Figure 5 , the second electronic device 500 (for example, Figure 2 the second electronic device 500 of FIG. 1) can include a second display 510, a second communication interface 540, a second memory 530, and a second processor 550, and various embodiments of the present document can be implemented even if at least some of the illustrated elements are omitted and / or replaced. According to an embodiment, some of the illustrated elements can be provided on another device separate from the second electronic device 500 and connected via a wired or wireless communication interface. For example, the second electronic device 500 can be implemented as a desktop PC that does not include a display, and in this case, the second display 510 can be implemented on a separate display device.

[0110] According to various embodiments, the second display 510 may display an image. For example, the second display 510 may display an image displayed by the first electronic device (eg, Figure 4 The second display 510 may be implemented as an external display device independent of the second electronic device 500.

[0111] According to various embodiments, the second communication interface 540 may support a wired or wireless connection with the first communication interface of the first electronic device. If the second communication interface 540 is implemented as a wired interface, an interface terminal to which a cable is connected may be provided on the second electronic device 500 or an external display device.

[0112] According to various embodiments, the second memory 530 may include a nonvolatile memory and a volatile memory.

[0113] According to various embodiments, the second processor 550 is an element capable of performing operations or data processing related to control and / or communication of corresponding elements of the second electronic device 500, and can be operatively connected, functionally connected, and / or electrically connected to corresponding elements of the second electronic device 500, including the second display 510, the second communication interface 540, and the second memory 530. The second processor 550 can run a plurality of applications via an operating system (e.g., Windows) independent of the first electronic device.

[0114] According to various embodiments, the second processor 550 may recognize a connection with the first electronic device via the second communication interface 540. The second processor 550 may receive a list of applications installed in the first electronic device from the connected first electronic device, and may display an icon corresponding to each application on the second display 510 based on the list.

[0115] According to various embodiments, in response to a user input for an icon displayed on the second display 510, the second processor 550 may send a request to an external device to execute a first task corresponding to the icon. Here, the user input may include a mouse click and a touch / hover input. The first electronic device may execute the first task in response to the execution request and may generate a first virtual display and transmit the first virtual display to the second electronic device 500.

[0116] According to various embodiments, the second processor 550 can receive the first virtual display from the first electronic device, and can display the first virtual display on the second display 510. The first electronic device can configure a virtual display, an encoder, and an input surface for each task as each container, and the second processor 550 can generate a container (e.g., a remote application container) corresponding to the container (e.g., an application running container) generated by the first electronic device. Each container can have an ID, and can be mapped to the container of the first electronic device via the ID. The containers corresponding to each other between the first electronic device and the second electronic device can form one mirroring session, which can be independent of the mirroring sessions formed between other containers. The mirroring session formed between the corresponding containers can be connected to other mirroring sessions via mutually independent communication channels (e.g., a USB channel and a Wi-Fi channel). The container configured by the second processor 550 will be described with reference to Figure 7 in more detail.

[0117] According to various embodiments, when transmitting a virtual display or container related information, the second processor 550 can determine whether to include a flag indicating that a plurality of tasks' virtual displays are transmitted, and if the flag is included, can configure the received respective virtual displays into different mirroring sessions, and can display the respective virtual displays on windows independent of each other. According to various embodiments, while the first virtual display is displayed, the second processor 550 can transmit a request for running a second task to the first electronic device in response to a second user input for running the second task. The second processor 550 can receive a second virtual display including a running screen of the second task from the first electronic device.

[0118] According to various embodiments, the second processor 550 can display the received first virtual display and the second virtual display on different windows. For example, the content of one application, task, or activity can be mirrored on each screen (or window) shared during a screen sharing service. The second processor 550 can configure a surface view drawn by a respective container in different windows so that the running screen of the respective task is displayed on different windows.

[0119] According to various embodiments, the second processor 550 can receive a user input in a state in which the first virtual display and the second virtual display are displayed. The second processor 550 can transmit input event information and an ID of a virtual display on which the user input has been received to the first electronic device, and the first electronic device can process the input event received for the virtual display mapped to the ID.

[0120] Figure 6 is a block diagram of a first electronic device for providing a screen sharing service according to various embodiments.

[0121] Figure 6 Each of the illustrated elements is a software component for a screen sharing service that can be stored and executed in the first electronic device 600 (e.g., the first electronic device 400 of FIG. 4). Figure 4 Each component name defines a class name for performing a related operation in a program, and does not limit the operation performed according to each component name.

[0122] Referring to Figure 6 , the first electronic device 600 can include a transmission manager 620, an input manager 625, an application execution container manager 610, and a system service component 630. The system service component 630 is a collection of service components that are executed at a system level, and can include a remote application mode service 631, a display manager 633, an active task manager 635, a task observer 637, and a rotation manager. At least some of the illustrated elements (e.g., the transmission manager 620, the input manager 625, the display manager 633, and the active task manager 635) can be components implemented on an Android framework, but some functions can be added to implement various embodiments.

[0123] According to various embodiments, the first electronic device 600 can manage components (e.g., a virtual display, an input surface, and a video encoder) that execute a task or an application to be executed as a container, and can generate a separate container for each application or task.

[0124] According to various embodiments, the application execution containers 612 and 614 can be used to generate and control respective virtual displays in the first electronic device 600. The application execution containers 612 and 614 can have an ID as an identifier, and can be mapped to designated remote application containers (e.g., the remote application containers 772 and 774 of the second electronic device 700) of the second electronic device via the ID. Figure 7

[0125] According to various embodiments, the content of one application, task, or activity can be mirrored onto each screen (or window) shared during a screen sharing service, and thus the application execution containers 612 and 614 can have a package name of an application or a task ID. Table 1 below illustrates an example of data owned and / or managed by the application execution containers 612 and 614.

[0126] [Table 1]

[0127]

[0128]

[0129] ​According to various embodiments, the application running container manager 610 can control overall logic related to a screen sharing service in the first electronic device 600. For example, the application running container manager 610 can receive an application (or task) running request from a second electronic device (e.g., the second electronic device 500) connected via a communication interface (e.g., the first communication interface 420) of the first electronic device 600, thereby generating, owning, and managing an application running container. When a request related to an application is received from the second electronic device, the application running container manager 610 can distribute the received request to a designated application running container by using an ID of the application running container. When an event occurs on the system of the first electronic device 600, the application running container manager 610 can designate an application running container by using a virtual display ID or a task ID, thereby transmitting the event to the application running container, or can be used for performing transmission to a remote application container manager (e.g., the remote application container manager 770) in the second electronic device. According to various embodiments, the transmission manager 620 can transmit screen data encoded by the application running container, user input information, various requests, and / or event information to the second electronic device. Figure 4 Figure 5 Figure 7

[0130] According to various embodiments, the input manager 625 can be a component that injects an input event transmitted from the second electronic device into the system. In the input manager 625, when a user input is sensed by an input listener of the second electronic device, this can be reported to a remote application container, and an input event can be transmitted to a corresponding application running container of the first electronic device 600 via the transmission manager 620 of the second electronic device. The application running container can inject the input event coming via the input manager 625 into the system.

[0131] According to various embodiments, the system service component 630 can include a service component running at a system level.

[0132] According to various embodiments, the remote application mode service 631 can perform a function that requires a system authority and a function of directly calling a function of each system service. For example, the remote application mode service 631 can receive a request from the application running container manager 610 and directly transmit the request to the display manager 633 or the active task manager 635, or can be used to receive a callback for various events from the system service component 630 (e.g., the active task manager 635, the task watcher 637, the rotation watcher 639, or the start activity interceptor) and transmit the callback to the application running container manager 610.

[0133] According to various embodiments, the display manager 633 can be used to generate a virtual display according to the running of an application. ​​​

[0134] According to various embodiments, the activity task manager 635 can be used to manage activities and tasks. The activity task manager 635 can perform a start activity interception related operation to run an application or a task in one mirror session.

[0135] According to various embodiments, the task observer 637 can receive a task running or termination event, and can use the event to process an application running or termination, sense a task running mode, and provide a notification to a user.

[0136] According to various embodiments, the rotation observer 639 can sense a rotation event of a virtual display, and can use the event to provide mirror content rotated in the same manner from the second electronic device.

[0137] Figure 7 is a block diagram of a second electronic device for providing a screen sharing service according to various embodiments.

[0138] Figure 7 Each element shown is a software component for a screen sharing service that can be stored and run in the second electronic device 750 (e.g.,the second electronic device 500 of FIG. 1A). Figure 5 Each component name defines a class name for performing a related operation in a program, and does not limit an operation performed according to each component name.

[0139] According to various embodiments, the transmission manager 760 corresponds to the transmission manager 760 (e.g., the transmission manager 620 of FIG. 1A) of the first electronic device, and can perform a function of transmitting or receiving virtual display data, user input information, various requests, and / or event information to or from the first electronic device. Figure 6

[0140] According to various embodiments, the input listener 765 can monitor a user input inputted on a virtual display. When a user input is sensed, the input listener 765 can report the user input to the remote application containers 772 and 774 of the virtual display in which the user input has occurred, and an input event can be transmitted to the first electronic device via the transmission manager 760.

[0141] According to various embodiments, the remote application containers 772 and 774 are used to display an application or a task designated in the second electronic device 750, and each remote application container 772 or 774 can correspond to a virtual display of one mirror. The remote application containers 772 and 774 have IDs, and can be mapped to an application running container (e.g., Figure 5application running containers 612 and 614) of the first electronic device. If the function of the remote application container 772 or 774 is called, the function of the application running container of the first electronic device corresponding to the called remote application container can be called via the remote application container manager 770, the transmission manager 760, and the application running container manager 770.

[0142] According to various embodiments, the remote application container manager 770 can generate and control the remote application containers 772 and 774 corresponding to the respective virtual displays in the second electronic device 750. The remote application containers 772 and 774 can receive events from the input listener 765 and / or the first electronic device, and can transmit the same to the respective remote application containers 772 and 774.

[0143] Figure 8 An example of containers configured in the first electronic device and the second electronic device according to various embodiments is illustrated.

[0144] According to various embodiments, the first electronic device (e.g., Figure 4 The first electronic device 400) can configure encoders (e.g., first and second encoders 814 and 824) (video encoders) in units of containers that compress image information generated by running tasks, input surfaces (e.g., first and second input surfaces 812 and 822) recording the image information, and virtual displays (e.g., first and second virtual displays 816 and 826) (virtual). For example, with respect to a screen sharing service, the first electronic device can generate and manage application running containers 810 and 820 corresponding to the number of applications or tasks being run.

[0145] According to various embodiments, the application running containers 810 and 820 can be used to generate and control respective virtual displays in the first electronic device, and the respective application running containers 810 and 820 can have IDs and can be mapped to designated remote application containers 860 and 870 of the second electronic device via the IDs.

[0146] According to various embodiments, the application running containers 810 and 820 of the first electronic device and the remote application containers 860 and 870 of the second electronic device can include components corresponding to each other. For example, if one application or task is run in a state in which a screen sharing service is turned on, the first electronic device can generate a first application running container 810, and the second electronic device can generate a first remote application container 860. Then, if a new application or task is run, the first electronic device can generate a new second application running container 820, and the second electronic device can generate a second remote application container 870. The corresponding application running containers and remote application containers can configure mirror sessions.

[0147] According to various embodiments, containers corresponding to each other can configure independent mirroring sessions. For example, the first application running container 810 and the first remote application container 860 can configure a first mirroring session, and the second application running container 820 and the second remote application container 870 can configure a second mirroring session. The mirroring sessions formed between the containers corresponding to each other can be connected through mutually independent communication channels. For example, if the first electronic device and the second electronic device are connected via a USB interface, the first mirroring session and the second mirroring session can be configured via mutually independent USB channels (logical channels), and if the first electronic device and the second electronic device are connected via Wi-Fi (or WFD), the first mirroring session and the second mirroring session can be configured via mutually independent Wi-Fi channels.

[0148] According to various embodiments, data received from the respective application running containers 810 and 820 is provided to the corresponding respective remote application containers 860 and 870 by using IDs, and the remote application containers can decode the incoming data and render the data in surface views. The remote application container manager 850 can configure the surface views 864 and 874 rendered by the respective remote application containers 860 and 870 in different windows, and can display the surface views 864 and 874 on a display (e.g., the second display 510 of the Figure 5

[0149] According to various embodiments, the application running containers 810 and 820 and the remote application containers 860 and 870 can transmit or receive user input and event information to or from each other via IDs of each other. For example, when a user input is received on a feature task running screen of the second electronic device, the application running container manager 800 can distribute the received user input by determining the application running container in which the user input is sensed based on an ID received together with the user input.

[0150] Figure 9 is a flowchart of a method of providing a screen sharing service by a first electronic device and a second electronic device according to various embodiments.

[0151] Figure 9 An operation is illustrated in which the first electronic device 900 (e.g., the first electronic device 400 of Figure 4 ) and the second electronic device 950 (e.g., the second electronic device 500 of Figure 5 ) are connected to each other via a wired or wireless interface, and a virtual display generated by the first electronic device 900 is displayed on the second electronic device 950 after a screen sharing service application is executed.

[0152] ​According to various embodiments, the first and second electronic devices 900 and 950 can be connected via short-range wireless communication such as Bluetooth, Wi-Fi, and Wi-Fi direct, or wired interface such as USB and HDMI, and can be connected in various ways such as TCP / IP and UDP via a cloud service, optionally. The first and second electronic devices 900 and 950 can generate a plurality of transmission channels, and can transmit control data, video data, and audio data of an application through different channels, respectively. For example, the first and second electronic devices 900 and 950 open three TCP / IP sockets as server-client, and after socket connection, the socket communication channel can be used as a control channel, a video data channel, and an audio data channel, respectively.

[0153] According to various embodiments, the first electronic device 900 can transmit information of an application name, an icon, and / or a package name of an installed application to the second electronic device 950, and the second electronic device 950 can parse the received information and display the parsed information on a screen of the second electronic device 950.

[0154] According to various embodiments, in operation 960, the second electronic device 950 can receive a user input for running an application of the first electronic device 900.

[0155] According to various embodiments, in operation 962, the second electronic device 950 can generate and / or initialize a remote application container manager (for example, the remote application container manager 770 of FIG. 7). Figure 7

[0156] According to various embodiments, in operation 964, the remote application container manager can generate a new remote application container according to an application running event, and can configure information of an application selected according to a user input for the generated remote application container. The remote application container manager can configure a virtual touch input panel for receiving a user input (for example, a mouse click and a touch / hover input) when the application is used for the remote application container, and can connect the virtual touch input panel to an input listener (for example, the input listener of FIG. 7). Figure 7

[0157] According to various embodiments, in operation 966, the remote application container manager can run the remote application container generated in operation 964.

[0158] According to various embodiments, in operation 968, the second electronic device 950 can transmit an application execution request to an application running container manager (for example, the application running container manager of FIG. 7) of the first electronic device 900 via a transmission manager (for example, the transmission manager 760 of FIG. 7). Figure 7 Figure 6 ​​​application (or task) running request can include package name information of the application to be run.

[0159] According to various embodiments, in operation 970, the second electronic device 950 can prepare an application mirror. For example, the second electronic device 950 can allocate and initialize resources for mirroring the virtual display received from the first electronic device 900. The second electronic device 950 can generate a window for displaying the virtual display, and can generate a surface view. The second electronic device 950 can be allocated a decoder for decoding the encoded virtual display data from the decoder manager, and can connect the output surface of the decoder to the surface view.

[0160] According to various embodiments, in operation 972, the second electronic device 950 can run the allocated decoder. The foregoing operations 970 and / or 972 performed by the second electronic device 950 can be performed at least partially simultaneously with operations 974 to 984 to be described later performed by the first electronic device 900, or can be performed later.

[0161] According to various embodiments, when the application (or task) running request is received from the second electronic device 950, in operation 974, the first electronic device 900 can generate and / or initialize an application running container manager (e.g., Figure 6 application running container manager 610). If the application running container manager has already been generated and / or initialized, operation 974 can be omitted. The first electronic device 900 can receive a running request including container ID and package name information of the application to be run from the second electronic device 950 via the transmission manager.

[0162] According to various embodiments, during the generation and / or initialization of the application running container manager, the application running container manager can register callbacks such as on start activity intercepted, on task played / on task removed, on rotation changed, which are required when running an application or a task. First, to explain "on start activity intercepted", there can be a case where on start activity should be intercepted in order to run one application or task in one mirror session, or to reduce unnecessary application termination. For this, the application running container manager can request event subscription registration to a remote application mode service (e.g., Figure 6 remote application mode service). In the remote application mode service, the event subscription can be registered in an activity task manager (e.g., Figure 6The event is registered in the active task manager of the first electronic device 900 and the second electronic device 950, and when an event occurs, the event can be transmitted to the application running container manager. When a task is run or terminated, the event is transmitted to the application running container manager, and the event can be processed in each device. The "on task played / ontask removed" callback is information required for event processing in the first electronic device 900 and the second electronic device 950 when an application or task is run or terminated. To this end, in the remote application mode service, a task observer (for example, Figure 6 Task observer 637 of the , and can register event subscription. "on rotation changed" is information required for screen rotation and input coordinate correction in the first electronic device 900 and the second electronic device 950 when the virtual display for mirroring is rotated. To this end, the remote application mode service ( Figure 6 The remote application mode service 631) generates a rotating viewer (for example, Figure 6 The first electronic device 900 and the second electronic device 950 may be used to process the rotation event.

[0163] According to various embodiments, in operation 976, the application execution container manager may generate and initialize a virtual display and an encoder for performing mirroring. In this case, a flag may be added to distinguish between the virtual display to be generated and virtual displays used for other purposes. Here, the flag may indicate that virtual displays for multiple tasks are being sent to the second electronic device 950. The application execution container manager may generate a new encoder and obtain an input surface from the generated encoder. Thereafter, a request is made to generate a virtual display in the remote application mode service, and the encoder's input surface may be configured as the surface of the virtual display.

[0164] According to various embodiments, in operation 978 , the application execution container manager may generate a new application execution container and may store information such as a package name of an application to be executed, a virtual display, an encoder, and a surface.

[0165] According to various embodiments, in operation 980 , the application execution container manager may send a request to the encoder manager to execute the encoder generated in operation 976 .

[0166] According to various embodiments, in operation 982, the application running container manager can run the application running container generated in operation 978. The application running container can extract an application launch intent from the application package name included in the running request received from the second electronic device 950, and can store the ID of the screen to be run in the activity option of the intent. If the intent is run, the base activity of the application can be run on the virtual display.

[0167] According to various embodiments, in operation 984, if the application is run on the virtual display and the content of the application is drawn on the surface, the screen information of the application can be converted into a video format via a video encoder, and thus output in a byte buffer format.

[0168] According to various embodiments, in operation 986, the application running container can add the ID of the application running container to the metadata of the buffer data to be output, and can transmit it to the transmission manager, which can transmit the virtual display data to the second electronic device 950.

[0169] According to various embodiments, in operation 988, upon receiving the encoded video data, the transmission manager of the second electronic device 950 can transmit the received data to the remote application container manager. The remote application container manager can extract the ID from the metadata of the video data, and can determine the corresponding remote application container by using the ID. When the remote application container manager transmits the data to the identified remote application container, the remote application container can input the data to a decoder. When the information on the virtual display is decoded via the decoder, the screen information can be drawn on the surface view connected to the decoder, and can be displayed in a designated window on the display of the second electronic device 950.

[0170] Figure 10 is a flowchart of a method for processing an event and a user input by a first electronic device and a second electronic device according to various embodiments.

[0171] Figure 10 The illustrated method can be performed by the first electronic device 1000 (e.g., the first electronic device 400 of FIG. 4) and the second electronic device 1050 (e.g., the second electronic device 500 of FIG. 5) described above, and the illustrated operations can be performed after the virtual display data generated by the first electronic device described with reference to FIGS. 10A and 10B is displayed on the second electronic device. Figure 4 Figure 5 Figure 9 The illustrated method can be performed by the first electronic device 1000 (e.g., the first electronic device 400 of FIG. 4) and the second electronic device 1050 (e.g., the second electronic device 500 of FIG. 5) described above, and the illustrated operations can be performed after the virtual display data generated by the first electronic device described with reference to FIGS. 10A and 10B is displayed on the second electronic device.

[0172] Operations 1060 and 1062 and operations 1070 and 1072 are independent of each other and the order thereof can be changed, and only some of the operations can be performed.

[0173] ​​According to various embodiments, in operation 1060, the second electronic device 1050 can sense a user input event and transmit it to the first electronic device 1000. For example, when a user's mouse click, keyboard input, and / or touch / hover input is received on a virtual display displayed on the second electronic device 1050, the input data can be sensed via an input listener (e.g., the input listener 765 of FIG. 7) through the formed control channel. Figure 5

[0174] The second electronic device 1050 can add an ID of a remote application container corresponding to the virtual display to the input data, can modulate the input data into a transmittable form, and then transmit it to the first electronic device 1000 via a transmission manager.

[0175] According to various embodiments, in operation 1062, when the user input data is transmitted via the transmission manager, the first electronic device 1000 can determine an application execution container via an ID of the transmitted data. The application execution container manager (e.g., the application execution container manager 610 of FIG. 6) can restore the input data into a processable input, and then can input the user input to the matched application execution container via the input manager (e.g., the input manager 625 of FIG. 6). According to an embodiment, an ID of the virtual display of the second electronic device 1050 can be configured for the input event, and the ID of the virtual display can be used to identify a virtual display of the first electronic device 1000 to which the user input is to be applied. Figure 6 Figure 6

[0176] According to various embodiments, in operation 1070, the first electronic device 1000 can identify an event occurring on a system or an application of the first electronic device 1000. For example, the event can be an orientation change (e.g., a change from a portrait mode to a landscape mode) of the first electronic device 1000. When the event occurs, the first electronic device 1000 can transmit the event to the application execution container manager via a callback registered during generation and / or initialization of the application execution container manager. The application execution container manager can directly process the event, or can transmit the event to the second electronic device 1050. According to an embodiment, the application execution container manager can determine an application execution container that needs to process the event via a transmitted virtual display ID and task ID, thereby transmitting the event, and the event can be processed in the corresponding application execution container.

[0177] ​​​According to various embodiments, in operation 1072, the second electronic device 1050 can process the event received from the first electronic device 1000. When the first electronic device 1000 transmits the occurred event to the second electronic device 1050, the event data can be received via the transmission manager of the second electronic device 1050. When the event is received, the remote application container manager can directly process the event, or can determine a remote application container that processes the event based on the ID, and can transmit the event to the corresponding container. The remote application container that has received the event can process the received event. For example, when the orientation of the first electronic device 1000 is changed from a portrait mode to a landscape mode and thus a rotation event occurs, the remote application container manager can determine a remote application container that performs the rotation based on the ID, and can transmit the orientation information. The remote application container can rotate the screen according to the orientation information, and can also rotate the virtual touch input panel that receives a user input.

[0178] Figure 11 is a flowchart of a method performed by a first electronic device and a second electronic device when an application is terminated according to various embodiments.

[0179] Figure 11 The illustrated method can be performed by the first electronic device 1100 (e.g., the first electronic device 400) and the second electronic device 1150 (e.g., the second electronic device 500) described above, and the illustrated operations can be performed after the virtual display data generated by the first electronic device described with reference to FIGS. 1 to 10 is displayed on the second electronic device. Figure 4 Figure 5 The illustrated method can be performed by the first electronic device 1100 (e.g., the first electronic device 400) and the second electronic device 1150 (e.g., the second electronic device 500) described above, and the illustrated operations can be performed after the virtual display data generated by the first electronic device described with reference to FIGS. 1 to 10 is displayed on the second electronic device. Figure 9

[0180] According to various embodiments, in operation 1160, if the user terminates one of the windows in which the virtual display is displayed, the remote application container manager (e.g., the remote application container manager 770) of the second electronic device 1150 can terminate the remote application container corresponding to the virtual display. Figure 7

[0181] According to various embodiments, in operation 1162, the remote application container manager can transmit a request for terminating the corresponding application or task to the application running container manager (e.g., the application running container manager 610) of the first electronic device 1100 via the transmission manager (e.g., the transmission manager 760). Here, the termination request can include the ID of the corresponding container or virtual display. Figure 7 Figure 6

[0182] ​​​​​According to various embodiments, in operation 1164, the application execution container manager of the first electronic device 1100 may determine the corresponding application execution container via the ID included in the termination request and may record the termination reason. The application execution container manager may terminate the application or task configured in the determined application execution container.

[0183] According to various embodiments, in operation 1166, when a task is terminated, the active task manager (e.g., Figure 6 The active task manager 635) generates a callback indicating that the task has been terminated, and the application execution container manager can receive the callback to remove the terminated application execution container.

[0184] According to various embodiments, in operation 1168 , when the task is terminated and the container is removed, the application execution container manager may send a termination event and a termination reason to the remote application container.

[0185] According to various embodiments, in operation 1170 , the remote application container may terminate the remote application container mapped to the terminated application or task.

[0186] According to various embodiments, when the application or task is terminated, the second electronic device 1150 may release all related resources in operation 1172. The second electronic device 1150 may close the window of the terminated virtual display and may stop and release a decoder that decodes virtual display data.

[0187] Figure 12a and Figure 12b Illustrated are screens provided on a display of a second electronic device according to various embodiments.

[0188] Reference Figure 12a , a second electronic device (e.g., Figure 5 The second electronic device 500) can be connected to the first electronic device (eg, Figure 4 400 ), while displaying the first electronic device 400 , Figure 5 The virtual display 1210 sent from the first electronic device is displayed on the second display 510).

[0189] The second electronic device 1200 may receive application list information from the first electronic device, may display icons 1202 and 1204 corresponding to applications of the first electronic device on a display, and may send a request to the first electronic device to execute the application based on a user input to the icon.

[0190] Reference Figure 12bIf multiple virtual displays are received simultaneously from the first electronic device, the second electronic device 1200 may configure and display each of the virtual displays 1230, 1232, and 1234 as a different window. The corresponding displayed windows may be virtual displays generated according to the execution of different tasks.

[0191] If a user input is sensed in the corresponding window, the second electronic device 1200 may transmit the IDs of the virtual displays (or containers) 1230 , 1232 , and 1234 displayed on the corresponding window to the first electronic device and input event information.

[0192] Figure 13 is a flowchart of a method of processing an execution screen by a first electronic device when a new task is executed during a screen sharing service according to various embodiments.

[0193] Figure 13 The method shown can be performed by the first electronic device (eg, Figure 4 The first electronic device 400) is executed, and the operations shown can be referred to Figure 9 The description of the virtual display data generated by the first electronic device is then executed after being displayed on the second electronic device.

[0194] According to various embodiments, the first electronic device can drive only one application or task in a mirroring session, so that the second electronic device displays the virtual display transmitted via each mirroring session in an independent window. For example, in addition to running the application of the first electronic device based on user input, the application can also run another application. For example, if the user presses the URL of the chat window while using the messenger application, the Internet browser application can be run. When running a new application or task as described above, various embodiments can provide a virtual display of the new application or task via a separate container.

[0195] According to various embodiments, an intent to run a new application may be generated based on an event occurring in an application running in an application execution container in operation 1310. For example, if a messenger application is running on a first device display and a user selects an Internet URL in the messenger application, an ACTION_VIEW intent may be generated along with the URL.

[0196] According to various embodiments, in operation 1315, an active task manager (e.g., Figure 6 The activity task manager 635 of the application can determine the activity corresponding to the intent. The activity task manager can run the appropriate activity from the intent based on the user configuration or application installation. For example, the intent of running the browser activity of the Internet browser application can be run based on the ACTION_VIEW intent.

[0197] According to various embodiments, in operation 1320, the active task manager can identify a virtual display in which the intent has been generated, and can determine whether a flag is attached to a container of the virtual display. Here, the flag can indicate that the virtual display of the plurality of tasks is transmitted to the second electronic device. For example, the active task manager can determine whether the flag has been configured for the first virtual display in which the intent has been generated.

[0198] According to various embodiments, in operation 1325, the active task manager can determine whether the virtual display in which the intent has been generated is configured for multi-screen mirroring according to the flag.

[0199] According to various embodiments, if the multi-screen mirroring is not configured, in operation 1330, the corresponding application can be run on the same virtual display. For example, the Internet browser can operate in the foreground of the first virtual display.

[0200] According to various embodiments, if the multi-screen mirroring is not configured, in operation 1335, it can be determined whether there is an application or task that has been run on the virtual display in which the intent has been generated. For example, if the messenger application has been run on the first virtual display, the on start activity intercepted callback can be generated with the APP_TO_APP_LAUNCHED flag.

[0201] According to various embodiments, if there is an application or task that has been run on the virtual display, in operation 1340, the active task manager can call the on start activity intercepted callback.

[0202] According to various embodiments, in operation 1345, the application running container manager can receive the callback, and can generate a new application running container. The new application running container can include the second virtual display.

[0203] According to various embodiments, in operation 1350, the application running container manager can configure the intercepted intent for the new application running container.

[0204] According to various embodiments, in operation 1355, the application execution container manager can execute the generated application execution container. The application execution container manager can execute the intent after configuring the second virtual display corresponding to the application execution container as the preferred display of the intent. In this case, while the application is being executed in the application execution container that is being deferred from the previously executed application execution container, the internet browser application can be executed on the second virtual display independently of the messenger application being executed on the first virtual display. Accordingly, the messenger application can be mirrored to the second electronic device via the first virtual display, and the internet browser application can be mirrored to the second electronic device via the second virtual display.

[0205] Figure 14a and Figure 14b FIGS. 1A and 1B illustrate screens provided by a first electronic device and a second electronic device when a new task is executed according to various embodiments.

[0206] According to an embodiment, while a task is being executed on a display (e.g., the first display 410 of the first electronic device 100) of the first electronic device, an execution intent of the task can be sensed according to a user input or an event of the second electronic device. Figure 4

[0207] Referring to Figure 14a , in a state in which the calendar application 1410 is displayed on the display of the first electronic device 1400 and the virtual display 1460 of the messenger application is provided to the second electronic device 1450, a request to execute the calendar application can be received by the first electronic device 1400 according to a user input for the application icon or a request of the messenger application.

[0208] In this case, as Figure 14b indicated, the first electronic device 1400 can display a dialog 1420 for determining whether to move the execution location of the application from the display of the first electronic device to the display of the second electronic device 1450, which can be displayed on the display of the first electronic device 1400.

[0209] If the user allows the movement of the application in the dialog, the first electronic device can terminate the application being executed on the main display and can execute the calendar application on the new virtual display.

[0210] Figure 15 is a flowchart of a method for processing a task on a new virtual display by a first electronic device according to various embodiments.

[0211] According to various embodiments, in operation 1510, the first electronic device can generate a new application execution container according to a task execution request received from the second electronic device.

[0212] ​According to various embodiments, in operation 1515, the first electronic device can run the application or the intercepted intent in the generated container.

[0213] According to various embodiments, the process when the requested application or task is already running in another display can be implemented via an on start activity intercepted callback. For example, if a task of the same application is already running, the active task manager can sense the running, thereby providing a running notification with a specified flag to the application running container manager. The application running container manager can determine a matching application running container, thereby transmitting it to the second electronic device. The second electronic device can receive its event and bring the matching remote application container to the foreground.

[0214] Table 2 shows conditions checked by the active task manager, generated flags, and operations of the application running container when the flag is received.

[0215] [Table 2]

[0216]

[0217]

[0218] According to various embodiments, in operation 1520, the first electronic device can identify an on start activity intercepted condition. For example, the application running container manager can register an on start activity intercepted callback during initialization, which can indicate that on start activity should be intercepted under a specified condition. According to various embodiments, if the on start activity intercepted condition is satisfied, in operation 1525, the application running container manager can call the on start activity intercepted callback.

[0219] According to various embodiments, in operation 1530, the first electronic device can identify a location of a running application with respect to an intercepted intent.

[0220] According to various embodiments, when the application is running on a virtual display, in operation 1535, the first electronic device can cause the second electronic device to bring the running application to the foreground, and in operation 1540, can terminate the generated application running container. For example, the application running container manager of the first electronic device can determine an application running container matching the running application, thereby transmitting it to the second electronic device. The second electronic device can receive its event and bring the matching remote application container to the foreground.

[0221] According to various embodiments, if the application is running on the main display of the first electronic device, the first electronic device can display a dialog for determining whether to move the running location of the application to the display of the second electronic device on the main display in operation 1545.

[0222] According to various embodiments, if the user allows the movement of the application on the dialog in operation 1550, the application running on the main display of the first electronic device can be terminated in operation 1550, and the intercepted intent can be run on the new virtual display in operation 1560.

[0223] According to an embodiment, when the user runs an application or a task, or when an intent is generated from the application so that a new application or task is run, there can be a case where the corresponding application is already running on another display (e.g., the main display or the virtual display of the first electronic device). In this case, depending on the display on which the application is running, the embodiments can be executed as follows. First, when the application is running on the main display of the first electronic device, if the application is moved to the virtual display, the user data can be lost so that the application can be moved after the user determines whether to move the application to the virtual display. Alternatively, when the application is running in another virtual display, the container generated according to the intent can be terminated, and the previously running container can be identified so as to be brought to the foreground.

[0224] According to various embodiments, if the on start activity intercepted condition is not satisfied as a result of the determination in operation 1520, the first electronic device can run the application of the corresponding intent in operation 1560.

[0225] According to various embodiments, the first electronic device can call the on task played callback in operation 1570.

[0226] According to various embodiments, the first electronic device can identify the number of running containers in operation 1575.

[0227] According to various embodiments, if the number of running containers is greater than a reference number, the first electronic device can extract a container to be terminated from the running containers in operation 1580, and can terminate the selected container in operation 1585.

[0228] According to various embodiments, the first electronic device can determine a maximum number of virtual displays (or containers) to be provided to the second electronic device. The maximum number can be pre-allocated according to the processor and the memory, or can be determined by the first electronic device based on at least one of the available resources of the processor and / or the memory, the temperature of the first electronic device, and the transmission rate of data transmitted to an external device via a communication interface.

[0229] For example, if the resources of the electronic device are limited, it is necessary to calculate and limit the number of virtual displays that can be mirrored simultaneously. According to an embodiment, the first electronic device can obtain the maximum number of virtual displays that can be mirrored simultaneously as follows.

[0230] N = Area of total encoder surface / size of maximum mirroring resolution

[0231] For example, if the area of the total encoder surface is 23,760,000 and the maximum mirroring resolution is FHD+ (2200*1080) which is the same resolution as the first electronic device, the maximum number of virtual displays that can be used for mirroring simultaneously can be 23,760,000 / (2200*1080), i.e., 10.

[0232] According to another embodiment, if camera recording encoding is supported during mirroring, a portion of the encoder surface is used so that the first electronic device can obtain the maximum number of virtual displays that can be mirrored simultaneously as follows.

[0233] N = (Area of total encoder surface - Chloe shooting resolution) / size of maximum mirroring resolution

[0234] For example, if the camera application is recording a video at a maximum shooting resolution of 3840*2160, the calculation is performed so that N = 6.50, and mirroring of up to 6 virtual displays can be performed simultaneously.

[0235] According to an embodiment, the maximum number of virtual displays that can be used for mirroring simultaneously can be determined as a number predefined for each chip set (e.g., processor and memory of the first electronic device). For example, the first electronic device can control virtual displays having a fixed maximum number according to the type of chip set included in the first electronic device.

[0236] According to an embodiment, the first electronic device can determine the maximum number of virtual displays that can be mirrored simultaneously according to the overheating state of the first electronic device. For example, if the temperature is lower than 45 degrees, the first electronic device can determine the maximum number as 5, if the temperature is between 45 degrees and 60 degrees, the maximum number can be determined as 1, and if the temperature is higher than 60 degrees, the first electronic device can not provide virtual displays to the second electronic device.

[0237] According to an embodiment, the first electronic device may determine the maximum number of virtual displays that can be simultaneously mirrored based on the average transmission rate. For example, if the average transmission rate is 5 MB / s or higher, the first electronic device may determine the maximum number to be 5; if the average transmission rate is 3 MB / s to 5 MB / s, the maximum number may be determined to be 3; if the average transmission rate is 1 MB / s to 3 MB / s, the maximum number may be determined to be 1.

[0238] According to various embodiments, if more mirroring sessions are generated than the maximum number of N mirroring sessions (or virtual displays or containers), the first electronic device may terminate the sessions that exceed the number. According to an embodiment, the first electronic device may terminate the mirroring session with the longest time since the user started using it. The application running container may store the timestamps of the generation time, activation time, and deactivation time of the container, and if the user selects a window on the second electronic device, the remaining containers except the container corresponding to the virtual display may be deactivated. The application running container manager may terminate the oldest deactivated mirroring session based on the timestamp of each application running container.

[0239] According to an embodiment, after generating application execution containers, if the number of running application execution containers exceeds the maximum number, the first electronic device terminates the oldest application execution container. In this case, the application execution containers may be inadvertently terminated. For example, when N mirroring sessions are running and container 1 is the oldest container, if a user input is received on the second electronic device to run a new application, container N+1 may be generated, and container 1, as the oldest container, may be terminated. If an application or task being run by the user is already running on the primary or virtual display of the first electronic device and the user does not want to move the application or task to another display, container N+1 may be terminated without the application or task running. In this case, due to the termination of container 1, only N-1 containers can be run. This situation may be due to the fact that when a user runs an application, it is impossible to know exactly which application or task will be executed. For example, if an email application can drive both an inbox task and a composer task, when the user runs the email application, an intent to run the email application's main launcher activity may be obtained. However, when the intent is initiated, the task to be generated is unknown, and the task may be identified after the intent is run. To address this situation, according to various embodiments, when the number of concurrently running application execution containers exceeds the maximum number, the routine that terminates an application execution container may not be executed when a new execution container is generated. Instead, the routine may be executed while a task is actually running in the application execution container. To this end, upon receiving an event indicating the execution of a new task, the task observer of the remote application mode service may terminate the number of tasks that exceeds the maximum number.

[0240] Figure 16 FIG. 9 is a flowchart of a method for processing a task by a first electronic device when the task is terminated immediately after being executed according to various embodiments.

[0241] According to an embodiment, when a virtual display is mirrored in a task unit, unexpected execution of some applications can occur. For example, when some applications are executed, a first task is generated and executed, and the task can be terminated immediately after causing execution of a second task. In this case, there can be a case where the virtual display of the first task that is terminated immediately should be terminated immediately after its generation. To prevent this, the first electronic device can perform an implementation to configure an execution mode of a task for an application execution container, and sense the configured mode when the task is terminated, thereby causing the virtual display to be terminated on the second electronic device.

[0242] According to various embodiments, in operation 1610, the first electronic device can sense termination of a task executed on a virtual display.

[0243] According to various embodiments, in operation 1615, an on task removed callback can be generated in a task observer (e.g., a task observer of FIG. 8). Figure 6

[0244] According to various embodiments, in operation 1620, when the on task removed callback is received, the application execution container manager can determine an application execution container via a task ID.

[0245] According to various embodiments, in operation 1625, the application execution container can check whether a new task has been launched and an execution time. For example, the application execution container manager can check whether a second task is executed immediately after a first task is executed on a virtual display, and a time at which the first task is terminated after its execution. When launch of the second task is sensed in the corresponding application execution container, the application execution container manager can configure a flag indicating that a task has been launched to be true. Then, if the task is terminated immediately, the task launch flag and the execution time of the task can be checked.

[0246] According to various embodiments, in operation 1630, the application execution container manager can determine whether an execution time of a first task executed and then terminated in the corresponding application execution container is less than a reference time (e.g., 2 seconds).

[0247] According to various embodiments, if the execution time of the first task executed and then terminated is less than the reference time, in operation 1635, a reason for termination can be recorded in the application execution container as termination after launching a new task.

[0248] ​According to various embodiments, in operation 1640 , the application execution container manager may remove the application execution container.

[0249] According to various embodiments, if the running time of the first task that is terminated after running is greater than a reference time, the reason for the termination may be recorded as application termination in the application running container.

[0250] The second electronic device may receive the termination reason value of the task and may process the virtual display in which the task is executed.

[0251] According to various embodiments, an electronic device may include a display, a communication interface, a memory, and a processor operatively connected to the display, the communication interface, and the memory, wherein the processor is configured to: determine a connection with an external device via the communication interface; receive a request to run a first task from the external device; run the first task based on an application stored in the memory in response to the request to run the first task, generate a first virtual display including a running screen of the first task, and send the first virtual display to the external device; during the running of the first task, receive a request to run a second task from the external device; and run the second task based on the application stored in the memory in response to the request to run the second task, generate a second virtual display including a running screen of the second task, and send the second virtual display to the external device.

[0252] According to various embodiments, the processor can be configured to: configure the first encoder, the first input surface and the first virtual display as a first container, the first encoder is configured to compress the first image information generated in the first task, and the first image information is recorded on the first input surface; and configure the second encoder, the second input surface and the second virtual display as a second container, the second encoder is configured to compress the second image information generated in the second task, and the second image information is recorded on the second input surface.

[0253] According to various embodiments, the processor may be configured to: generate identification information of the first virtual display and send it together with the first virtual display to the external device; and generate identification information of the second virtual display and send it together with the second virtual display to the external device.

[0254] According to various embodiments, the processor may be configured to, when receiving an input event of the virtual display from the external device, process the input event in the first task or the second task by using identification information included in the input event.

[0255] According to various embodiments, the processor may be configured to, if an intention to execute the third task on the first virtual display is received from the external device, determine a virtual display corresponding to the third task.

[0256] According to various embodiments, the processor can be configured to, if the virtual display corresponding to the third task is not generated, generate a third virtual display including a running screen of the third task, and transmit the third virtual display to the external device.

[0257] According to various embodiments, the processor can be configured to, if the virtual display corresponding to the third task is generated, cause the third task to operate in the foreground.

[0258] According to various embodiments, the processor can be configured to, when the running screen of the third task is being displayed on the display, stop the display of the running screen of the third task on the display, generate a third virtual display including the running screen of the third task, and transmit the third virtual display to the external device, or transmit the running screen of the third task being displayed on the display to the external device.

[0259] According to various embodiments, the processor can be configured to, if a request to run the third task is received from the external device, identify a number of generated virtual displays including the first virtual display and the second virtual display, and if the number of virtual displays exceeds a maximum number, terminate at least one of the generated virtual displays.

[0260] According to various embodiments, the maximum number of virtual displays can be configured to be pre-allocated according to the processor and the memory, or determined by the processor based on at least one of available resources of the processor and the memory, a temperature of the electronic device, and a transmission rate of data transmitted to the external device via the communication interface.

[0261] According to various embodiments, the processor can be configured to, if the number of virtual displays exceeds the maximum number, terminate a virtual display corresponding to a task having a longest deactivation time among tasks constituting the virtual display.

[0262] According to various embodiments, the processor can be configured to, if at least one of the virtual displays is terminated, transmit termination cause information of the task to the external device.

[0263] According to various embodiments, the communication interface can be configured to establish a wireless connection with the external device via Wi-Fi Direct.

[0264] The method of providing a screen on a display of an external device by an electronic device according to various embodiments can include an operation of determining a connection with the external device, an operation of receiving a request to run a first task from the external device, an operation of running the first task based on an application stored in a memory in response to the request to run the first task, generating a first virtual display including a running screen of the first task, and transmitting the first virtual display to the external device, an operation of receiving a request to run a second task from the external device during the running of the first task, and an operation of running the second task based on an application stored in the memory in response to the request to run the second task, generating a second virtual display including a running screen of the second task, and transmitting the second virtual display to the external device.

[0265] According to various embodiments, the generating of the first virtual display can include configuring the first encoder, the first input surface, and the first virtual display as a first container, the first encoder being configured to compress first image information generated in the first task, the first image information being recorded on the first input surface.

[0266] According to various embodiments, the transmitting of the first virtual display to the external device can include generating identification information of the first virtual display and transmitting the same to the external device along with the first virtual display, the transmitting of the second virtual display to the external device can include generating identification information of the second virtual display and transmitting the same to the external device along with the second virtual display.

[0267] According to various embodiments, the method further includes, when an input event of a virtual display is received from the external device, processing the input event in the first task or the second task by using identification information included in the input event.

[0268] According to various embodiments, the method further includes, if a request to run a third task is received from the external device, identifying a number of generated virtual displays including the first virtual display and the second virtual display, and if the number of virtual displays exceeds a maximum number, terminating at least one of the generated virtual displays.

[0269] An electronic device according to various embodiments can include a display, a communication interface, and a processor operatively connected to the display and the communication interface, wherein the processor is configured to: determine a connection with an external device via the communication interface; transmit, in response to a first user input to execute a first task executable in the external device, a request to execute the first task to the external device; receive, from the external device, a first virtual display including an execution screen of the first task; transmit, in response to a second user input to execute a second task executable in the external device, a request to execute the second task to the external device; receive, from the external device, a second virtual display including an execution screen of the second task; and display the first virtual display and the second virtual display on different windows on the display.

[0270] According to various embodiments, the processor can be configured to, if a third user input is sensed on a window of the first virtual display, transmit an event corresponding to the third user input to the external device with identification information of the first virtual display.

Claims

1. An electronic device comprising: monitor; a communication interface, including a communication circuit; Memory; as well as a processor operatively connected to the display, the communication interface, and the memory, The processor configuration is: establishing a connection to an external device via a communication interface, receiving a request to execute a first task from the external device via a communication interface, In response to a request to execute a first task, executing the first task based on an application stored in a memory, generating a first container corresponding to a first virtual display including a first execution screen of the first task, and transmitting the first container to the external device via a first communication channel of the communication interface to form a first mirroring session corresponding to the first container on the external device. During the execution of the first task, receiving a request to execute the second task from the external device, In response to a request to run a second task, the second task is run based on an application stored in the memory, a second container corresponding to a second virtual display including a second execution screen of the second task is generated, and the second container is sent to the external device via a second communication channel of the communication interface that is independent of the first communication channel to form a second mirroring session on the external device that corresponds to the second container and is independent of the first mirroring session.

2. The electronic device according to claim 1, wherein the processor is configured to: configuring a first encoder, a first input surface, and a first virtual display as a first container, wherein the first encoder is configured to compress first image information associated with a first execution screen generated in a first task, and the first image information is recorded on the first input surface; and The second encoder, the second input surface, and the second virtual display are configured as a second container, wherein the second encoder is configured to compress second image information related to the second execution screen generated in the second task, and the second image information is recorded on the second input surface.

3. The electronic device according to claim 1 , wherein the processor is configured to: generating identification information of a first virtual display, and sending the identification information of the first virtual display and the first container to the external device, and Identification information of the second virtual display is generated, and the identification information of the second virtual display and the second container are sent to the external device.

4. The electronic device according to claim 3, wherein the processor is configured to: Based on receiving an input event for the virtual display from the external device, the input event is processed in the first task or the second task using identification information included in the input event.

5. The electronic device according to claim 1 , wherein the processor is configured to: Based on receiving an intention to execute a third task on the first virtual display from the external device, a third virtual display corresponding to the third task is confirmed.

6. The electronic device according to claim 5, wherein the processor is configured to: Based on the fact that the third virtual display corresponding to the third task is not generated, a third container corresponding to the third virtual display including the third execution screen of the third task is generated, and the third container is transmitted to the external device.

7. The electronic device according to claim 5, wherein the processor is configured to: based on the third execution screen of the third task being displayed on the display, stopping displaying the third execution screen of the third task on the display, generating a third container corresponding to a third virtual display including the third execution screen of the third task, and transmitting the third container to the external device, or The third execution screen of the third task being displayed on the display is transmitted to the external device.

8. The electronic device according to claim 1, wherein the processor is configured to: identifying a number of generated virtual displays including a first virtual display and a second virtual display based on receiving a request to execute a third task from the external device, and Based on the number of virtual displays exceeding a maximum number, at least one of the generated virtual displays is terminated.

9. The electronic device according to claim 8, The maximum number of virtual displays is pre-allocated according to the processor and memory, or is determined by the processor based on at least one of available resources of the processor and memory, temperature of the electronic device, and a transmission rate of data sent to an external device via a communication interface.

10. The electronic device according to claim 9, wherein the processor is configured to: Based on the number of virtual displays exceeding a maximum number, a virtual display corresponding to a task having a longest deactivation time among tasks of the virtual displays is terminated, and termination reason information of the task is transmitted to the external device.

11. A method for providing a screen on a display of an external device by an electronic device, the method comprising: establishing a connection to the external device; receiving a request to execute a first task from the external device; In response to a request to execute a first task, executing the first task based on an application stored in a memory, generating a first container corresponding to a first virtual display including a first execution screen of the first task, and transmitting the first container to the external device via a first communication channel to form a first mirroring session corresponding to the first container on the external device; During the execution of the first task, receiving a request to execute the second task from the external device; as well as In response to a request to execute a second task, the second task is executed based on an application stored in the memory, a second container corresponding to a second virtual display including a second execution screen of the second task is generated, and the second container is transmitted to the external device via a second communication channel independent of the first communication channel to form a second mirroring session on the external device corresponding to the second container and independent of the first mirroring session.

12. The method according to claim 11, wherein generating the first container includes configuring a first encoder, a first input surface, and a first virtual display as a first container, wherein the first encoder is configured to compress first image information related to the first running screen generated in the first task, and the first image information is recorded on the first input surface.

13. The method according to claim 12, wherein sending the first container to the external device comprises generating identification information of the first virtual display and sending the identification information of the first virtual display and the first container to the external device, and Sending the second container to the external device includes generating identification information of the second virtual display and sending the identification information of the second virtual display and the second container to the external device.

14. The method according to claim 13, further comprising: Based on receiving an input event for the virtual display from the external device, the input event is processed in the first task or the second task using identification information included in the input event.

15. The method according to claim 14, further comprising: identifying a number of generated virtual displays including a first virtual display and a second virtual display based on receiving a request to execute a third task from the external device, and Based on the number of virtual displays exceeding a maximum number, at least one of the generated virtual displays is terminated.

Images