Screen expansion device in electronic device and operation method thereof

Abstract

The present invention relates to an electronic device for displaying a mirror screen by interacting with an external electronic device and an operation method thereof. An external device may obtain mirroring data and display a mirroring image on a first display area allocated in a display using the obtained mirroring data. The external device may obtain extended mirroring data from previous mirroring data that has been used to display the mirroring image, and in response to a user gesture, display an extended mirroring image on at least one extended display area other than the first display area allocated in the display using the extended mirroring data.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a screen expansion device and a method for operating the screen expansion device in an electronic device. Background Art

[0002] A digital device may install various applications (hereinafter referred to as "apps") that provide specific functions or services. A digital device may execute an installed app in response to a user's request. A digital device may display information through a display according to the execution of an app program. A digital device may include an electronic device that is easily carried by a user, such as a smart phone, a mobile phone, or a tablet device.

[0003] Efforts are being made to reduce the size of portable digital devices so that they are easy to hold by users or for aesthetic reasons. Therefore, the digital devices may have a relatively small screen size compared to a display device such as a television or a monitor. When multimedia content such as providing a video in a movie, a television program, or a game is used on a digital device having a relatively small screen size compared to the display device, user satisfaction may be reduced. Summary of the invention

[0004] Technical Solution Embodiments of the present disclosure may provide an electronic device for extending a mirrored screen through interaction with a user in a mirroring system for mirroring a screen of a digital device, and an operating method thereof.

[0005] According to an embodiment, an electronic device may include: a display module configured to display a mirror image on a display window; a communication module configured to perform communication with an external device; a sensor module configured to detect a user gesture and generate a sensing signal corresponding to the detected user gesture; a memory configured to allocate one or more buffer areas; and at least one processor electrically connected to the display module, the communication module, the sensor module or the memory, and configured to control the operation of the display module, the communication module, the sensor module or the memory. At least one processor may receive mirror data from the external device through the above-mentioned communication module, control the display module to display the mirror image in a first display area allocated in the display window using the mirror data, and control the display module to display an extended mirror image in at least one extended display area allocated in the display window using the mirror data 31 according to the user gesture detected by the sensor module through interaction with the user.

[0006] According to an embodiment, a method for displaying in an electronic device may include displaying a mirror image in a first display area allocated on a display based on a mirroring function. The display method may include temporarily storing mirroring data corresponding to the mirror image. The display method may include: displaying an extended mirror image in at least one extended display area allocated on the display using the mirroring data according to a user gesture obtained through interaction with a user.

[0007] According to an embodiment of the present disclosure, since an extended mirroring screen is provided by reusing a mirroring image provided as a mirroring screen through interaction with a user on an external device providing the mirroring screen, traffic or power consumption of a radio channel may be reduced.

[0008] The technical objectives of the present disclosure are not limited to the foregoing, and a person of ordinary skill in the art may derive other technical objectives from the exemplary embodiments of the present disclosure.

[0009] The effects of the present invention are not limited to the foregoing, and other unmentioned effects will be apparent to those skilled in the art from the following description. In other words, those skilled in the art may also derive unexpected effects in practicing the embodiments of the present invention from the exemplary embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram conceptually illustrating a mirroring function performed in a mirroring system according to an embodiment.

[0011] Figure 2 is a diagram illustrating a process of a mirroring function in a mirroring system according to an embodiment.

[0012] Figure 3 is a block diagram illustrating a mirroring system for supporting screen mirroring according to an embodiment.

[0013] Figure 4 is a diagram illustrating a functional configuration of a source device supporting screen mirroring according to an embodiment.

[0014] Figure 5 is a diagram illustrating a functional configuration of an external device supporting screen mirroring according to an embodiment.

[0015] Figure 6 is an exemplary diagram illustrating a screen display of a virtual space in an external device according to an embodiment.

[0016] Figure 7 is an exemplary diagram illustrating extending a mirroring screen in an external device of a mirroring system according to an embodiment.

[0017] Figure 8is a diagram illustrating a process for providing a mirroring function in a mirroring system according to an embodiment.

[0018] Fig. 9 is a diagram illustrating a processing procedure for mirroring and event handling operations in a mirroring system according to an embodiment.

[0019] Fig.10a and Fig.10b is a control flow chart illustrating displaying a mirroring image by an external device in a mirroring system according to an embodiment.

[0020] Fig.11a is an exemplary diagram illustrating extending a mirroring screen in an external device supporting a mirroring function according to an embodiment.

[0021] Fig.11b is an exemplary diagram illustrating extending a mirroring screen in an external device supporting a mirroring function according to an embodiment.

[0022] Fig.12 is an exemplary diagram illustrating extending a mirroring screen using a 3D mirroring image in a mirroring system according to an embodiment.

[0023] Fig.13 is an exemplary diagram illustrating that an interaction is provided in an extended display area by an external device constituting a mirroring system according to an embodiment.

[0024] Fig.14 is a history hierarchy tree showing management of an extended display area in a mirroring system according to an embodiment.

[0025] Fig.15 is an exemplary diagram illustrating a self-operation extension of a display area in a mirroring system according to an embodiment.

[0026] Fig.16 2 is a diagram illustrating an example in which an external device expands a mirror image in a time space according to an embodiment.

[0027] Fig.17 is a control flow chart for expanding a mirror image in a time space by an external device according to an embodiment.

[0028] Fig.18 is a block diagram illustrating electronic devices in a network environment according to various embodiments.

[0029] In conjunction with the description of the drawings, the same or similar reference numerals may be used to designate the same or similar elements. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure belongs can easily practice the present disclosure. However, the present disclosure can be implemented in various other forms and is not limited to the embodiments set forth herein. Throughout the specification and the accompanying drawings, the same or similar reference numerals may be used to refer to the same or similar elements. In addition, for clarity and brevity, well-known functions and configurations are not described in the accompanying drawings and related descriptions.

[0031] Figure 1 is a diagram conceptually illustrating a mirroring function performed in the mirroring system 1 according to the embodiment.

[0032] Reference Figure 1 , the mirroring system 1 may include at least two electronic devices. The at least two electronic devices may include a source device 10 as a first electronic device and an external device 20 as at least one second electronic device.

[0033] The source device 10 may generate a mirror image according to the mirroring function and provide the mirror image to the external device 20. The source device 10 may generate a virtual display in response to an app execution request from the external device 20, and execute the target app on the virtual display to obtain an image to be mirrored to the external device 20 (hereinafter referred to as a "mirror image"). The external device 20 may include the mirror image in, for example, mirroring data and send it to the external device 20. The mirroring data may include data for mirroring the mirror image (hereinafter referred to as "mirror metadata"). The mirroring metadata may include, for example, information such as the size, resolution, DPI, or display position of the mirror image. The source device 10 may be, for example, a digital device such as a smart phone or a tablet PC, in which a program such as an app may be installed or executed.

[0034] The external device 20 may be a screen expansion device that expands an area for displaying a mirror image provided by the source device 10 (hereinafter referred to as a "mirror display area") in order to implement a mirroring function. The external device 20 may be, for example, a digital device such as a wearable device, a smart phone, a tablet PC, or a personal computer having its own display. The wearable device may be a digital electronic device that can be attached to / detached from the user's body or clothing. The wearable device may have various shapes, such as, for example, a T-shirt, glasses, a bracelet, a watch, and a shoe. For example, the wearable device may include a smart watch that a user can wear on her wrist, a Bluetooth headset that a user can wear on her ear, or a head-mounted display (HMD) such as smart glasses that a user can wear like glasses. Smart glasses, which are one of the external devices 20, may display a desired image in a space processed based on an extended reality technology. The space processed based on the extended reality technology (e.g., a mirror display area) may be, for example, a portion or the entire display area for displaying a virtual image (e.g., a mirror image) in a display area (or display window) of the external device 20 that can display a real image. The extended reality technology may be a technology that supports at least one of virtual reality (VR), augmented reality (AR), and mixed reality (MR). The mirroring screen may extend the screen of the app executed by the source device 10 in a virtual display. For example, depending on the application technology, the screen extension in the mirroring function may include a VR extended screen, an AR extended screen, or an MR extended screen.

[0035] The source device 10 and the external device 20 may be connected for the mirroring function based on a protocol supporting a predetermined communication scheme (hereinafter referred to as a "communication protocol"). The connection for the mirroring function may be based on the mirroring function (eg, Figure 2 The communication protocol may be a protocol provided for use in a network environment (such as a short-range communication network (e.g., Fig.18 1898) or a long-distance communication network (e.g. Fig.18 The protocols provided for the short-range communication network 1898 may include, for example, Bluetooth, Wi-Fi Direct, or IrDA. The long-range communication network 1899 may include, for example, a traditional cellular network, a 5G network, a next-generation telecommunication network, the Internet, or a computer network (e.g., a remote communication network such as a LAN or WAN).

[0036] The source device 10 and the external device 20 may enter a state in which a connection process is performed to form a state in which data can be transmitted to the other device (for example, Figure 2The mirroring function request of the external device 20 may be performed, for example, after the radio channel is formed with the source device 10 through the connection process.

[0037] The source device 10 may transmit or receive mirroring related data for the mirroring function 30 to or from the external device 20 through the radio channel allocated by the connection process. The source device 10 may transmit, for example, mirroring data for the mirroring function (e.g., Figure 2 The mirroring data 31 may transmit the image data of the screen itself for display on the virtual display of the source device 10 as it is, for example, instead of transmitting it as a separate layout, code or function unit.

[0038] The external device 20 may send a signal to the source device 10, for example, through interaction with a user (eg, Figure 2 The event information obtained by the interaction 25) (for example, Figure 2 The user can interact 25 to control the event information 33 in the extended space (for example, Figure 2 The interaction 25 may be performed by, for example, tracking the movement of the user's hand (hand tracking) or tracking the user's gaze (eye tracking).

[0039] The external device 20 may use the mirroring data provided from the source device 10 to display the image on a display (eg, Figure 3 The mirror image is displayed on a display module 210) in a display area provided for the mirror function (eg, Figure 2 The mirroring screen in the external device 20 may be a screen that displays a mirror image on all or part of the mirroring display area 23.

[0040] According to an example, the external device 20 may allocate the mirror display area 23 to include the first display area (eg, Figure 6 620) and / or at least one extended display area (eg, Figure 6 The external device 20 may allocate the mirror display area 23 to include, for example, the first display area (eg, Figure 6 The external device 20 may allocate the mirror display area 23 to include at least one extended display area (eg, Figure 6 The external device 20 may allocate the mirror display area 23 to include, for example, the first display area (eg, Figure 6 The first display area 620) and at least one extended display area (eg, Figure 6 of the extended display area 640).

[0041] According to an example, the external device 20 may provide a buffer (or buffer area) for temporarily storing or rendering the mirror image. In addition to the first display area (eg, Figure 6 In addition to the first display area 620), the external device 20 can also implement an extended display area (for example, Figure 6 The external device 20 may have a first buffer (or a first buffer area (e.g., Figure 3 The first buffer 241) is used to temporarily store or render the content to be displayed in, for example, the first display area (eg, Figure 6 The mirror image data (eg, Figure 7 The external device 20 may have a second buffer (or a second buffer area (eg, Figure 3 A second buffer 243)) is used to temporarily store or render the content to be displayed in, for example, at least one extended display area (eg, Figure 6 The extended mirror data (eg, Figure 7 The external device 20 may analyze the first to third extended mirror data 731, 733, and 735 through, for example, interaction with the user (eg, Figure 2 The information obtained by the interaction 25 (for example, the motion information according to the user's gesture) and the analysis result is obtained by using the information temporarily stored in the second buffer (or the second buffer area (for example, Figure 3 The extended mirror data in the second buffer 243)) (eg, Figure 7 The first to third extended mirror data 731, 733 and 735) are displayed in at least one extended display area (eg, Figure 6 The extended mirror image is displayed in the extended display area 640).

[0042] If a request to terminate the mirroring function occurs, the source device 10 and / or the external device 20 may terminate the mirroring function after releasing all resources allocated for the mirroring function. The source device 10 may discard, for example, at least one virtual display generated for the mirroring function. The external device 20 may release, for example, a display area allocated for the mirroring function (e.g., Figure 2 The external device 20 may, for example, release allocation of a buffer (eg, the first buffer 21 or the second buffer 23) or a buffer area (eg, the first buffer area or the second buffer area) used for the mirroring function.

[0043] Figure 2 is a diagram showing a mirroring system (eg, Figure 1 1) is a diagram of the process of the mirroring function in the mirroring system.

[0044] Reference Figure 2 , the mirroring system 1 may include a source device 10 and at least one external device 20. The source device 10 may provide one or more virtual displays to provide a mirror image to the external device 20 for a mirroring function. The virtual display may correspond to, for example, an area on a memory allocated to generate image data for displaying a mirroring screen.

[0045] The source device 10 may share its app list with the external device 20. The app list may be, for example, a list of apps installed in the source device 10. The source device 10 may generate an initial mirror image including an icon of an app for providing a mirroring function on a virtual display (e.g., a first virtual display 11 and a second virtual display 13), and provide the external device 20 with initial mirror data 31 corresponding to the initial mirror image. The app list may include, for example, a two-dimensional app (2D app), a three-dimensional widget app (3D widget app), or a three-dimensional (3D) immersive app. A 2D app may be, for example, an app that provides a flat image. A 3D widget app may be, for example, a 3D app that can be arranged in large quantities in a processing space (AR space, VR space, and extended reality space). A 3D immersive app may be, for example, a 3D app that can control the entire processing space (AR space, VR space, and extended reality space).

[0046] The external device 20 may obtain information about an app for using a mirroring function from an app list shared by the source device 10. The external device 20 may receive initial mirroring data 31 from the source device 10. The external device 20 may display an initial mirroring screen in the entire or partial area of ​​the mirroring display area 23 provided in the display module 110 using the information about the app and / or the initial mirroring data 31 to use the mirroring function.

[0047] The source device 10 may start the mirroring function in response to an application execution request from the external device 20. If the mirroring function is started, the source device 10 may generate an encoder and / or a virtual display (e.g., a first virtual display 11 and / or a second virtual display 13) for each one or more apps requested to be executed by the external device 20. The source device 10 may generate a plurality of virtual displays to execute a plurality of apps simultaneously. The source device 10 may generate and operate, for example, one to six virtual displays.

[0048] The source device 10 may start an app corresponding to the virtual display. The source device 10 may transmit the mirroring data 31 including the mirroring image that can be displayed on the virtual display to the external device 20 by executing the app. The mirroring data 31 may include, for example, data for mirroring the virtual screen displayed on the virtual display generated by the source device 10 as it is to the mirroring display area 23 allocated to the display window 29 of the external device 20.

[0049] The external device 20 may display or project a real space image through the entire area of ​​the display window 29, and allocate a mirror display area 23 to a partial area of ​​the display window 29 to display a mirror image 27 through the mirror data 31 provided from the source device 10. The mirror display area 23 may include, for example, a plurality of display areas (e.g., Figure 6 a first display area 620 or at least one extended display area 640).

[0050] If an event is generated by the interaction 25 with the user, the external device 20 may transmit event information 33 corresponding to the generated event to the source device 10. If it is determined that the external device 20 can process the generated event by itself, for example, the external device 20 may process the generated event by itself. The event may be generated by the interaction 25 (for example, the user touches a specific position of the mirror image displayed in the mirror display area 27 or scrolls).

[0051] If the source device 10 receives event information from the external device 20, the source device 10 may process an operation corresponding to the event information on a corresponding application running on the virtual displays 11 and 13. The source device 10 may provide the external device 20 with mirroring data 31 corresponding to an image to be displayed on the virtual displays 11 and 13 according to the operation processing corresponding to the event information.

[0052] Figure 3 FIG. 2 is a diagram showing a mirroring system (eg, Figure 1 Block diagram of a mirror system).

[0053] Reference Figure 3 , source device (e.g., Figure 1 source device 10) and an external device (eg, Figure 1 The source device 10 and the external device 20 may be connected through the network 40 based on a predetermined communication protocol. If the source device 10 and the external device 20 are connected through the network 40, a communication channel 70 according to the predetermined communication protocol may be allocated. The communication channel 70 may be, for example, a radio channel or a wired channel.

[0054] For example, the source device 10 may include a display module 110 , a processor 120 , an input / output unit 130 , a memory 140 , or a communication module 150 .

[0055] The display module 110 may include a display panel or a touch panel. The display panel may visually provide information to the outside of the source device 10 (e.g., a user). The display 110 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. The touch panel may include a touch sensor configured to detect the touch of a user, or a pressure sensor configured to measure the strength of the force generated by the touch. The operation of the display module 110 may be controlled by an electrical connection with the processor 120. The operation of the display panel or the touch panel included in the display module 110 may be independently controlled by an electrical connection with the processor 120. In an embodiment, the display panel may be controlled by the processor 120 to visually provide display information to be displayed to the outside (e.g., a user).

[0056] The processor 120 may execute software (e.g., Fig.18 The processor 120 may control at least one other component (e.g., a hardware or software component) such as the electrically connected display module 110, the input / output module 130, the memory 140, or the communication module 150, and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 120 may store instructions or data received from other components (e.g., the display module 110, the communication module 150, or the input / output portion 127) in the memory 140 (e.g., a volatile memory), or process instructions or data stored in the memory 140 and store the processed result data in the memory 140.

[0057] The input / output unit 130 may receive a command or data to be used by a component (e.g., the processor 120) of the source device 10 from the outside (e.g., a user) of the source device 10. The input / output unit 130 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0058] The communication module 150 may establish a direct (e.g., wired) communication channel or a wireless communication channel with the external device 20, or support communication through the established communication channel. The communication module 150 may include one or more communication processors that may operate independently of the processor 120 and support direct (e.g., wired) communication or wireless communication. The communication module 150 may include a wireless communication module (e.g., Fig.18 A wireless communication module 1892) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., Fig.18A wired communication module 1894 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with each other via a network 40 (e.g., a short-range communication network such as Bluetooth TM The communication module 20 may be implemented as a single component (e.g., a single chip), or may be implemented as a plurality of components (e.g., a plurality of chips) separated from each other.

[0059] The memory 140 may store various data used by at least one component of the source device 10 (eg, the display module 110, the processor 120, the communication module 150, or the input / output unit 127). The various data may include, for example, software (eg, Fig.18 The memory 140 may include, for example, a volatile or non-volatile memory.

[0060] According to an example, the processor 120 may share an app list with the external device 20 by transmitting its app list to the external device 20 via the communication module 150. The app list may be a list of apps installed in the source device 10. The processor 120 may control, for example, the communication module 150 to connect the external device 20, and then, if a mirroring request is received from the external device 20, the app list is sent to the external device 20 through the communication module 150. For example, the processor 120 may generate a message including a message for displaying a virtual display (e.g., Figure 2 The external device 20 provides an initial mirror image of an icon of an app of a mirroring function on the first virtual display 11 and the second virtual display 13 , and provides initial mirror data 31 corresponding to the initial mirror image to the external device 20 .

[0061] According to an example, the processor 120 may activate the mirroring function in response to an app execution request transmitted from the external device 20 through the communication module 150. If the mirroring function is activated, the processor 120 may generate an encoder and / or a virtual display (e.g., Figure 2The processor 120 may generate a plurality of virtual displays to execute a plurality of apps simultaneously. The processor 120 may generate and operate, for example, one to six virtual displays. The processor 120 may start an app corresponding to the virtual display. The processor 120 may send mirroring data 31 including a mirror image that may be displayed on the virtual display to the external device 20 by executing the app. The mirroring data 31 may include, for example, data for mirroring the virtual screen displayed on the virtual display generated by the processor 120 as is to the mirroring display area 23 of the display window 29 assigned to the external device 20.

[0062] According to an embodiment, if event information is received from the external device 20 through the communication module 150, the processor 120 may process an operation corresponding to the event information on a corresponding app running on the virtual displays 11 and 13. The source device 10 may provide the external device 20 with mirroring data 31 corresponding to an image to be displayed on the virtual displays 11 and 13 according to the operation processing corresponding to the event information.

[0063] For example, the external device 20 may include a display module 210, a processor 220, a sensor module 230, a memory 240, or a communication module 250. The memory 240 may include a first buffer 241 or a second buffer 243. Although not shown, at least one of the first buffer area and the second buffer area may be allocated to a register inside the processor 220. The first buffer 241 or the first buffer area may temporarily store the mirroring data provided from the source device 10. The mirroring data temporarily stored in the first buffer 241 or the first buffer area may be discarded after another mirroring image is displayed in the first display area.

[0064] The display module 210 can visually provide information (e.g., real image information or mirror image information) to the outside of the external device 20 (e.g., a user). The display 210 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. The operation of the display module 210 may be controlled by an electrical connection with the processor 220.

[0065] The processor 220 may execute software (e.g., Fig.18The processor 220 may control at least one other component (e.g., a hardware or software component) such as the electrically connected display module 210, the sensor module 230, the memory 240, or the communication module 250, and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 220 may store instructions or data received from other components (e.g., the display module 210, the sensor module 230, or the communication module 250) in the memory 240 (e.g., a volatile memory), or process instructions or data stored in the memory 240 and store the processed result data in the memory 240.

[0066] The sensor module 230 may detect an operating state (e.g., power or temperature) of the external device 20, an external environment state (e.g., user state), or movement of a hand or gaze for user interaction, and may generate an electrical signal or data value corresponding to the detected state. The sensor module 230 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0067] The communication module 250 may establish a direct (e.g., wired) communication channel or a wireless communication channel with the source device 10, or support communication through the established communication channel. The communication module 250 may include one or more communication processors that may operate independently of the processor 220 and support direct (e.g., wired) communication or wireless communication. The communication module 250 may include a wireless communication module (e.g., Fig.18 A wireless communication module 1892) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., Fig.18 A wired communication module 1894 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with each other via a network 40 (e.g., a short-range communication network such as Bluetooth TM The communication module 10 may communicate with the source device 10 as an external electronic device through a wireless communication network such as a wireless cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network (LAN) or a wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as a plurality of components (e.g., a plurality of chips) separated from each other.

[0068] The memory 240 may store various data used by at least one component of the external device 20 (eg, the display module 210, the processor 220, the communication module 250, or the input / output unit 127). The various data may include, for example, software (eg, Fig.18 The memory 240 may include, for example, a volatile or non-volatile memory.

[0069] The external device 20 may also include a camera module. The camera module may capture still images and videos. The camera module may include one or more lenses, an image sensor, an image signal processor, or a flash. For example, the camera module may be used to interact with a user.

[0070] According to an example, the processor 220 may include a face tracking function, a hand tracking function, or a motion tracking function. The processor 220 may perform interaction with the user using at least one of the face tracking function, the hand tracking function, or the motion tracking function. The face tracking function may extract data of a portion corresponding to the face from the user image captured by the camera module, and may track changes in facial expressions based on the extracted data. The hand tracking function may extract data of a portion corresponding to the hand from the user image captured by the camera module, and track the movement of the hand based on the extracted data. The hand tracking function may track the movement of the hand based on the sensing information provided from the sensor module 230 or the sensing information provided from the source device 10. The motion tracking function may track the movement of the user from the user image captured by the camera module. The motion tracking function may track the movement of the user based on the sensing information provided from the sensor module 230 or the sensing information provided from the source device 10.

[0071] According to an example, the processor 220 may control the communication module 250 to connect the source device 10 and then request the mirroring function. The processor 220 may receive an app list from the source device 10 through the communication module 250 in response to the mirroring request. The processor 220 may obtain information about an app for using the mirroring function from the app list. The processor 220 may receive the initial mirroring data 31 from the source device 10. The processor 220 may display an initial mirroring screen in all or part of the mirroring display area provided in the display module 210 using the information about the app and / or the initial mirroring data 31 to use the mirroring function.

[0072] According to an example, the external device 220 may display or project a real space image through the entire area of ​​the display window of the display module 210, and display the mirror image 27 in a mirror display area allocated in a partial area of ​​the display window using the mirror data 31 provided from the source device 10. The mirror display area may include, for example, a plurality of display areas (e.g., Figure 6The processor 220 may display a mirror image in the first display area 620 using the mirror data transmitted in real time from the source device 10. The mirror data transmitted in real time may be temporarily stored in the first buffer 241 or the first buffer area allocated in the memory 240. If the mirror image presented in the first display area 620 changes, the mirror data temporarily stored in the first buffer 241 or the first buffer area may be updated with the next received mirror data. The processor 220 may display an extended mirror image on the extended display area 640 using the mirror data displayed in the first display area 620 or having a display history. The mirror data displayed in the first display area 620 or having a display history may be temporarily stored in the second buffer 243 or the second buffer area allocated in the memory 240. The mirror data stored in the second buffer 243 or the second buffer area may be discarded in order from old to new, or may be discarded according to the user's selection in consideration of the free space of the storage area.

[0073] According to an example, if an event is generated by interaction 25 with the user, the processor 220 may transmit event information 33 corresponding to the generated event to the source device 10. If the processor 220 determines that the generated event can be processed by itself, for example, the processor 220 may directly process the generated event. The event may be generated by interaction 25 such as, for example, a user touching a specific position of the mirror image displayed in the mirror display area 27 or scrolling.

[0074] Figure 4 FIG. 2 is a diagram showing a source device (eg, Figure 1 10 ).

[0075] Reference Figure 4 , the source device 10 may include a transmission management module (transmission manager) 410 , a container management module (app execution container manager) 420 , or a system service module (system service) 430 .

[0076] The transmission management module 410 can be used to transmit Figure 1 The transmission management module 410 may be used to transmit the mirror data (eg, Figure 2 The mirror data 310) is sent to the external device 20.

[0077] The container management module 420 may be used to control the mirroring logic. If, for example, an app execution request is provided from the external device 20 through the transmission management module 410, the container management module 420 may generate container units (app execution containers) 420-1, 420-2, and 420-N for executing the requested app. The container management module 420 may own or manage one or more container units 420-1, 420-2, and 420-N for the mirroring function. If an app-related request (e.g., Figure 2 33), the container management module 420 may use a container identifier (id) generated in response to the app for which the request is received to distribute the request and / or data to the corresponding container. For example, if an event is generated in the system service module 430, the container management module 420 may use a display identifier (display Id) or a task identifier (task Id) to specify a target container. The container management module 420 may be used to send the event to a specified target container or transmit the event to the external device 20 through the transmission management module 410.

[0078] The container units 420-1, 420-2, and 420-N may be used to control each mirroring session for the mirroring function. The container units 420-1, 420-2, and 420-N may have a unique identifier ID. The container units 420-1, 420-2, and 420-N may form a mapping relationship with a remote app container unit (remote app container manager) provided in the external device 20 using the unique identifier. The container units 420-1, 420-2, and 420-N may have a package name of the running app, a task identifier (id), user identification information (user ID), etc. The container units 420-1, 420-2, and 420-N may include virtual displays 421-1, 421-2, and 421-N, input surfaces 423-1, 423-2, and 423-N, or video encoders 425-1, 425-2, and 425-N, respectively.

[0079] For example, the container units 420-1, 420-2, and 420-N may generate a virtual display by generating video encoders 425-1, 425-2, and 425-N and designating input surfaces 423-1, 423-2, and 423-N of the video encoders 425-1, 425-2, and 425-N as surfaces of the virtual displays 421-1, 421-2, and 421-N. The container units 420-1, 420-2, and 420-N may execute apps corresponding to the virtual displays 421-1, 421-2, and 421-N.

[0080] According to an example, if an app is executed on the virtual displays 421-1, 421-2, and 421-N, the surface finger unit 431 included in the system service module 430 may display the app screen on the input surface of the corresponding virtual display. Mirroring data about the image or screen displayed on the virtual displays 421-1, 421-2, and 421-N by the surface finger unit 431 may be transmitted to the video encoders 425-1, 425-2, and 425-N to be compressed. The compressed mirroring data may be transmitted to the external device 20 through the transmission management module 410. The external device 20 may output a mirroring screen using the transmitted mirroring data.

[0081] The system service module 430 may include a remote app mode service unit 435, a surface finger 431, or a display manager 433. The remote app mode service unit 435 may perform functions requiring system permissions, such as executing an app on the virtual displays 421-1, 421-2, and 421-N or terminating an app.

[0082] As described above, the source device 10 may execute each of the plurality of apps on an independently generated virtual display, and send the screen to the external device 20 according to the execution of the app on each virtual display, so that the plurality of apps may float in the virtual space at the same time. The execution state of each app executed on the virtual display may be managed by the container management module 420. For example, the container management module 420 may manage each app based on a container, wherein the container includes a class responsible for input processing and a video encoder.

[0083] Figure 5 FIG. 2 is a diagram showing an external device (eg, Figure 1 20).

[0084] Reference Figure 5 , the external device 20 may include a transmission manager 510 , a remote app container manager 520 , or a container window manager 530 .

[0085] The remote app container management module 520 may be used to generate, disassemble / destroy or manage one or more remote app container units 520-1, 520-2 and 520-N. The remote app container management module 520 may be used to distribute the mirror data transmitted from the source device 10 through the transmission management module 510 to the corresponding remote app container units 520-1, 520-2 and 520-N. The remote app container management module 520 may be provided corresponding to the container management module 420 of the source device 10.

[0086] The remote app container units 520-1, 520-2, and 520-N may be responsible for one screen mirroring session. The remote app container units 520-1, 520-2, and 520-N may have a unique identifier (id). The remote app container units 520-1, 520-2, and 520-N may use the container id to correspond to a specific container unit that is one of the container units 420-1, 420-2, and 420-N of the source device 10.

[0087] The remote app container units 520 - 1 , 520 - 2 , and 520 -N may display the mirror image in the first display area or the extended area using the resolution and / or DPI determined by the container window management module 530 .

[0088] The remote app container units 520-1, 520-2, and 520-N may include video decoders 521-1, 521-2, and 521-N, output surfaces 523-1, 523-2, and 523-N, or surface views 525-2, 525-2, and 525-N. The remote app container units 520-1, 520-2, and 520-N may generate the video decoders 521-1, 521-2, and 521-N or the surface views 525-2, 525-2, and 525-N in response to execution of an app for a mirroring function in the source device 10. The video decoders 521-1, 521-2, and 521-N may decode the compressed mirroring data transmitted from the source device 10 through the transmission management module 510. The output surfaces 523-1, 523-2, and 523-N may transmit the mirror data decompressed by the video decoders 521-1, 521-2, and 521-N to the surface views 525-2, 525-2, and 525-N. The surface views 525-2, 525-2, and 525-N may display a mirror image in the first display area or display an extended mirror image in the extended display area using the mirror data transmitted through the output surfaces 523-1, 523-2, and 523-N.

[0089] The container window management module 530 may be used to determine a resolution and / or DPI of a mirroring screen corresponding to an app that will provide a mirroring function, and provide the determined resolution and / or DPI to the remote app container units 520-1, 520-2, and 520-N.

[0090] The transmission management module 510 may be used to transmit a mirroring request or response (ACK) message to the source electronic device 10 , or receive mirroring related data from the source electronic device 10 and transmit it to the remote app container management module 520 .

[0091] Figure 6 is a diagram showing an external device (eg, Figure 1An exemplary diagram of a screen display of a virtual space in an external device 20).

[0092] Reference Figure 6 , the external device 20 may include a display 600 that displays a virtual space. The display 600 may be, for example, a transparent display. The transparent display may have a transparent window. For example, if the external device 20 is a smart glass in a wearable device, the user may view a real image 610 of the real space through the transparent window. The real image 610 may be, for example, a perspective image or an image captured by a camera. The image captured by the camera and displayed on the display 600 may be processed or modified as needed.

[0093] According to an example, the display 600 may include a display area (eg, Figure 2 The mirrored display area 23 may include a first display area 620 or an extended display area 640. The extended display area 640 may include one or more extended display areas 641 and 642 in response to an event 630 generated by interaction with a user, such as a user's gesture.

[0094] can be based on the source device (e.g., Figure 1 The source device 10) finally provides the mirror data (for example, Figure 2 The mirror image data 310 is provided, and the mirror image 650 is displayed in the first display area 620. The mirror image data 310 provided last may be temporarily stored in the first buffer (eg, Figure 3 The mirror image 650 is displayed in the first display area 620, and the mirror data 31 temporarily stored in the first buffer 241 or the first buffer area may also be stored in the second buffer 243 or the second buffer area. The previous mirror data stored in the second buffer 243 or the second buffer area may be used to display the extended mirror image 660 in the extended display area 640 in response to the event 630 generated by the interaction with the user (such as the user's gesture). The extended mirror image 660 may include, for example, a first extended mirror image 661 displayed in the first extended display area 641 or a second extended mirror image 662 displayed in the second extended display area 642.

[0095] As described above, the extended mirroring image 660 may be displayed in the extended display area 640 of the display 600 using the mirroring data pre-transmitted and temporarily stored from the source device 10 in response to a user's request. This may avoid additional resource consumption for extending the mirroring screen and provide an efficient rendering process, thereby saving power consumption.

[0096] Although the extended display area 640 is shown in the drawings to be formed in the upward direction of the first display area 620, the present disclosure is not limited thereto. For example, the direction in which the extended display area 640 is provided may be determined by considering event information obtained through interaction with the user. The direction in which the extended display area 640 is provided may be, for example, one of the left direction, the right direction, or the lower direction relative to the first display area 620. The extended display area 640 may also be extended in the backward direction relative to the first display area 620. The extended display area 640 may be extended in, for example, a direction that is easy to extend (for example, a direction that is easy to extend in the display 600). The direction in which the extended display area 640 extends is not only in one direction, but may be extended in multiple directions in a radial form. The number of extended display areas 640 that can be extended in response to event generation in the display 600 may be limited by the number of mirror data to be used to obtain the extended mirror image. The second buffer 243 or the second buffer area that can limit the number of mirror data may be increased or decreased as needed. The extended display area 640 enables the functions and scope of the app provided with the mirror function to be additionally extended and expanded. Since the extended display area 640 can utilize the entire area in the field of view (FOV) of the user wearing the external device 20, the mirrored screen can be displayed by enlarging or extending the mirrored screen.

[0097] As described above, whenever an event is generated through interaction with the user, an immersive experience may be provided to the user by enlarging the mirror screen to display a mirror image.

[0098] Although in Figure 6 In the embodiment, one first display area 620 is assumed, but when multiple first display areas are presented, there may be a situation where the display area for providing the extended display area is insufficient. For example, the external display 20 may be operated to change the arrangement layout of the first display area and / or the extended display area displayed in the display area in response to a user's request input through interaction with the user, or to remove or hide some display areas.

[0099] According to the example, although it is assumed that the user can Figure 6 The real image 610 is actually viewed through the transparent window of the display 600 in the display 600, but the external device 20 can use a camera to capture an image in front in combination with the user's gaze and present the captured image through the display area.

[0100] According to the example, although it is assumed that the user can Figure 6 The real image 610 is actually viewed through the transparent window of the display 600 in the display area, but the external device 20 can present any prepared image through the display area.

[0101] According to an example, the external device 20 may divide the display area 600 into an area of ​​interest and an area of ​​non-interest. For example, the area of ​​interest may be specified in the display area 600 in consideration of the surrounding environment viewed by the user through eye tracking or the area reached by the user's sight. In this case, the external device 20 may also expand the extended display area 640 in the space divided into the area of ​​interest in the display area. Even if the display area 640 is not extended, the performance may be enhanced by operating a static image through the second buffer within the area of ​​interest.

[0102] Figure 7 is a diagram showing a mirroring system (eg, Figure 1 Mirror system 1) external device (for example, Figure 1 An exemplary diagram of extending a mirrored screen in an external device 20).

[0103] Reference Figure 7 , if the external device 20 receives data from a source device (e.g., Figure 1 If the source device 10 receives the first mirroring data 720a, the external device 20 may temporarily store the received first mirroring data 720a in a first buffer (eg, Figure 3 The external device 20 may also allocate a first display area to the display, and display the first mirror image 710a in the allocated first display area using the first mirror data 720a. If the first mirror image 710a is displayed in the first display area, the external device 20 may store the first mirror data 720a temporarily stored in the first buffer 241 or the first buffer area as the first extended mirror data 731 in the second buffer (e.g., Figure 3 the second buffer 243) or the second buffer area.

[0104] If the external device 20 receives the second mirroring data 720b from the source device 10, the external device 20 may temporarily store the received second mirroring data 720b in the first buffer (eg, Figure 3 The external device 20 may also use the second mirror image data 720b to display the second mirror image 710b in the first display area allocated to the display. If the extended mirror image is displayed by the interactive request (750b) with the user, the external device 20 may use the data stored in the second buffer (e.g., Figure 3The external device 20 may allocate the extended display area to the display and display the first extended mirror image 740b in the extended display area. If the second mirror image 710b is displayed in the first display area, the external device 20 may store the second mirror data 720b temporarily stored in the first buffer 241 or the first buffer area as the second extended mirror data 733 in the second buffer (e.g., Figure 3 the second buffer 243) or the second buffer area.

[0105] If the external device 20 receives the third mirroring data 720c from the source device 10, the received third mirroring data 720c may be temporarily stored in the first buffer (eg, Figure 3 The external device 20 may use the third mirror image data 720c to display the third mirror image 710c in the first display area allocated to the display. If the extended mirror image is displayed by the interactive request (750c, 760c) with the user, the external device 20 may use the data stored in the second buffer (e.g., Figure 3 The external device 20 may store the third mirror image 720c temporarily stored in the first buffer 241 or the first buffer area as the third extended mirror data 735 in the second buffer (e.g., Figure 3 the second buffer 243) or the second buffer area.

[0106] Figure 8 is a diagram showing a method for performing a mirroring system (eg, Figure 1 An illustration of a processing procedure for providing a mirroring function in a mirroring system 1).

[0107] Reference Figure 8 , the mirroring system 1 may include at least two electronic devices. The at least two electronic devices included in the mirroring system 1 may include, for example, providing mirroring data (for example, Figure 2 The mirror data 31) of the first electronic device (for example, Figure 1 The source device 10) and at least one second electronic device (eg, Figure 1The first electronic device 10 may be an external electronic device of the second electronic device 20. The second electronic device 20 may be an external electronic device of the first electronic device 10. The first electronic device 10 may be a digital device in which a program such as an app may be installed or executed, such as a smart phone or a tablet PC. The second electronic device 20 may be, for example, not only an external device 20 such as smart glasses, but also a digital device such as a smart phone, a tablet PC, or a personal computer with its own display. Below, for the sake of convenience of description, the first electronic device 10 is referred to as the source device 10, and the second electronic device 20 is referred to as the external device 20. However, the operations according to the embodiments to be described below may be equally applied to an electronic device with a display, such as the source device 10 or the external device 20.

[0108] In operation 811, the source device 10 and the external device 20 may perform a connection process. The connection process may include an operation in which at least one of the source device 10 or the external device 20 recognizes the counterpart device and then requests a connection. The connection process may include an operation in which at least one of the source device 10 or the external device 20 accepts the connection in response to a connection request from the counterpart device. Detailed operations according to the connection process may be defined by a protocol provided for each communication scheme supported by the source device 10 or the external device 20 (hereinafter referred to as a "communication protocol"). The communication protocol may be a protocol provided for communicating with the counterpart device in a network environment such as a short-distance communication network (e.g., Fig.18 1898) or a long-distance communication network (e.g. Fig.18 The protocol provided for the short-distance communication network 1898 may include, for example, Bluetooth, Wi-Fi Direct, or IrDA. The long-distance communication network 1899 may include, for example, a traditional cellular network, a 5G network, a next-generation telecommunication network, the Internet, or a computer network (e.g., a LAN or a remote communication network such as a WAN). The source device 10 and the external device 20 may enter the following state: by performing a connection process, a connection is formed that can send data to the other device (e.g., Figure 2 The mirroring function request of the external device 20 may be made, for example, after the radio channel is formed with the source device 10 through the connection process.

[0109] If the radio channel is formed according to the execution connection process, the source device 10 may monitor whether an app execution event is generated in operation 813. For example, when the external device 20 requests execution of a specific app, the source device 10 may determine the request as generating an app execution event. To this end, the source device 10 may provide the external device 20 with app information (e.g., an app list) that can be executed by the source device 10 in response to a mirroring function request from the external device 20. The app information may be included in the mirroring data generated using a screen (e.g., a virtual screen) on which an icon created when the app is installed in the source device 10 is displayed.

[0110] If the external device 20 receives the app information, the external device 20 may use the app information to configure and display a mirroring screen (eg, Figure 6 The external device 20 may recognize, through interaction with the user, that the user's input for requesting execution of a specific app is generated on the mirrored screen displayed in the mirrored display area 23.

[0111] If the mirroring function for a specific app is requested to be provided, the external device 20 may share the mirroring function with the source device 10. Therefore, in operation 817, the source device 10 and the external device 20 may perform mirroring and event processing operations. The mirroring and event processing operations may include, for example, operations performed by the external device 20 to present a mirroring image in the first display area 620 using mirroring data provided by the source device 10. The mirroring and event processing operations may include, for example, operations performed by the external device 20 to present an extended mirroring image in at least one extended display area 640 by reusing the mirroring data being displayed in the first display area 620 or mirroring data that has been previously displayed as a mirroring image.

[0112] In operation 819, the external device 20 may determine whether an app termination event is generated. For example, when a user's input indicates a request for terminating the mirroring function through interaction of the external device 20 with the user, the app termination event may be generated.

[0113] If an app termination event is generated, the external device 20 may transmit an app termination request to the source device 10 in operation 821. If app termination is requested, the external device 20 may transmit identification information indicating an app to be requested to be terminated together with the app termination request.

[0114] If the source device 10 receives an app termination request from the external device 20, the source device 10 may terminate the corresponding app and / or related tasks in operation 823. In operation 825, the source device 10 may release related resources for the mirroring function of the corresponding app requested to be terminated. For example, the source device 10 may release the container unit (e.g., Figure 4 If the container units 420-1, 420-2, and 420-N are released, the virtual display (eg, Figure 4 virtual displays 421-1, 421-2, and 421-N), input surfaces (e.g., Figure 4 Input surfaces 423-1, 423-2, and 423-N) and / or video encoders (e.g., Figure 4 The video encoders 425-1, 425-2 and 425-N) may also be discarded.

[0115] If the external device 20 transmits an app termination request to the source device 10, the external device 20 may release the related resources of the mirroring function for the app requested to be terminated in operation 827. For example, the external device 20 may release the remote app container unit (e.g., Figure 5 If the remote app container units 520-1, 520-2, and 520-N are released, the video decoders (e.g., Figure 5 video decoders 521-1, 521-2, and 521-N), output surfaces (eg, Figure 5 output surfaces 523-1, 523-2, and 523-N) or surface views (e.g., Figure 5 Surface views 525-1, 525-2, and 525-N) may also be discarded.

[0116] Fig. 9 is a diagram showing a mirroring system (eg, Figure 1 1) is a diagram showing the processing process of mirroring and event handling operations in a mirroring system.

[0117] Reference Fig. 9 If the mirroring and event handling operations are enabled, in operation 911, the source device 10 may allocate relevant resources to provide the mirroring function to each of the one or more apps that request the mirroring function. The source device 10 may generate, for example, a container unit (e.g., Figure 4Container units 420-1, 420-2, and 420-N). Container units 420-1, 420-2, and 420-N may include virtual displays (e.g., Figure 4 virtual displays 421-1, 421-2, and 421-N), input surfaces (e.g., Figure 4 Input surfaces 423-1, 423-2, and 423-N) and / or video encoders (e.g., Figure 4 As a result, in operation 911, the source device 10 may generate video encoders 425-1, 425-2, 425-N and virtual displays 421-1, 421-2, and 421-N for each app to perform a mirroring function.

[0118] In operation 913, the source device 10 may execute the corresponding app on the virtual displays 421-1, 421-2, and 421-N generated for each app. The operation of executing the app may correspond to executing the app in the allocated memory area. The source device 10 may obtain an image that can be displayed on the virtual displays 421-1, 421-2, and 421-N as a mirror image by executing the app on the virtual displays 421-1, 421-2, and 421-N. The source device 10 may encode the mirror image using the video encoders 425-1, 425-2, and 425-N generated for each app.

[0119] In operation 915, the source device 10 may transmit mirroring data including the mirroring image compressed by encoding to the external device 20. The mirroring data may include mirroring metadata for mirroring the mirroring image. The mirroring metadata may include, for example, information such as the size, resolution, DPI, or display position of the mirroring image. The source device 10 may be, for example, a digital device such as a smart phone or a tablet PC in which a program such as an app may be installed or executed.

[0120] The external device 20 may allocate relevant resources to provide the mirroring function for each app to be provided with the mirroring function. The external device 20 may generate, for example, a remote app container unit (eg, Figure 5 The remote app container units 520-1, 520-2, and 520-N may generate video decoders 521-1, 521-2, and 521-N, output surfaces 523-1, 523-2, and 523-N, or surface views 525-1, 525-2, and 525-N.

[0121] In operation 917, the external device 20 may decode the compressed mirroring data transmitted from the source device 10 through the corresponding video decoders 521-1, 521-2, and 521-N. In operation 917, the external device 20 may use the decompressed mirroring data to configure a first display area (eg, Figure 6 In operation 917, the external device 20 may temporarily store the decompressed mirroring data in a first buffer (eg, Figure 3 When new mirroring data is temporarily stored in the first buffer 241, the external device 20 may discard the existing mirroring data temporarily stored. In operation 917, the external device 20 may cache the mirroring data for displaying the mirroring image in the first display area 620 in the second buffer (e.g., Figure 3 in the second buffer 243).

[0122] In operation 919 , the external device 20 may detect an event generated by interaction with the user. The interaction may correspond to, for example, a touch or drag operation of the user performed in the first display area 620 .

[0123] If it is detected that an event is generated by interaction with the user, the external device 20 may determine whether the event is requesting an extension of the mirroring screen in operation 921. If the generated event does not correspond to an extension request, the external device 20 may transmit information about the input event to the source device 10 in operation 923. If the generated event corresponds to an extension request, the external device 20 may obtain an extended mirroring image using the mirroring data cached in the second buffer 243 and configure an extended display area (e.g., Figure 6 An extended display area 640) is used to display the obtained extended mirror image.

[0124] If the information about the input event is received from the external device 20, the source device 10 may process an operation according to the corresponding event using the information about the input event in operation 925. The source device 10 may configure mirror data reflecting the result of the processing operation for the corresponding event and transmit it to the external device 20.

[0125] Fig.10a and Fig.10b is an external device according to an embodiment (e.g., Figure 1 The external device 20) in the mirroring system (e.g., Figure 1 A control flow chart showing a mirror image in the mirror system 1).

[0126] Reference Fig.10a and Fig.10bIn operation 1011, the external device 20 may request the source device (eg, Figure 1 The source device 10 of the user executes the app selected by the user using the app information to execute the app. The app information may be, for example, information about the app that can be used on the source device (e.g., Figure 1 The external device 20 may receive, for example, app information from the source device 10. After the source device 10 generates a virtual display and then executes the app on the generated virtual display, the external device 20 may receive, from the source device 10, mirroring data of the app screen executed on the virtual display.

[0127] In operation 1013, the external device 20 may determine whether a mirroring event is generated by interaction with the user. If a mirroring event is generated, in operation 1015, the external device 20 may use the mirroring data provided from the source device 10 to display the image in the first display area (eg, Figure 6 The mirror image is displayed in the first display area 620).

[0128] If the mirror image is displayed in the first display area 620, the external device 20 may determine whether the extended mirror image is being allocated to the extended display area (eg, Figure 6 is displayed in the extended display area 640).

[0129] If the extended mirror image is displayed in the extended display area 640, the external device 20 may reconfigure the extended display area 640 in operation 1019. Reconfiguring the extended display area 640 may include, for example, newly configuring and displaying the extended mirror image to be displayed in the extended display area 640.

[0130] If the extended mirror image is not displayed in the extended display area 640 or the reconstruction of the extended display area is completed, the external device 20 may cache the mirror image displayed in the first display area 620 in a second buffer (eg, Figure 3 in the second buffer 243).

[0131] In operation 1023, the external device 20 may determine whether a user input is detected through interaction with the user. The user input may correspond to, for example, touching or dragging a specific position of the first display area 620 or the extended display area 640.

[0132] If the user input is detected, the external device 20 may analyze the user input in operation 1025. The analysis of the user input may be, for example, an operation for obtaining a type of the user input.

[0133] If the analysis of the user input is completed, the external device 20 may determine whether the user input is a request for displaying an extended mirroring image in the extended display area 640 in operation 1027. If the user input is not a request for displaying an extended mirroring image, the external device 20 may determine whether an app operation related event is generated in the first display area 620 in operation 1031.

[0134] If the user input is a request for displaying an extended mirroring image, the external device 20 may generate an extended mirroring image using the mirroring data cached in the second buffer 243 and display the generated extended mirroring image in the extended display area 640 in operation 1029 .

[0135] If a user input as an app-related event is generated in the first display area 620, the external device 20 may transmit input information according to the user input to the source device 10 in operation 1033. The input information may be used, for example, for the source device 10 to drive a corresponding app.

[0136] If a user input as an app-related event is generated in the extended display area 640, the external device 20 may transmit input information mapped to the extended display area 640 to the source device 10 in operation 1035. The input information may be used, for example, for the source device 10 to drive a corresponding app.

[0137] In operation 1037, the external device 20 may receive the mirroring data created by the source device 10 driving the corresponding app using the input information, and update the mirroring image to be displayed in the first display area 620 using the received mirroring data. The mirroring data may be temporarily stored in the first buffer 241 when received, for example. After, for example, updating the mirroring image to be displayed in the first display area 620, the mirroring data may be temporarily stored in the second buffer 243.

[0138] In operation 1039, the external device 20 may determine whether a termination event for requesting termination of the mirroring function is generated through interaction with the user. If a termination event is generated, in operation 1041, the external device 20 may send an app termination request to the source device 10. The app termination request may, for example, enable the source device 10 to discard a virtual display generated for executing the app.

[0139] Fig.11a FIG. 1 is a diagram showing an external device (eg, Figure 1 An exemplary diagram of extending a mirrored screen in an external device 20).

[0140] Reference Fig.11a, if the external device 20 displays a mirroring screen including a mirroring image in the first display area 1110, the external device 20 may expand the display area (eg, Figure 6 of the extended display area 640) to display the Figure 6 The extended mirroring screen is displayed according to the extended mirroring image around the first display area 1110 on the display 600).

[0141] According to an example, the external device 20 may additionally allocate one or more extended display areas in the -x axis direction 1131 in the first display area 1110. The external device 20 may display an extended mirror image in one or more extended display areas using a mirror image previously displayed in the first display area 1110 or currently displayed in the first display area 1110.

[0142] According to an example, the external device 20 may additionally allocate one or more extended display areas in the +x axis direction 1133 in the first display area 1110. The external device 20 may display an extended mirror image in one or more extended display areas using a mirror image previously displayed in the first display area 1110 or currently displayed in the first display area 1110.

[0143] According to one example, the external device 20 may additionally allocate one or more extended display areas in the +y axis direction 1121 in the first display area 1110. The external device 20 may display an extended mirror image in one or more extended display areas using a mirror image previously displayed in the first display area 1110 or currently displayed in the first display area 1110.

[0144] According to an example, the external device 20 may additionally allocate one or more extended display areas in the -y axis direction 1123 in the first display area 1110. The external device 20 may display an extended mirror image in one or more extended display areas using a mirror image previously displayed in the first display area 1110 or currently displayed in the first display area 1110.

[0145] According to an example, the external device 20 may additionally allocate one or more extended display areas in the +z axis direction 1141 in the first display area 1110. The external device 20 may display an extended mirror image in one or more extended display areas using a mirror image previously displayed in the first display area 1110 or currently displayed in the first display area 1110.

[0146] Fig.11b FIG. 1 is a diagram showing an external device (eg, Figure 1 An exemplary diagram of extending a mirrored screen in an external device 20).

[0147] Reference Fig.11b , if the external device 20 is not on the display (e.g., Figure 6 The mirror screen is provided on the display 600 of the source device 10 (eg, Figure 1 The source device 10) has a display (eg, Figure 3 The screen 1150 of the source device 10 seen through the transparent window (hereinafter referred to as “source device screen 1150”) may be used as the first display area (eg, Fig.11a For example, the external device 20 may expand the display area (eg, Figure 6 The extended mirroring screen may be displayed according to the mirroring image around the source device screen 1150 seen through the transparent window.

[0148] According to an example, the external device 20 may additionally allocate one or more extended display areas in the −x axis direction 1171 on the source device screen 1150. The external device 20 may display an extended mirror image in one or more extended display areas using the mirroring data provided by the source device 10.

[0149] According to an example, for example, the external device 20 may additionally allocate one or more extended display areas in the +x axis direction 1173 on the source device screen 1150. The external device 20 may display an extended mirroring image in one or more extended display areas using the mirroring data provided by the source device 10.

[0150] According to an example, for example, the external device 20 may additionally allocate one or more extended display areas in the +y axis direction 1161 on the source device screen 1150. The external device 20 may display an extended mirroring image in one or more extended display areas using the mirroring data provided by the source device 10.

[0151] According to an example, the external device 20 may additionally allocate one or more extended display areas in the −y axis direction 1163 on the source device screen 1150. The external device 20 may display an extended mirror image in one or more extended display areas using the mirroring data provided by the source device 10.

[0152] According to an example, for example, the external device 20 may additionally allocate one or more extended display areas in the +z axis direction 1181 on the source device screen 1150. The external device 20 may display an extended mirroring image in one or more extended display areas using the mirroring data provided by the source device 10.

[0153] In the above Fig.11a and Fig.11b In the embodiment, the position or travel direction (e.g., +x-axis direction, -x-axis direction, +y-axis direction, -y-axis direction, +z-axis direction, or -z-axis direction) for allocating the extended display area is determined based on input information obtained through interaction with the user, but when there is not enough space to provide the extended display area in the determined position or travel direction, the position of the first display area may be moved, the position of the extended display area may be changed, or the direction in which the extended display area is to be provided may be changed. To this end, the external device 20 may additionally perform interaction with the user.

[0154] Fig.12 is a diagram showing a mirroring system (eg, Figure 1 An exemplary diagram of a mirroring system 1) using a 3D mirroring image to extend a mirroring screen.

[0155] Reference Fig.12 , including an external device in the mirroring system 1 (e.g., Figure 1 The external device 20 ) may display a 3D mirroring image in the first display area 1210 using the 3D mirroring data provided from the source device 10 .

[0156] According to an example, the external device 20 may reuse the 3D mirroring data previously displayed in the first display area 1210, or may reuse the 3D mirroring data displayed in the first display area 1210 when the 3D mirroring data is newly provided from the source device 10, to display the 3D extended mirroring image in the extended display area 1220 extended upward (+y-axis direction).

[0157] According to an example, the external device 20 may reuse the 3D mirroring data previously displayed in the first display area 1210, or may reuse the 3D mirroring data displayed in the first display area 1210 when the 3D mirroring data is newly provided from the source device 10, to display the 3D extended mirroring image in the extended display area 1231 corresponding to the extended display area 1230 extended in the right direction (+x-axis direction). In this case, the external device 20 may display the three-dimensional extended mirroring image previously displayed in the first extended display area 1231 in the second extended display area 1233.

[0158] Fig.13 is a diagram showing a mirroring system (eg, Figure 1 Mirror system 1) external device (for example, Figure 1 The external device 20) in the extended display area (for example, Figure 6 An exemplary diagram of the interaction is provided in the extended display area 640 ).

[0159] Reference Fig.13 , external devices (e.g. Figure 1 The external device 20 of the embodiment of the present invention may arrange an extended display screen 1330 in which the immediately preceding mirror image is displayed, thereby hiding the first display screen 1310 in which the mirror image is displayed. In response to the number of times mirroring has been performed (e.g., the number of TABs), the external device 20 may extend the mirror image corresponding to each number of times and present it on an independent extended display screen 1340. Extending the extended display screens 1330, 1341, and 1343 in response to the number of mirroring times may be performed, for example, by the external device 20 interacting with the user in response to the number of TABs of the user.

[0160] If the external device 20 recognizes an input requesting display of the extended display screen 1320 in the upper direction (+y-axis direction) through interaction with the user (e.g., scrolling), it may present the extended display screen 1320 in series in the upper direction (+y-axis direction) of the first display screen 1310.

[0161] As described above, the external device 20 can determine the display direction of the extended display screen (e.g., +x axis direction, -x axis direction, +y axis direction, -y axis direction, +z axis direction or -z axis direction) according to the user's gesture type through interaction with the user.

[0162] According to an example, the external device 20 may prepare a table containing commands to be executed if an interaction is generated for each area or range of the first display screen or the extended display screen. If an event is generated in a specific area or range of the first display screen or the extended display screen while performing an interaction with the user, the external device 20 may perform an operation according to the command defined in the table for the specific area or range generated by the event. The table may be provided in an external server as well as the external device 20. The table may be provided in software. When a specific activity or screen is executed, a new command may be added to the table according to the implementation.

[0163] According to an example, instead of providing a command according to a motion (eg, active touch, scroll, etc.) in interaction with a user for each app, basic commands that may be commonly applied may be equally applied to all apps.

[0164] According to an example, a callback structure in software may be utilized to implement a different method to be executed in each case.

[0165] Table 1 below shows an example of a table defining commands executed for each interaction.

[0166] [Table 1]

[0167] According to an example, if the touch point after scrolling is where a touch event is generated, and if the corresponding page is dynamic when the corresponding command is executed, the external device 20 calls out the page again so that it can change in a different state when the touch input is provided. In this case, it can be determined whether the corresponding area can be interactive based on the pixel value difference between the image held by the buffer set in the external device 20 and the newly opened image.

[0168] According to an example, when a new mirror image is mirrored in the first display area, the external device 20 may move the previously displayed mirror image to the extended display area and display it. In this case, if the user touches a specific position of the extended mirror image presented in the extended display area, the external device 20 may move and display the extended mirror image displayed in the extended display area to the first display area. The external device 20 may also generate an event by considering that a position corresponding to the specific position touched by the user is touched in the mirror image moved and displayed in the first display area when the extended display area is displayed.

[0169] Fig.14 is a method for managing a mirroring system (e.g., Figure 1 The historical hierarchy tree in the extended display area of ​​the mirror system 1).

[0170] Reference Fig.14 , an external device in the mirroring system 1 that provides an extended area for the mirroring function (eg, Figure 1 The external device 20) can be used through interaction with the user (for example, Figure 2 Interaction 25) will expand the display area (for example, Figure 6 The extended display area 640) is allocated to the display (eg, Figure 6 The extended display area 640 allocated to the display 600 may have a hierarchical structure over time. The hierarchical structure may include a second layer (group 1, group 2, group 3) 1420, a third layer (subgroup 11, subgroup 12, subgroup 21, subgroup 22, subgroup 31, subgroup 32) 1430, or a fourth layer (question) 1440 relative to the first layer (estimate 1410) as the highest layer. For example, the external device 20 may map the extended display area 640 displaying the extended mirror image on the display 600 to one of the layers constituting the historical hierarchical tree and manage it. For example, if a specific interaction with the user is performed, layer management by interaction, such as removing areas in other trees, may be performed.

[0171] Fig.15 is a diagram showing a mirroring system (eg, Figure 1 An exemplary illustration of a self-operated extended display area in a mirroring system 1).

[0172] Reference Fig.15 , if the external device 20 can directly execute and operate software such as an app, the external device 20 may receive mirroring data 1510 about an app executed on a virtual display of the source device 10 from the source device 10. The external device 20 may display a mirror image in a first display area 1520 using the mirroring data 1510 provided from the source device 10. The external device 20 may execute the same app as the app executed on the virtual display of the source device 10 by itself. The external device 20 may display a screen 1530 according to the app executed by itself in an extended display area 1540.

[0173] Fig.16 is a diagram showing an external device (eg, Figure 1 An illustration of an example of extending a mirror image in time space by an external device 20).

[0174] Reference Fig.16 , the external device 20 may display a mirror image 1620 (hereinafter, referred to as a “basic mirror image 1620”) in the first display area 1610. If a future predicted image request event is generated, the external device 20 may obtain a duplicate mirror image 1640 by duplicating the mirror image displayed in the first display area 620. The external device 20 may render the duplicate mirror image 1640 and display it in an extended display area 1630 that is allocated to cover the basic mirror image 1620 displayed in the first display area 1610.

[0175] The external device 20 may obtain a future predicted image by performing an operation corresponding to an input provided in the extended display area 1630 displaying the replica mirror image 1640. The future predicted image may be rendered to the external device 20. The future predicted image 1660 may be displayed in the extended display area 1650 through interaction with a user after rendering.

[0176] Fig.17 is a method for transmitting data to an external device (e.g., Figure 1 A control flow chart for extending a mirror image in time and space in an external device 20).

[0177] Reference Fig.17 In operation 1711, the external device 20 may identify the occurrence of a future predicted image request event. The future predicted image request event may be generated, for example, through interaction between the external device 20 and a user. The future predicted image request event may be a request to predict an image currently displayed in the first display area (e.g., Fig.16 The basic mirror image (eg, Fig.16The mirror image appears next to the basic mirror image 1620).

[0178] For example, the external device 20 may maintain the current mirror image in response to a specific situation (e.g., Fig.16 In the state of the basic mirror image 1620), at least one interactive input that is expected to occur in the future is expected, an operation is performed according to the corresponding event, and the mirror display area (for example, Fig.16 The extended display area 1650 of the display unit 1650 visualizes the mirror image obtained due to the operation performed (e.g., Fig.16 Future prediction image 1660).

[0179] More specifically, in operation 1713, the external device 20 may determine whether the app for which the future prediction image is requested is an app capable of linear movement. Being capable of linear movement may mean that the linear movement is free. For example, an app capable of linear movement may include an app in which changes in the mirror image do not occur frequently. An app capable of linear movement may be, for example, an app that provides a weather forecast function.

[0180] In operation 1715, the external device 20 may determine whether no interaction with the user occurs within a predetermined time. The reason why no interaction with the user occurs within the predetermined time is that it can be predicted that the mirror image is unlikely to be changed. No interaction occurs within the predetermined time may correspond to the following situation: the app being executed to provide content to the basic mirror image 1620 displayed in the first display area 1610 is an app that does not need to interact frequently with the user.

[0181] If the app can be moved linearly and no interaction occurs within a predetermined period of time, the external device 20 may perform rendering by duplicating the basic mirror image 1620 displayed in the first display area 1610 to serve as a shield for the first display area 1610 in operation 1717. For example, the external device 20 may render the duplicated mirror image (e.g., Fig.16 The future predicted image 1660 is displayed in the extended display area 1650, so that the basic mirror image 1620 displayed in the first display area 1610 is covered.

[0182] In operation 1719, the external device 20 may display the copied mirror image 1640 to a region (eg, Fig.16 The input is provided to the extended display area 1630 of the display device so that a specified operation is performed, and the specified operation is performed to obtain the future predicted image 1660. The step (depth) of performing the operation for obtaining the future predicted image 1660 may be performed more than once.

[0183] If the future prediction image 1660 is obtained, in operation 1721, the external device 20 may use the second buffer (eg, Figure 3 The second buffer 243 of the image processing unit 160 is used to render the obtained future prediction image 1660. After rendering, the future prediction image 1660 can be visually presented to the user through the extended display area 1650 through interaction with the user.

[0184] In operation 1723 , the external device 20 removes the shielding so that the substantially mirroring image 1620 previously displayed in the first display area 1610 may be presented to the user again.

[0185] Through what has been described above, even if a function such as “preview next item” is not separately provided, the external device 20 can allow the user to experience a future predicted mirror image by utilizing a wide display area.

[0186] Fig.18 is a block diagram illustrating an electronic device 1801 in a network environment 1800 according to various embodiments.

[0187] Reference Fig.18 , the electronic device 1801 in the network environment 1800 may communicate with at least one of the electronic devices 1802 via a first network 1898 (e.g., a short-range wireless communication network), or communicate with the electronic device 1804 or the server 1808 via a second network 1899 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 1801 may communicate with the electronic device 1804 via the server 1808. According to an embodiment, the electronic device 1801 may include a processor 1820, a memory 1830, an input module 1850, a sound output module 1855, a display module 1860, an audio module 1870, a sensor module 1876, an interface 1877, a connection terminal 1878, a haptic module 1879, a camera module 1880, a power management module 1888, a battery 1889, a communication module 1890, a user identification module (SIM) 1896, or an antenna module 1897. In an embodiment, at least one of the components (e.g., the connection terminal 1878) may be omitted from the electronic device 1801, or one or more other components may be added in the electronic device 1801. According to an embodiment, some of the components (e.g., the sensor module 1876, the camera module 1880, or the antenna module 1897) may be integrated into a single component (e.g., the display module 1860).

[0188] The processor 1820 may execute, for example, software (e.g., program 1840) to control at least one other component (e.g., hardware or software component) of the electronic device 1801 coupled to the processor 1820, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 1820 may store a command or data received from another component (e.g., sensor module 1876 or communication module 1890) in the volatile memory 1832, process the command or data stored in the volatile memory 1832, and store the resulting data in the non-volatile memory 1834. According to an embodiment, the processor 1820 may include a main processor 1821 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 1823 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)), which may operate independently of the main processor 121 or in conjunction with the main processor 121. For example, when the electronic device 1801 includes a main processor 1821 and an auxiliary processor 1823, the auxiliary processor 1823 may be configured to use lower power than the main processor 1821 or be designated for a designated function. The auxiliary processor 1823 may be implemented separately from the main processor 1821 or as part of the main processor 1821.

[0189] When the main processor 1821 is in an inactive (e.g., sleep) state, the auxiliary processor 1823 may control at least some of the functions or states related to at least one of the components of the electronic device 1801 (e.g., display module 1860, sensor module 1876, or communication module 1890) instead of the main processor 1821, or when the main processor 1821 is in an active state (e.g., executing an application), the auxiliary processor 1823 may control at least some of the functions or states related to at least one of the components of the electronic device 1801 (e.g., display module 1860, sensor module 1876, or communication module 1890) together with the main processor 1821. According to an embodiment, the auxiliary processor 1823 (e.g., an image signal processor or a communication processor) may be implemented as a part of another component (e.g., a camera module 1880 or a communication module 1890) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 1823 (e.g., a neural processing unit) may include a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be generated via machine learning. Such learning may be performed, for example, by an electronic device 1801 executing artificial intelligence or via a separate server (e.g., server 1808). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recursive neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network (BRDNN), a deep Q network, or a combination of two or more thereof, but is not limited thereto. The artificial intelligence model may additionally or alternatively include a software structure in addition to a hardware structure.

[0190] The memory 1830 may store various data used by at least one component of the electronic device 1801 (e.g., the processor 1820 or the sensor module 1876). The various data may include, for example, software (e.g., program 1840) and input data or output data for commands related thereto. The memory 1830 may include a volatile memory 1832 or a nonvolatile memory 1834.

[0191] The program 1840 may be stored as software in the memory 1830 , and may include, for example, an operating system (OS) 1842 , middleware 1844 , or an application 1846 .

[0192] The input module 1850 may receive commands or data to be used by other components (e.g., the processor 1820) of the electronic device 1801 from outside (e.g., a user) of the electronic device 1801. The input module 1850 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0193] The sound output module 1855 can output sound signals to the outside of the electronic device 1801. The sound output module 1855 may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing recordings. The receiver may be used to receive incoming calls. Depending on the embodiment, the receiver may be implemented as being separated from the speaker, or as a part of the speaker.

[0194] The display module 1860 can visually provide information to the outside of the electronic device 1801 (e.g., a user). The display 1860 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. According to an embodiment, the display 1860 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the strength of a force generated by a touch.

[0195] The audio module 1870 may convert sound into an electrical signal, and vice versa. According to an embodiment, the audio module 1870 may obtain sound via the input module 1850, or output sound via the sound output module 1855 or an earphone of an external electronic device (e.g., electronic device 1802) directly (e.g., wired) or wirelessly coupled to the electronic device 1801.

[0196] The sensor module 1876 may detect an operating state (e.g., power or temperature) of the electronic device 1801 or an environmental state (e.g., a user's state) outside the electronic device 1801, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 1876 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an accelerometer, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0197] The interface 1877 may support one or more designated protocols for the electronic device 1801 to be directly (e.g., wired) or wirelessly coupled to an external electronic device (e.g., the electronic device 1802). According to an embodiment, the interface 1877 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0198] The connection terminal 1878 may include a connector via which the electronic device 1801 may be physically connected to an external electronic device (eg, the electronic device 1802). Depending on the embodiment, the connection terminal 1878 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

[0199] The haptic module 1879 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that can be recognized by the user via his tactile sense or kinesthetic sense. According to an embodiment, the haptic module 1879 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0200] The camera module 1880 may capture still images or moving images. According to an embodiment, the camera module 1880 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0201] The power management module 1888 may manage power supplied to the electronic device 1801. According to an embodiment, the power management module 1888 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0202] The battery 1889 may supply power to at least one component of the electronic device 1801. According to an embodiment, the battery 1889 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0203] The communication module 1890 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 1801 and an external electronic device (e.g., electronic device 1802, electronic device 1804, or server 1808), and perform communication via the established communication channel. The communication module 1890 may include one or more communication processors that can operate independently of the processor 1820 (e.g., an application processor (AP)) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 1890 may include a wireless communication module 1892 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1894 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate via a first network 1898 (e.g., a short-range communication network such as Bluetooth TMThe wireless communication module 1892 may communicate with the external electronic device 1804 through a wireless communication network (e.g., a wireless Fidelity (Wi-Fi) Direct connection, or an Infrared Data Association (IrDA)) or a second network 1899 (e.g., a long-distance communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network (LAN) or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 1892 may identify and authenticate the electronic device 1801 in a communication network (such as a first network 1898 or a second network 1899) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 1896.

[0204] The wireless communication module 1892 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technology). NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 1892 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 1892 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 1892 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 1804), or a network system (e.g., a second network 1899). According to an embodiment, the wireless communication module 1892 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0205] The antenna module 1897 may transmit a signal or power to the outside (e.g., an external electronic device) or receive a signal or power from the outside (e.g., an external electronic device). According to an embodiment, the antenna module 1897 may include an antenna including a radiator formed by a conductor or a conductive pattern formed on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 1897 may include a plurality of antennas (e.g., an antenna array). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 1898 or the second network 1899) may be selected from the plurality of antennas by, for example, the communication module 1890. A signal or power may then be transmitted or received between the communication module 1890 and the external electronic device via the selected at least one antenna. According to an embodiment, other components (e.g., a radio frequency integrated circuit (RFIC)) other than the radiator may be further formed as a part of the antenna module 197.

[0206] According to various embodiments, the antenna module 1897 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, an RFIC, and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or is adjacent to the first surface and is capable of supporting a specified high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or is adjacent to the second surface and is capable of transmitting or receiving signals of the specified high frequency band.

[0207] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicate signals (e.g., commands or data) therebetween.

[0208] According to an embodiment, a command or data may be sent or received between the electronic device 1801 and the external electronic device 1804 via the server 108 coupled to the second network 1899. Each of the external electronic device 1802 or the external electronic device 1804 may be a device of the same type as the electronic device 1801, or a device of a different type from the electronic device 1801. According to an embodiment, all or some operations to be executed in the electronic device 1801 may be executed in one or more of the external electronic device 1802, the external electronic device 1804, or the external electronic device 1808. For example, if the electronic device 1801 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 1801. The electronic device 1801 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 1801 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 1804 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 1899. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0209] According to an embodiment, an electronic device (e.g., Figure 1The external device 20) may include a display module 210 configured to display a mirror image on the display window 600, a communication module 250 configured to perform communication with the external device 10, a sensor module 230 configured to detect a user's gesture and generate a sensing signal corresponding to the detected user gesture, a memory 240 configured to allocate one or more buffer areas 241, 243, and at least one processor 220 electrically connected to the display module 210, the communication module 250, the sensor module 230 or the memory 240 and configured to control operations of the display module 210, the communication module 250, the sensor module 230 or the memory 240.

[0210] According to one embodiment, the at least one processor 220 may be configured to receive the mirroring data 31 from the external device 10 through the communication module 250 and control the display module 210 to display the mirroring image 650 in the first display area 620 allocated in the display window 600 using the mirroring data 31 .

[0211] According to an embodiment, at least one processor 220 may control the display module 210 to display an extended mirroring image 660 in at least one extended display area 640 allocated in the display window 600 using the mirroring data 31 according to a user gesture 630 detected by the sensor module 230 through interaction 25 with the user.

[0212] According to an embodiment, the extended mirroring data 731 , 733 , or 735 may be obtained from the previous mirroring data 731 , 733 , or 735 that has been used to display the mirroring image 650 .

[0213] According to an embodiment, the at least one processor 220 may be configured to temporarily store mirroring data corresponding to a new mirroring image displayed in the first display area 620 according to the user gesture 630 in the first buffer area 241 provided in the memory 240 .

[0214] According to an embodiment, the at least one processor 220 may be configured to temporarily store the mirroring data temporarily stored in the first buffer area as extended mirroring data in the second buffer area allocated to the memory if the mirroring image is displayed in the first display area.

[0215] According to an embodiment, at least one processor 220 may be configured to, if a mirror image is displayed in the first display area, obtain new extended mirror data from the mirror data temporarily stored in the first buffer area and temporarily store the new extended mirror data in a second buffer area allocated to the memory.

[0216] According to an embodiment, the at least one processor 220 may be configured to render the mirror image 650 in response to the user gesture 630 to obtain an extended mirror image 660 .

[0217] According to an embodiment, the mirroring data 31 may include meta information indicating a link or coordinates and image information to be mirrored.

[0218] According to an embodiment, if a new mirroring image 650 is displayed in the first display area 620 , the at least one processor 220 may reconfigure the extended mirroring image 660 displayed in the at least one extended display area 640 using mirroring data corresponding to the new mirroring image 650 .

[0219] According to an embodiment, at least one processor 220 may be configured to allocate at least one extended display area 640 to be adjacent to the first display area 620 in an up / down direction (e.g., +y-axis direction or -y-axis direction) or a left / right direction (e.g., +x-axis direction or -x-axis direction) according to a user gesture.

[0220] According to an embodiment, if a user gesture 630 is detected in at least one extended display area 640, at least one processor may identify input information mapped to the location where the user gesture 630 is detected in at least one extended display area 640, and control the communication module 250 to send the identified input information to the external electronic device 10.

[0221] According to an embodiment, the at least one processor 220 may be configured to set an interest region on the display window 600 through eye tracking of the sensor module 230 and allocate at least one extended display area 640 to the set interest region.

[0222] According to an embodiment, for use in an electronic device (e.g., Figure 1 The method for displaying in an external device 20) may include displaying a mirror image in a first display area allocated on the display based on a mirroring function.

[0223] According to an embodiment, the display method may include temporarily storing mirroring data corresponding to a mirroring image, and displaying the extended mirroring image in at least one extended display area allocated on a display using the mirroring data according to a user gesture obtained through interaction with a user.

[0224] According to an embodiment, the display method may include temporarily storing mirroring data corresponding to a new mirroring image displayed in the first display area according to a user gesture.

[0225] According to an embodiment, the display method may include temporarily storing the temporarily stored mirroring data as extended mirroring data if the mirroring image is displayed in the first display area.

[0226] According to an embodiment, the display method may include, if the mirroring image is displayed in the first display area, obtaining new extended mirroring data from the temporarily stored mirroring data, and temporarily storing the new extended mirroring data.

[0227] According to an embodiment, the step of displaying the extended mirror image may include rendering the mirror image in response to a user gesture to obtain the extended mirror image.

[0228] According to an embodiment, the mirroring data may include meta information indicating a link or coordinates and image information to be mirrored.

[0229] According to an embodiment, the displaying of the extended mirroring image may include, if a new mirroring image is displayed in the first display area, reconfiguring the extended mirroring image displayed in the at least one extended display area using mirroring data corresponding to the new mirroring image.

[0230] According to an embodiment, the step of displaying an extended mirror image may include: determining a direction in which at least one extended display area is to be allocated in an up / down direction (e.g., a +y-axis direction or a −y-axis direction) or a left / right direction (e.g., a +x-axis direction or a −x-axis direction) in the first display area according to a user gesture.

[0231] According to an embodiment, the display method may include: if a user gesture is detected in at least one extended display area, identifying input information mapped to a position where the user gesture is detected in the at least one extended display area, and transmitting the identified input information to an external electronic device.

[0232] According to an embodiment, the step of displaying the extended mirror image may include setting an interest region on the display by tracking eyes of the user, and allocating at least one extended display region to the set interest region.

[0233] The electronic device according to various embodiments of the present disclosure may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those described above.

[0234] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, and include various changes, equivalents or replacements of the corresponding embodiments. With respect to the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It should be understood that, unless otherwise clearly stated in the relevant context, the singular form of the noun corresponding to the project may include one or more things. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include all possible combinations of the items listed together in a corresponding phrase. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish a corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It should be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., the second element)”, “combined to another element (e.g., the second element)”, “connected with another element (e.g., the second element)” or “connected to another element (e.g., the second element)” when the term “operably” or “communicatively” is used or when the term “operably” or “communicatively” is not used, it means that the element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0235] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "part," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or part of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0236] The various embodiments described herein may be implemented as software (e.g., program 1840) including one or more instructions stored in a storage medium (e.g., memory 140) that can be read by a machine (e.g., digital device 120). For example, under the control of a processor, a processor (e.g., controller 123) of the machine (e.g., digital device 120) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0237] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., Play Store TM ) the computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a forwarding server).

[0238] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. Some of the multiple entities may be separately arranged in different components. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding one of the multiple components performs one or more functions before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

Claims

1. An electronic device (20), comprising: A display module (210) configured to display a mirror image on a display window (600); a communication module (250) configured to perform communication with an external device (10); A sensor module (230) configured to detect a user gesture and generate a sensing signal corresponding to the detected user gesture; A memory (240) configured to allocate one or more buffer areas (241, 243); as well as at least one processor (220), electrically connected to the display module (210), the communication module (250), the sensor module (230) or the memory (240), and configured to control the operation of the display module (210), the communication module (250), the sensor module (230) or the memory (240), The at least one processor (220) performs the following operations: Receiving mirror data (31) from the external device (10) via the communication module (250); Controlling the display module (210) to display a mirror image (650) in a first display area (620) allocated in the display window (600) using the mirror data (31); as well as The display module (210) is controlled to display an extended mirror image (660) in at least one extended display area (640) allocated in the display window (600) using the mirror data (31) according to the user gesture (630) detected by the sensor module (230) through interaction (25) with the user.

2. The electronic device according to claim 1, wherein: The at least one processor (220) is configured to temporarily store mirror data corresponding to a new mirror image displayed in the first display area (620) according to the user gesture (630) in a first buffer area (241) provided in the memory (240).

3. The electronic device according to claim 1, wherein: The at least one processor (220) is configured to render the mirror image (650) in response to the user gesture (630) to obtain the extended mirror image (660).

4. The electronic device according to claim 1, wherein: The mirroring data (31) includes meta information indicating a link or coordinates and image information to be mirrored.

5. The electronic device according to claim 1, wherein: If a new mirror image (650) is displayed in the first display area (620), the at least one processor (220) reconfigures the extended mirror image (660) displayed in the at least one extended display area (640) using mirror data corresponding to the new mirror image (650).

6. The electronic device according to any one of claims 1 to 5, wherein: The at least one processor (220) is configured to: allocate the at least one extended display area (640) to be adjacent to the first display area (620) in an up / down direction or a left / right direction according to the user gesture (630).

7. The electronic device according to any one of claims 1 to 5, wherein: If the user gesture (630) is detected in the at least one extended display area (640), the at least one processor (220) identifies input information mapped to the position where the user gesture (630) is detected in the at least one extended display area (640), and controls the communication module (250) to send the identified input information to the external electronic device (10).

8. The electronic device according to claim 7, wherein: The at least one processor (220) is configured to set an area of ​​interest on the display window (600) through eye tracking of the sensor module (230), and allocate at least one extended display area (640) to the set area of ​​interest.

9. A method for displaying in an electronic device, the method comprising: displaying (1015) a mirror image in a first display area allocated on the display based on the mirror function; Temporarily storing (1017, 1019) mirror data corresponding to the mirror image; as well as An extended mirror image is displayed (1019) in at least one extended display area allocated on the display using the mirror data according to a user gesture obtained through interaction with the user.

10. The method according to claim 9, further comprising: The mirroring data corresponding to the new mirroring image displayed in the first display area according to the user gesture is temporarily stored.

11. The method according to claim 9, wherein: The step of displaying the extended mirror image includes: rendering the mirror image in response to the user gesture to obtain the extended mirror image.

12. The method according to claim 9, wherein: The mirroring data includes meta information indicating a link or coordinates and image information to be mirrored.

13. The method according to claim 9, wherein: The displaying of the extended mirror image includes, if a new mirror image is displayed in the first display area, reconfiguring the extended mirror image displayed in the at least one extended display area using mirror data corresponding to the new mirror image.

14. The method according to any one of claims 9 to 13, wherein: The steps to display an extended mirror image include: Setting a region of interest on the display by tracking the user's eyes; and The at least one extended display area is allocated to the set region of interest.

15. The method according to any one of claims 9 to 13, further comprising: If the user gesture is detected in the at least one extended display area, identifying input information mapped to a location where the user gesture is detected in the at least one extended display area; as well as The recognized input information is sent to an external electronic device.

Images