Electronic device for providing augmented reality service and method of operating the same

Abstract

An apparatus and method for providing an augmented reality service in an electronic device by interoperating with another electronic device are provided. The first electronic device includes: a communication circuit; a camera; a sensor; a display device; and a processor operatively connected to the communication circuit, the display device, the sensor, and the camera. The processor is configured to: establish communication with a second electronic device; obtain information about at least one object included in the field of view of a user wearing the first electronic device through the camera or the sensor; obtain information about the user's gaze; determine whether to interoperate with the second electronic device for the augmented reality service based on the information about the user's gaze and the information about the at least one object; and when it is determined to interoperate with the second electronic device, send interoperability information about the augmented reality service to the second electronic device.

Description

Technical Field

[0001] The present disclosure relates to an apparatus and method for providing an augmented reality service in an electronic device. Background Art

[0002] The development of information and communication technology and semiconductor technology has promoted the development of various electronic devices into multimedia devices that provide various multimedia services. The multimedia services may include at least one of a voice call service, a messaging service, a broadcast service, a wireless Internet service, a camera service, an electronic payment service, or a music playback service.

[0003] An electronic device may provide various experiences to its user. For example, the electronic device may provide an augmented reality (AR) service that adds virtual information (e.g., a virtual object) to the real space.

[0004] The above information is presented only as background information to help understand the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above may be applicable as prior art to the present disclosure.

[0005] The above information is presented only as background information to help understand the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above may be applied as prior art to the present disclosure. Summary of the Invention

[0006] Technical Problem

[0007] Aspects of the present disclosure at least solve the above problems and / or disadvantages and at least provide the following advantages. Accordingly, one aspect of the present disclosure is to provide an apparatus and method for an augmented reality service in an electronic device (e.g., a mobile device) by interoperating with another electronic device (e.g., an electronic device providing an augmented reality service).

[0008] Additional aspects will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the presented embodiments.

[0009] According to an aspect of the present disclosure, the augmented reality device is provided by a mobile device, a projector, or a wearable device. The electronic device providing the augmented reality service is configured in various forms to enable a user to experience a real augmented reality service. For example, the electronic device providing the augmented reality service may be configured in the form of glasses worn on a user's face.

[0010] An electronic device (e.g., a wearable device) that provides an augmented reality service can provide the augmented reality service independently of an electronic device (e.g., a mobile device) such as a smartphone or a mobile communication terminal. For example, when an application (e.g., an augmented reality application) installed in the electronic device that provides the augmented reality service is executed, the electronic device can display virtual information (e.g., a virtual object) such that the virtual information overlaps at least a portion of the real space corresponding to the user's field of view.

[0011] Electronic devices such as smartphones or mobile communication terminals have become popular and can be used by a relatively larger number of users.

[0012] Therefore, an electronic device that provides an augmented reality service needs to interoperate with an electronic device such as a smartphone or a mobile communication terminal to provide various augmented reality services to users.

[0013] Solution to the problem

[0014] According to another aspect of the present disclosure, a first electronic device is provided. The electronic device includes: a communication circuit; a camera; a sensor; a display device; and a processor operatively connected to the communication circuit, the display device, the sensor, and the camera. Wherein, the processor can be configured to: establish communication with a second electronic device through the communication circuit; obtain information about at least one object corresponding to the field of view of a user wearing the first electronic device through at least one of the camera or the sensor; obtain information about the user's gaze; determine whether to use the second electronic device as an input device for an augmented reality service based on the information about the user's gaze and the information about the at least one object; and when it is determined to use the second electronic device as an input device for the augmented reality service, send information related to the input device for the augmented reality service to the second electronic device through the communication circuit. <*

[0015] According to another aspect of the present disclosure, a second electronic device is provided. The electronic device includes: a communication circuit; an input device; and a processor operatively connected to the communication circuit and the input device. Wherein, the processor can be configured to: establish communication with the first electronic device through the communication circuit; confirm the input processing method of the second electronic device if an input is detected through the input device; send input information to the first electronic device through the communication circuit if the input processing method is a first method for sharing input; and perform an operation corresponding to the input information if the input processing method is a second method for restricting input sharing.

[0016] According to another aspect of the present disclosure, a second electronic device is provided. The electronic device includes: a communication circuit; a display device; and a processor operatively connected to the communication circuit and the display device. Wherein, the processor may be configured to: receive, via the communication circuit, information about at least one object corresponding to the field of view of a user wearing the first electronic device from the first electronic device; based on the information about the at least one object, identify an object corresponding to the gaze of the user wearing the first electronic device; identify whether the object corresponding to the gaze of the user wearing the first electronic device is an object related to the second electronic device; and when the object corresponding to the gaze of the user wearing the first electronic device is an object different from an object related to the second electronic device, determine to use the second electronic device as an input device for an augmented reality service.

[0017] Other aspects, advantages, and salient features of the present disclosure will be apparent to those skilled in the art from the following detailed description and will be disclosed in conjunction with Figure 1 the appended drawings to disclose various embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1 is a block diagram of an electronic device in a network environment according to an embodiment of the present disclosure;

[0020] Figure 2 illustrates a configuration of electronic devices that interoperate with each other for an augmented reality service according to an embodiment of the present disclosure;

[0021] Figure 3 is a block diagram of a first electronic device for providing an augmented reality service according to an embodiment of the present disclosure;

[0022] Figure 4 is a block diagram of a second electronic device for providing an augmented reality service according to an embodiment of the present disclosure;

[0023] Figure 5 is a flowchart of the interoperability between a first electronic device and a second electronic device according to an embodiment of the present disclosure;

[0024] Figure 6A illustrates a screen configuration related to an augmented reality service displayed on a first electronic device according to an embodiment of the present disclosure;

[0025] Figure 6B illustrates a screen configuration related to an augmented reality service displayed on a first electronic device according to an embodiment of the present disclosure;

[0026] Figure 6C Shows a screen configuration related to an augmented reality service displayed on a first electronic device according to an embodiment of the present disclosure;

[0027] Figure 6D Shows screen configurations related to an augmented reality service displayed on a first electronic device according to various embodiments of the present disclosure;

[0028] Figure 7 Is a flowchart in which a first electronic device interoperates with a second electronic device based on information about a gazed object according to an embodiment of the present disclosure;

[0029] Figure 8A Shows a screen configuration of a first electronic device for interoperating with a second electronic device based on information about a gaze on an object according to an embodiment of the present disclosure;

[0030] Figure 8B Shows a screen configuration of a first electronic device for interoperating with a second electronic device based on information about a gaze on an object according to an embodiment of the present disclosure;

[0031] Figure 8C Shows a screen configuration of a first electronic device for interoperating with a second electronic device based on information about a gaze on an object according to an embodiment of the present disclosure;

[0032] Figure 8D Shows screen configurations of a first electronic device for interoperating with a second electronic device based on information about a gaze on an object according to various embodiments of the present disclosure;

[0033] Figure 9 Is a flowchart in which a first electronic device interoperates with a second electronic device based on information about a gaze on an object related to the second device according to an embodiment of the present disclosure;

[0034] Figure 10A Shows a screen configuration of a first electronic device for interoperating with a second electronic device based on user gaze information according to an embodiment of the present disclosure;

[0035] Figure 10B Shows a screen configuration of a first electronic device for interoperating with a second electronic device based on user gaze information according to an embodiment of the present disclosure;

[0036] Figure 10C Shows a screen configuration of a first electronic device for interoperating with a second electronic device based on user gaze information according to an embodiment of the present disclosure;

[0037] Figure 10DShows a screen configuration of a first electronic device for interacting with a second electronic device based on user gaze information according to an embodiment of the present disclosure;

[0038] Figure 11A Shows a screen configuration of a first electronic device for interacting with a second electronic device based on information about an object related to the second device with respect to a gaze;

[0039] Figure 11B Shows a screen configuration of a first electronic device for interacting with a second electronic device based on information about an object related to the second device with respect to a gaze;

[0040] Figure 11C Shows a screen configuration of a first electronic device for interacting with a second electronic device based on information about an object related to the second device with respect to a gaze;

[0041] Figure 12 Is a flowchart in which a first electronic device interacts with a second electronic device based on FOV information according to an embodiment of the present disclosure;

[0042] Figure 13 Shows a screen configuration of a first electronic device for interacting with a second electronic device based on FOV information about the first electronic device according to an embodiment of the present disclosure;

[0043] Figure 14 Is a flowchart in which a first electronic device interacts with a second electronic device based on location information according to an embodiment of the present disclosure;

[0044] Figure 15 Is a flowchart in which a first electronic device processes input information provided from a second electronic device according to an embodiment of the present disclosure;

[0045] Figure 16A Shows a screen configuration of a first electronic device for displaying a function UI based on input information provided from a second electronic device according to an embodiment of the present disclosure;

[0046] Figure 16B Shows a screen configuration of a first electronic device for displaying a function UI based on input information provided from a second electronic device according to an embodiment of the present disclosure;

[0047] Figure 16C Shows a screen configuration of a first electronic device for displaying a function UI based on input information provided from a second electronic device according to an embodiment of the present disclosure;

[0048] Figure 17AShows a screen configuration of a first electronic device for displaying a control UI based on input information provided from a second electronic device according to an embodiment of the present disclosure;

[0049] Figure 17B Shows a screen configuration of a first electronic device for displaying a control UI based on input information provided from a second electronic device according to an embodiment of the present disclosure;

[0050] Figure 17C Shows a screen configuration of a first electronic device for displaying a control UI based on input information provided from a second electronic device according to an embodiment of the present disclosure;

[0051] Figure 18A Shows a screen configuration of a first electronic device for controlling video content based on input information provided from a second electronic device according to an embodiment of the present disclosure;

[0052] Figure 18B Shows a screen configuration of a first electronic device for controlling video content based on input information provided from a second electronic device according to an embodiment of the present disclosure;

[0053] Figure 19A Shows a screen configuration of a first electronic device for providing guide information according to an embodiment of the present disclosure;

[0054] Figure 19B Shows a screen configuration of a first electronic device for providing guide information according to an embodiment of the present disclosure;

[0055] Figure 20A Shows a screen configuration of a first electronic device for displaying a function UI based on the form of a second electronic device according to an embodiment of the present disclosure;

[0056] Figure 20B Shows a screen configuration of a first electronic device for displaying a function UI based on the form of a second electronic device according to an embodiment of the present disclosure;

[0057] Figure 21 Is a flowchart of the second electronic device interacting with the first electronic device according to an embodiment of the present disclosure;

[0058] Figure 22 Is a flowchart of the second electronic device interacting with the first electronic device based on the control of the first electronic device according to an embodiment of the present disclosure;

[0059] Figure 23A Shows an example of changing a display method in the second electronic device based on the interaction with the first electronic device according to an embodiment of the present disclosure;

[0060] Figure 23BShows an example of a method for changing a display in a second electronic device based on an interaction operation with a first electronic device according to an embodiment of the present disclosure;

[0061] Figure 24A Shows an example of changing content to be displayed on a display device in a second electronic device based on an interaction operation with a first electronic device according to an embodiment of the present disclosure;

[0062] Figure 24B Shows an example of changing content to be displayed on a display device in a second electronic device based on an interaction operation with a first electronic device according to various embodiments of the present disclosure;

[0063] Figure 25 Is a flowchart for a second electronic device to determine an interaction operation with a first electronic device according to an embodiment of the present disclosure;

[0064] Figure 26 Is a flowchart for providing an IoT service in a first electronic device according to an embodiment of the present disclosure;

[0065] Figure 27A Shows a screen configuration of a first electronic device for registering an IoT device according to an embodiment of the present disclosure;

[0066] Figure 27B Shows a screen configuration of a first electronic device for registering an IoT device according to an embodiment of the present disclosure;

[0067] Figure 28 Is a flowchart for a second electronic device to provide an IoT service based on a first electronic device according to an embodiment of the present disclosure.

[0068] In all the drawings, the same reference numerals will be understood to refer to the same components, elements, and structures. Detailed Description

[0069] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of various embodiments of the present invention defined by the claims and their equivalents. The following description includes various specific details to assist in the understanding, but these details are regarded as merely exemplary. Therefore, those of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0070] The terms and vocabulary used in the claims and the following description are not limited to their literal meanings. Instead, these terms and vocabulary are used only by the inventors to make the understanding of the present disclosure clear and consistent. Therefore, those skilled in the art should understand that the following descriptions of various embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the present disclosure defined by the appended claims and their equivalents.

[0071] It should be understood that, unless otherwise clearly indicated in the context, the singular forms “a,” “an,” and “the” also include plural referents. Thus, for example, a reference to “a component surface” includes a reference to one or more such surfaces.

[0072] Figure 1 is a block diagram showing an electronic device 101 in a network environment 100 according to various embodiments. Referring to Figure 1 , the electronic device 101 in the network environment 100 may communicate with an electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or communicate with an electronic device 104 or a server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit. For example, the sensor module 176 (e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented as being embedded in the display device 160 (e.g., a display).

[0073] The processor 120 may run software (e.g., program 140), for example, to control at least one other component (e.g., a hardware component or a software component) connected to the electronic device 101 and may perform various data processing or computations. According to one embodiment, as at least part of the data processing or computation, the processor 120 may load commands or data received from another component (e.g., the sensor module 176 or the communication module 190) into the volatile memory 132, process the commands or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent of or combined with the main processor 121. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121 or may be adapted to be specifically used for a specified function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0074] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one of the components of the electronic device 101 (other than the main processor 121) (e.g., the display device 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control, together with the main processor 121, at least some of the functions or states related to at least one of the components of the electronic device 101 (e.g., the display device 160, the sensor module 176, or the communication module 190). According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 180 or the communication module 190) that is functionally related to the auxiliary processor 123.

[0075] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0076] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or an application 146.

[0077] The input device 150 may receive commands or data to be used by other components (e.g., the processor 120) of the electronic device 101 from the outside of the electronic device 101 (e.g., a user). The input device 150 may include, for example, a microphone, a mouse, or a keyboard.

[0078] The sound output device 155 may output a sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or playing a record, and the receiver may be used for an incoming call. According to an embodiment, the receiver may be implemented separately from the speaker or as a part of the speaker.

[0079] The display device 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the intensity of a force caused by the touch.

[0080] The audio module 170 may convert sound into an electrical signal and vice versa. According to an embodiment, the audio module 170 may obtain sound via the input device 150, or output sound via the sound output device 155 or headphones of an external electronic device (e.g., the electronic device 102) directly (e.g., wiredly) or wirelessly connected to the electronic device 101.

[0081] The sensor module 176 may detect an operating state of the electronic device 101 (e.g., power or temperature) or an environmental state outside the electronic device 101 (e.g., a state of a user), and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0082] The interface 177 may support one or more specific protocols used to directly (e.g., wiredly) or wirelessly connect the electronic device 101 to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0083] The connection end 178 may include a connector, where the electronic device 101 may be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0084] The haptic module 179 may convert an electrical signal into a mechanical stimulus (e.g., vibration or motion) or an electrical stimulus that can be confirmed by the user via his sense of touch or kinesthesia. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0085] The camera module 180 may capture a still image or a moving image. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash. <*

[0086] The power management module 188 may manage the power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0087] The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0088] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently of the processor 120 (e.g., an application processor (AP)), and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). Each corresponding one of these communication modules may communicate with an external electronic device via a first network 198 (e.g., a short-range communication network, such as Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or these various types of communication modules may be implemented as multiple separate components (e.g., multiple chips). The wireless communication module 192 may confirm and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an international mobile subscriber identity (IMSI)) stored in the user authentication module 196.

[0089] The antenna module 197 may transmit a signal or power to the outside of the electronic device 101 (e.g., an external electronic device) or receive a signal or power from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include one or more antennas, and thus, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected by, for example, the communication module 190 (e.g., the wireless communication module 192). Subsequently, a signal or power may be transmitted or received between the communication module 190 and an external electronic device via the selected at least one antenna.

[0090] At least some of the above components may be interconnected with each other via an inter-peripheral communication scheme (e.g., a bus, a general-purpose input / output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0091] According to an embodiment, commands or data may be sent or received between the electronic device 101 and an external electronic device 104 via a server 108 connected to a second network 199. Each of the electronic devices 102 and 104 may be a device of the same type as the electronic device 101 or a device of a different type from the electronic device 101. According to an embodiment, all or some of the operations running on the electronic device 101 may be run on one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 is to automatically execute a function or service or is to execute a function or service in response to a request from a user or another device, the electronic device 101 may request one or more of the external electronic devices to execute at least part of the function or service instead of running the function or service, or in addition to running the function or service, the electronic device 101 may also request one or more of the external electronic devices to execute at least part of the function or service. One or more of the external electronic devices that receive the request may execute the requested at least part of the function or service, or execute additional functions or additional services related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least part of a reply to the request with or without further processing of the result. For this purpose, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.

[0092] An electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to an embodiment of the present disclosure, the electronic device is not limited to the electronic devices described above.

[0093] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms corresponding to the respective embodiments. For the description of the drawings, like reference numerals may be used to refer to like or related elements. It will be understood that a singular noun corresponding to a term may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include all possible combinations of the items listed together in the corresponding one of the plurality of phrases. As used herein, terms such as "first" and "second" or "1st" and "2nd" may be used to simply distinguish the corresponding components from another component and do not limit the components in other respects (e.g., importance or order). It will be understood that, in the case where the terms "operatively" or "communicatively" are used or where the terms "operatively" or "communicatively" are not used, if one element (e.g., the first element) is referred to as "coupled with another element (e.g., the second element)", "coupled to another element (e.g., the second element)", "connected with another element (e.g., the second element)" or "connected to another element (e.g., the second element)", it means that the one element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0094] As used herein, the term "module" may include a unit implemented in hardware, software or firmware and may be used interchangeably with other terms (e.g., "logic", "logic block", "portion" or "circuit"). A module may be a single integrated component adapted to perform one or more functions or the smallest unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0095] The various embodiments described herein can be implemented as software (e.g., program 140) including one or more instructions readable by a machine (e.g., electronic device 101) stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke at least one of the one or more instructions stored in the storage medium and run the at least one instruction with or without using one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction invoked. The one or more instructions can include code generated by a compiler or code that can be run by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Herein, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being stored semi-permanently in the storage medium and data being stored temporarily in the storage medium.

[0096] According to an embodiment, a method according to various embodiments of the present disclosure can be included and provided in a computer program product. The computer program product can be traded between a seller and a purchaser as a product. The computer program product can be distributed (e.g., downloaded or uploaded) online via an application store (e.g., Play Store TM ) or can be directly distributed (e.g., downloaded or uploaded) between two user devices (e.g., smart phones). If it is distributed online, at least part of the computer program product can be generated temporarily, or at least part of the computer program product can be stored at least temporarily in a machine-readable storage medium (such as the memory of a manufacturer's server, an application store's server, or a forwarding server).

[0097] According to various embodiments, each of the above components (e.g., a module or a program) may include a single entity or multiple entities. According to various embodiments, one or more of the above components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding one of the multiple components performed the one or more functions before integration. According to various embodiments, the operations performed by a module, a program, or another component may be performed sequentially, in parallel, repeatedly, or in a heuristic manner, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

[0098] Figure 2 The configuration of an electronic device for interoperating with each other for an augmented reality service according to an embodiment of the present disclosure is shown.

[0099] Referring to Figure 2 , according to various embodiments, the first electronic device 200 may be worn on a user's head and may provide an image to the user. According to an embodiment, the first electronic device 200 may provide an augmented reality (AR) service by adding virtual information (e.g., a virtual object) to at least a part of an image (e.g., a preview image) corresponding to a real space obtained through a camera (e.g., Figure 1 the camera module 180 in ) of the first electronic device 200. According to an embodiment, the first electronic device 200 may provide an augmented reality service in which virtual information (e.g., a virtual object) is displayed so as to overlap at least a part of a real space corresponding to a field of view (FOV) of a user wearing the first electronic device 200. For example, the first electronic device 200 may provide an augmented reality service by adding virtual information (e.g., a virtual object) to be displayed on at least a part of a real space to light entering the eyes of a wearer of the first electronic device 200. As an example, the first electronic device 200 may be configured in the form of at least one of glasses, goggles, a helmet, or a hat, but is not limited thereto.

[0100] According to various embodiments, the second electronic device 210 may detect a user input. As an example, the second electronic device 210 may include, but is not limited to, at least one of a portable communication device (e.g., a smartphone) or a portable multimedia device that may be held and carried by a user.

[0101] According to various embodiments, the first electronic device 200 and the second electronic device 210 may interoperate with each other through wireless and / or wired connections. According to an embodiment, if the first electronic device 200 and the second electronic device 210 are authenticated by the same user, they may interoperate with each other through wireless and / or wired connections. According to an embodiment, the first electronic device 200 may provide an augmented reality service based on input information provided from the second electronic device 210 wirelessly connected thereto. As an example, the input information provided from the second electronic device 210 may include at least one of a touch input, an input on a physical button, or information about the movement of the second electronic device 210. As an example, the first electronic device 200 and the second electronic device 210 may interoperate with each other through at least one of short-range communication or cellular communication. For example, short-range communication may include at least one of Bluetooth, Bluetooth Low Energy (BLE), or Wireless LAN (e.g., Wi-Fi direct). As an example, cellular communication may include at least one of Long-Term Evolution (LTE), LTE-Advanced (LTE-A), 5G (or New Radio (NR)), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), or Global System for Mobile Communications (GSM).

[0102] Figure 3 is a block diagram of a first electronic device for providing an augmented reality service according to an embodiment of the present disclosure. As an example, the first electronic device 200 may be Figure 1 the electronic device 101 shown in the figure.

[0103] Referring to Figure 3 , according to various embodiments, the first electronic device 200 may include a processor 300, a camera module 310, a display device 320, a communication circuit 330, and / or a sensor module 340. According to an embodiment, the processor 300 may be the same as Figure 1 the processor 120 shown in the figure, or may be included in the processor 120. The camera module 310 may be the same as Figure 1 the camera module 180 shown in the figure, or may be included in the camera module 180. The display device 320 may be the same as Figure 1 the display device 160 shown in the figure, or may be included in the display device 160. The communication circuit 330 may be the same as Figure 1 the communication module 190 shown in the figure, or may be included in the communication module 190. The sensor module 340 may be the same as Figure 1 the sensor module 176 shown in the figure, or may be included in the sensor module 176.

[0104] According to various embodiments, the processor 300 may provide an augmented reality service. According to an embodiment, the processor 300 may control the display device 320 to display at least one virtual object such that the at least one virtual object is added to at least a portion of the real space corresponding to the field of view of the user wearing the first electronic device 200. According to an embodiment, the processor 300 may control the display device 320 to display an augmented reality image obtained by adding at least one virtual object to at least a portion of a real-time image (e.g., a preview image) captured by the camera module 310. As an example, the at least one virtual object may include at least one piece of information related to at least one object located in the real space obtained through at least one of the camera module 310 or the sensor module 340 (e.g., a spatial sensor), or content related to an application executed thereon. [[ID=...]] [[ID=...]]

[0105] According to various embodiments, the processor 300 may obtain information about the gaze of the user wearing the first electronic device 200. According to an embodiment, the processor 300 may determine the user's gaze based on at least one of head gaze or eye gaze. As an example, head gaze corresponds to the movement of the head of the user wearing the first electronic device 200 and may be configured based on the center of the display device 320 or the center of a real-time image obtained through the camera module 310 (e.g., an external camera). As an example, eye gaze may be configured based on the movement of the pupil of the user wearing the first electronic device 200, which is obtained through the sensor module 340 (e.g., an eye tracking sensor). [[ID=...]] [[ID=...]]

[0106] According to various embodiments, the processor 300 may determine whether to interoperate with the second electronic device 210 based on at least one object included in a region corresponding to the field of view of the user wearing the first electronic device 200. According to an embodiment, the processor 300 may determine whether to interoperate with the second electronic device 210 based on the dwell time of the user's gaze corresponding to an object included in the region corresponding to the field of view of the user wearing the first electronic device 200. For example, if the user's gaze corresponding to an object included in the region corresponding to the field of view of the user wearing the first electronic device 200 continuously maintains a reference time, the processor 300 may determine to interoperate with the second electronic device 210. For example, if the user's gaze corresponding to an object included in the region corresponding to the field of view of the user wearing the first electronic device 200 does not continuously maintain the reference time, the processor 300 may determine not to interoperate with the second electronic device 210. As an example, the object included in the region corresponding to the field of view of the user wearing the first electronic device 200 may include at least one real object (e.g., an IoT device) or virtual object detected through at least one of the sensor module 340 (e.g., a spatial sensor) or the camera module 310. [[ID=...]]

[0107] According to an embodiment, the processor 300 may determine whether to interoperate with the second electronic device 210 based on whether an object related to the second electronic device 210 is included in a region corresponding to the field of view of a user wearing the first electronic device 200. For example, if an object related to the second electronic device 210 is included in a region corresponding to the field of view of a user wearing the first electronic device 200, the processor 300 may determine not to interoperate with the second electronic device 210. For example, if an object related to the second electronic device 210 does not exist in a region corresponding to the field of view of a user wearing the first electronic device 200, the processor 300 may determine to interoperate with the second electronic device 210. As an example, if object information corresponding to an external configuration exceeding a reference range in the overall external configuration of the second electronic device 210 is obtained through at least one of the camera module 310 or the sensor module 340 (e.g., a space sensor), the processor 300 may determine that an object related to the second electronic device 210 is included in a region corresponding to the field of view of a user wearing the first electronic device 200. As an example, an object related to the second electronic device 210 may include at least one of an object related to the exterior of the second electronic device 210 obtained through at least one of the camera module 310 or the sensor module 340 (e.g., a space sensor) or an object related to the content displayed on the display device of the second electronic device (e.g., Figure 4 the display device 440 in ). As an example, the interoperability information with the second electronic device may include an operation in which the first electronic device uses the second electronic device as an input device for an augmented reality service.

[0108] According to an embodiment, the processor 300 may determine whether to interoperate with the second electronic device 210 based on a user's gaze corresponding to an object related to the second electronic device 210, the object being included in a region corresponding to the field of view of a user wearing the first electronic device 200. For example, if a user's gaze corresponding to an object related to the second electronic device 210, the object being included in a region corresponding to the field of view of a user wearing the first electronic device 200, is confirmed, the processor 300 may determine not to interoperate with the second electronic device 210. For example, if a user's gaze corresponding to an object related to the second electronic device 210 is maintained for a reference time, the object being included in a region corresponding to the field of view of a user wearing the first electronic device 200, the processor 300 may determine not to interoperate with the second electronic device 210. For example, if an object related to the second electronic device 210 does not exist in a region corresponding to the field of view of a user wearing the first electronic device 200, or if a user's gaze corresponding to an object other than an object related to the second electronic device 210 is confirmed, the processor 300 may determine to interoperate with the second electronic device 210.

[0109] According to an embodiment, the processor 300 may determine whether to interoperate with the second electronic device 210 based on information about the positions of the first electronic device 200 and the second electronic device 210. For example, if the second electronic device 210 is within the reference range of the first electronic device 200, the processor 300 may determine to interoperate with the second electronic device 210. For example, if the second electronic device 210 is outside the reference range of the first electronic device 200, the processor 300 may determine not to interoperate with the second electronic device 210. As an example, information about the position of the first electronic device 200 may be obtained based on the sensor module 340. Information about the position of the second electronic device 210 may be provided from the second electronic device 210 through a wireless path with the second electronic device 210.

[0110] According to various embodiments, if the processor 300 determines to interoperate with the second electronic device 210, the processor 300 may control (or manage) an augmented reality service based on input information provided from the second electronic device 210. According to an embodiment, if the processor 300 determines to interoperate with the second electronic device 210, the processor 300 may send interoperability information to the second electronic device 210. If the processor 300 receives input information from the second electronic device 210 in response to the interoperability information, the processor 300 may control the augmented reality service based on the input information provided from the second electronic device 210. For example, based on the input information provided from the second electronic device 210, the processor 300 may control the display device 320 to display a functional user interface (UI) corresponding to an object included in a region corresponding to the field of view of a user wearing the first electronic device 200. The processor 300 may control the operation of a corresponding object based on a menu corresponding to additional input information provided from the second electronic device 210 among one or more menus included in the functional UI. As an example, an object for displaying a functional user interface may include an object included in the objects included in a region corresponding to the field of view of a user wearing the first electronic device 200 and corresponding to the user's gaze at the time of receiving input information from the second electronic device 210. For example, the processor 300 may provide a control user interface corresponding to an object included in a region corresponding to the field of view of a user wearing the first electronic device 200 based on the input information provided from the second electronic device 210. As an example, the processor 300 may adjust a display form (e.g., size, position, or angle) of an object corresponding to the user's gaze based on the input information provided from the second electronic device 210. As an example, the input information provided from the second electronic device 210 may include, through an input device of the second electronic device 210 (e.g., Figure 4at least one of user input detected by the input device 420) or sensed data obtained through a sensor module (e.g., Figure 4 the sensor module 430) in the second electronic device 210. According to an embodiment, if the processor 300 determines not to interoperate with the second electronic device 210, the processor 300 may send non-interoperation information to the second electronic device 210. As an example, the interoperation information includes information related to an input device of an augmented reality service of the second electronic device 210.

[0111] According to various embodiments, if the processor 300 determines to interoperate with the second electronic device 210, the processor 300 may configure a display method related to an augmented reality service based on the orientation of the second electronic device 210. As an example, if the orientation of the relatively long axis of the second electronic device 210 configured in a vertical shape is parallel to the direction of gravity, the second electronic device 210 may be determined to be in the vertical direction, and if the orientation of the relatively long axis is perpendicular to the direction of gravity, the second electronic device 210 may be determined to be in the horizontal direction. According to an embodiment, the processor 300 may configure a method of displaying at least one virtual object for an augmented reality service based on the orientation of the second electronic device 210. For example, if the second electronic device 210 is in the vertical direction, the display device 320 may display the virtual object in the vertical direction. For example, if the second electronic device 210 is in the horizontal direction, the display device 320 may display the virtual object in the horizontal direction. According to an embodiment, the processor 300 may differently configure the form of the function UI based on the orientation of the second electronic device 210. As an example, the form of the function UI may include at least one of a menu type included in the function UI, a menu arrangement, or a method of switching menus. As an example, the orientation of the second electronic device 210 may be confirmed based on the sensed data provided from the second electronic device 210.

[0112] According to various embodiments, the processor 300 may interoperate with the second electronic device 210 to register an Internet of Things (IoT) device. According to an embodiment, the processor 300 may use at least one of the camera module 310 or the sensor module 340 (e.g., a spatial sensor) to detect an IoT device located in a region corresponding to the field of view of a user wearing the first electronic device 200. If there is no registration information on the IoT device, the processor 300 may send information related to the IoT device to the second electronic device 210. For example, if registration request information related to the IoT device is received from the second electronic device 210, the processor 300 may control the display device 320 to display a registration screen for the IoT device. If a registration input corresponding to the registration screen displayed on the display device 320 is detected, the processor 300 may register the IoT device at the location of the first electronic device 200 (e.g., in the living room). As an example, if registration information related to the IoT device is received from the second electronic device 210, the processor 300 may control the display device 320 to display a virtual object corresponding to the IoT device (e.g., information about the IoT device). As an example, information related to the IoT device may include the type of the device and / or information about its location.

[0113] According to various embodiments, the processor 300 may change the display method and / or display information of the display device 320 based on user gaze information. According to an embodiment, if it is determined that the gaze of a user wearing the first electronic device 200 points to an object related to the second electronic device 210, the object being included in a region corresponding to the user's field of view, the processor 300 may adjust the transparency and / or brightness of the virtual object displayed on the display device 440. For example, if it is determined that the user's gaze points to an object related to the second electronic device 210, the processor 300 may control the display device 320 so as to relatively increase the brightness of at least one virtual object displayed on the display device 320, or adjust its transparency such that the virtual object is relatively transparent, thereby facilitating the user to focus on the display screen of the second electronic device 210.

[0114] According to various embodiments, the camera module 310 may capture still images and / or videos. According to an embodiment, the camera module 310 may include an external camera (not shown) for capturing real-time images corresponding to the field of view of a user wearing the first electronic device 200.

[0115] According to various embodiments, the display device 320 may display information processed in the first electronic device 200. According to an embodiment, when a user gazes at the real space through the display device 320, the display device 320 may display at least one virtual object in an area corresponding to the field of view of the user wearing the first electronic device 200. As an example, the field of view of the user may include an angle or a range within which the user can recognize an object without turning the head or moving the pupil while looking straight ahead. As an example, the area corresponding to the user's field of view may be the same as the entire display area of the display device 320 or may be configured to be smaller than the entire display area. In addition, the area corresponding to the user's field of view may be determined or adjusted based on the distance from the user's eyes to the display device 320. According to an embodiment, the display device 320 may display an image in the form of augmented reality (AR) by adding at least one virtual object to a real-time image obtained through the camera module 310.

[0116] According to various embodiments, the communication circuit 330 may support communication between the first electronic device 200 and the second electronic device 210 using wired and / or wireless communication channels. For example, a wireless channel between the first electronic device 200 and the second electronic device 210 may be established based on a short-range communication scheme or a cellular communication scheme.

[0117] According to various embodiments, the sensor module 340 may detect an operating state or an external environmental state of the first electronic device 200 and may provide the detected information to the processor 300. According to an embodiment, the sensor module 340 may include a motion sensor for detecting motion of the first electronic device 200 (or the user). As an example, the motion sensor may include at least one of an acceleration sensor, a tilt sensor, a gyro sensor, or a 3-axis magnetic sensor. According to an embodiment, the sensor module 340 may include a position sensor (e.g., a global navigation satellite system (GNSS)) for detecting information about the position of the first electronic device 200. According to an embodiment, the sensor module 340 may include a detection sensor (e.g., a proximity sensor) for determining whether the first electronic device 200 is worn on the user. According to an embodiment, the sensor module 340 may include a space sensor for identifying an object included in an area (e.g., the real space) corresponding to the field of view of the user wearing the first electronic device 200. As an example, the space sensor may include at least one of an optical sensor (e.g., an infrared sensor) or a depth sensor.

[0118] According to various embodiments, if an object associated with the second electronic device 210 is included in a region corresponding to the field of view of a user wearing the first electronic device 200, the first electronic device 200 may determine to interoperate with the second electronic device 210. According to an embodiment, if a user gaze corresponding to an object associated with the second electronic device 210 is maintained for a reference time and the object is included in a region corresponding to the field of view of a user wearing the first electronic device 200, the processor 300 may determine to interoperate with the second electronic device 210.

[0119] Figure 4 is a block diagram of a second electronic device for providing an augmented reality service according to an embodiment of the present disclosure. As an example, the second electronic device 210 may be Figure 1 the electronic device 101 shown.

[0120] Refer to Figure 4 According to various embodiments, the second electronic device 210 may include a processor 400, a communication circuit 410, an input device 420, a sensor module 430, and / or a display device 440. According to an embodiment, the processor 400 may be the same as Figure 1 the processor 120 shown, or may be included in the processor 120. The communication circuit 410 may be the same as Figure 1 the communication module 190 shown, or may be included in the communication module 190. The input device 420 may be the same as Figure 1 the input device 150 shown, or may be included in the input device 150. The sensor module 430 may be the same as Figure 1 the sensor module 176 shown, or may be included in the sensor module 176. The display device 440 may be the same as Figure 1 the display device 160 shown, or may be included in the display device 160.

[0121] According to various embodiments, the processor 400 may configure an input processing method of the second electronic device 210. According to an embodiment, the processor 400 may configure the input processing method of the second electronic device 210 based on interaction operation information provided from the first electronic device 200. For example, if the interaction operation information is received from the first electronic device 200 through the communication circuit 410, the processor 400 may determine to interact with the first electronic device 200 to provide an augmented reality service. In this case, the processor 400 may configure the input processing method of the second electronic device 210 as a first method for sharing input information with the first electronic device 200. For example, if non-interaction operation information is received from the first electronic device 200 through the communication circuit 410, the processor 400 may determine that the interaction operation information for providing the augmented reality service with the first electronic device 200 is restricted. In this case, the processor 400 may configure the input processing method of the second electronic device 210 as a second method for restricting sharing of input information with the first electronic device 200.

[0122] According to an embodiment, the processor 400 may configure an input processing method of the second electronic device 210 based on information related to an augmented reality service provided from the first electronic device 200. As an example, the information related to the augmented reality service may include at least one of the following items: information about an object included in an area corresponding to a field of view of a user wearing the first electronic device 200, information about a gaze of the user wearing the first electronic device 200, or information about a position of the first electronic device 200. For example, the processor 400 may confirm whether the second electronic device 210 is to interoperate with the first electronic device 200 to provide an augmented reality service based on the information about the object included in the area corresponding to the field of view of the user wearing the first electronic device 200. As an example, the processor 400 may determine whether to interoperate with the first electronic device 200 based on at least one of a dwell time of a user gaze corresponding to an object included in the area corresponding to the field of view of the user wearing the first electronic device 200, whether an object related to the second electronic device 210 is included in the area corresponding to the field of view of the user wearing the first electronic device 200, or user gaze information corresponding to the object related to the second electronic device 210. For example, the processor 400 may confirm whether the second electronic device 210 is to interoperate with the first electronic device 200 to provide an augmented reality service based on the information about the position of the first electronic device 200. As an example, if the second electronic device 210 interoperates with the first electronic device 200 to provide an augmented reality service, the input processing method of the second electronic device 210 may be configured as a first method. As an example, if the second electronic device 210 that interoperates with the first electronic device 200 to provide an augmented reality service is restricted, the input processing method of the second electronic device 210 may be configured as a second method.

[0123] According to various embodiments, the processor 400 may perform control such that input information is selectively shared with the first electronic device 200. According to an embodiment, if the processor 400 detects an input through the input device 420 and / or the sensor module 430, the processor 400 may confirm the input processing method of the second electronic device 210. For example, if the input processing method of the second electronic device 210 is the first method, the processor 400 may control the communication circuit 410 to send the input information to the first electronic device 200. In this case, the processor 400 may restrict the second electronic device 210 from performing an operation corresponding to the respective input. For example, if the input processing method of the second electronic device 210 is the second method, the processor 400 may perform an operation corresponding to the input information.

[0124] According to various embodiments, if the second electronic device 210 interoperates with the first electronic device 200 to provide an augmented reality service, the processor 400 may change the display method and / or display information of the display device 440. According to an embodiment, if the processor 400 determines that the second electronic device 210 will interoperate with the first electronic device 200 to provide an augmented reality service, the processor 400 may adjust the active mode and / or brightness of the display device 440. For example, if the processor 400 determines to interoperate with the first electronic device 200, the processor 400 may control the display device 440 to switch to a deactivated mode. For example, if the processor 400 determines to interoperate with the first electronic device 200, the processor 400 may control the display device 440 to dim the display area of the display device 440. According to an embodiment, if the processor 400 determines that the second electronic device 210 will interoperate with the first electronic device 200 to provide an augmented reality service, the processor 400 may change the content of the display device 440. For example, if the processor 400 determines to interoperate with the first electronic device 200, the processor 400 may control the display device 440 to display content related to the augmented reality service provided from the first electronic device 200.

[0125] According to various embodiments, the processor 400 may interoperate with the first electronic device 200 to register an IoT device. According to an embodiment, the processor 400 may receive information related to the IoT device from the first electronic device 200 through the communication circuit 410. The processor 400 may confirm the registration information related to the IoT device provided from the first electronic device 200 through a memory (e.g., Figure 1 the memory in the memory 130). For example, if the registration information about the IoT device is confirmed, the processor 400 may control the communication circuit 410 to send the registration information about the IoT device to the first electronic device 200. For example, if the registration information on the IoT device is not confirmed, the processor 400 may control the communication circuit 410 to send registration request information to the first electronic device 200. If the registration information about the IoT device is received from the first electronic device 200 through the communication circuit 410, the processor 400 may register the IoT device at the location of the first electronic device 200 (or the second electronic device 210). As an example, the information related to the IoT device may include the type of the device identified in the first electronic device 200 and / or the information about the location of the first electronic device 200.

[0126] According to various embodiments, the communication circuit 410 may support communication between the first electronic device 200 and the second electronic device 210 using wired and / or wireless communication channels. For example, a wireless channel between the first electronic device 200 and the second electronic device 210 may be established based on a short-range communication scheme such as Bluetooth, BLE, or wireless LAN.

[0127] According to various embodiments, the input device 420 may receive commands or data related to the second electronic device 210 from the outside of the second electronic device 210 (e.g., a user). As an example, the input device 420 may include a microphone, a mouse, a keyboard, a digital pen (e.g., a stylus), or a touch circuit capable of detecting a touch input.

[0128] According to various embodiments, the sensor module 430 may detect an operating state or an external environmental state of the second electronic device 210 and may provide the detected information to the processor 400. According to an embodiment, the sensor module 430 may include a motion sensor for detecting the motion of the second electronic device 210 (or a user). According to an embodiment, the sensor module 430 may include a position sensor (e.g., GNSS) for detecting information about the position of the second electronic device 210. According to an embodiment, the sensor module 430 may include a detection sensor (e.g., a proximity sensor) for determining whether the second electronic device 210 is worn on a user.

[0129] According to various embodiments, the display device 440 may display information processed by the second electronic device 210. According to an embodiment, the display device 440 may display content related to an application executed in the second electronic device 210.

[0130] According to various embodiments, the first electronic device 200 may be based on the second electronic device 210 or an external server (e.g., Figure 1to provide an augmented reality service under the control of the server 108) in. According to an embodiment, based on information related to a virtual object provided from the second electronic device 210 or an external server, the first electronic device 200 may display at least one virtual object such that the at least one virtual object is added to at least a part of the real space corresponding to the field of view of the user wearing the first electronic device 200. As an example, the information related to the virtual object may include at least one of a position where the virtual object will be displayed, a type of the virtual object, or a configuration of the virtual object. According to an embodiment, the first electronic device 200 may send information (or spatial information) about an object included in an area corresponding to the field of view of the user wearing the first electronic device 200 or user gaze information obtained through at least one of the camera module 310 or the sensor module 340 (e.g., a spatial sensor) to the second electronic device 210 or the external server. According to an embodiment, the second electronic device 210 or the external server may determine an input processing method of the second electronic device 210 based on at least one of the information (or spatial information) about an object included in an area corresponding to the field of view of the user wearing the first electronic device 200 or the user gaze information provided from the first electronic device 200. For example, if a user gaze corresponding to an object related to the second electronic device 210 is confirmed and the object is included in an area corresponding to the field of view of the user wearing the first electronic device 200, the second electronic device 210 or the external server may configure the input processing method of the second electronic device 210 as a second method. For example, if an object related to the second electronic device 210 does not exist in an area corresponding to the field of view of the user wearing the first electronic device 200, or if a user gaze corresponding to an object other than the object related to the second electronic device 210 is confirmed, the second electronic device 210 or the external server may configure the input processing method of the second electronic device 210 as a first method. In this case, the second electronic device 210 may determine that the input of the second electronic device 210 is an input for controlling the augmented reality service.

[0131] According to various embodiments, the first electronic device (e.g., Figure 1 the electronic device 101 in or Figure 2 the first electronic device 200 in) may include: a communication circuit (e.g., Figure 1 the communication module 190 in or Figure 3 the communication circuit 330 in); a camera (e.g., Figure 1 the camera module 180 in or Figure 3 the camera module 310 in); a sensor (e.g., Figure 1 the sensor module 176 in or Figure 3 the sensor module 340 in); a display device (e.g., Figure 1the display device 160 in or Figure 3 the display device 320 in); and a processor operatively connected to the communication circuit, the display device, the sensor, and the camera (e.g., Figure 1 the processor 120 in or Figure 3 the processor 300 in). Wherein, the processor may be configured to: establish communication with a second electronic device through the communication circuit; obtain information about at least one object corresponding to the field of view of the user wearing the first electronic device through at least one of the camera or the sensor; obtain information about the user's gaze; determine whether to use the second electronic device as an input device for an augmented reality service based on the information about the user's gaze and the information about at least one object; and when it is determined to use the second electronic device as an input device for an augmented reality service, send information related to the input device for the augmented reality service to the second electronic device through the communication circuit.

[0132] According to various embodiments, the processor is further configured to obtain information about the user's gaze based on at least one of the head gaze or the eye gaze of the first electronic device.

[0133] According to various embodiments, the processor is further configured to control the display device to display at least one virtual object on at least a part of the area corresponding to the field of view of the user wearing the first electronic device.

[0134] According to various embodiments, the processor is further configured to determine to use the second electronic device as an input device for an augmented reality service when the user's gaze corresponding to any one of at least one object or at least one virtual object corresponding to the field of view of the user wearing the first electronic device has continuously maintained a reference time, and determine not to use the second electronic device as an input device for an augmented reality service when the user's gaze corresponding to any one object has not maintained the reference time.

[0135] According to various embodiments, the processor is further configured to: when an object related to the second electronic device exists in the area corresponding to the field of view of the user wearing the first electronic device, confirm whether the user's gaze points to the object related to the second electronic device; determine to use the second electronic device as an input device for an augmented reality service when the user's gaze points to an object other than the object related to the second electronic device; and determine not to use the second electronic device as an input device for an augmented reality service when the user's gaze points to the object related to the second electronic device.

[0136] According to various embodiments, the processor is further configured to: when it is determined to use the second electronic device as an input device for an augmented reality service, confirm whether input information is received from the second electronic device through the communication circuit; and when the input information is received from the second electronic device, perform an operation corresponding to the input information.

[0137] According to various embodiments, the input information received from the second electronic device may include at least one of a touch input, a physical button input, or information about the movement of the second electronic device.

[0138] According to various embodiments, the processor is further configured to, when receiving the input information from the second electronic device, identify an object corresponding to the user's gaze and control the display device to display a control menu related to the object corresponding to the user's gaze.

[0139] According to various embodiments, the second electronic device (e.g., Figure 1 the electronic device 101 in Figure 2 or the second electronic device 210 in Figure 1 the communication module 190 in Figure 4 or the communication circuit 410 in Figure 1 the input device 150 in Figure 4 or the input device 420 in Figure 1 the processor 120 in Figure 4 or the processor 400 in

[0140] According to various embodiments, the processor is further configured to: when receiving information related to the input device of the augmented reality service from the first electronic device through the communication circuit, configure the input processing method of the second electronic device as the first method; and when not receiving information related to the input device of the augmented reality service from the first electronic device through the communication circuit, or when receiving non-interactive operation information therefrom, configure the input processing method of the second electronic device as the second method.

[0141] According to various embodiments, the second electronic device may further include a display device, and the processor is further configured to, when receiving information related to the input device of the augmented reality service from the first electronic device through the communication circuit, control the display device to perform one of changing the display information or changing the display method of the display device.

[0142] According to various embodiments, the input device may include at least one of a touch circuit or a physical button.

[0143] According to various embodiments, a second electronic device (e.g., the electronic device 101 in Figure 1 or the second electronic device 210 in Figure 2 ) may include: a communication circuit; a display device; and a processor operatively connected to the communication circuit and the display device, wherein the processor may be configured to: receive, via the communication circuit, information of at least one object corresponding to a field of view of a user wearing a first electronic device; based on the information of the at least one object, identify an object corresponding to the gaze of the user wearing the first electronic device, and determine whether the object corresponding to the gaze of the user wearing the first electronic device is an object related to the second electronic device; and when the object corresponding to the gaze of the user wearing the first electronic device is an object different from an object related to the second electronic device, determine to use the second electronic device as an input device for an augmented reality service.

[0144] According to various embodiments, the processor is further configured to obtain information about the user's gaze based on at least one of a head gaze or an eye gaze of the first electronic device.

[0145] According to various embodiments, the processor is further configured to control the first electronic device to display at least one virtual object on at least a part of an area corresponding to a field of view of a user wearing the first electronic device.

[0146] According to various embodiments, the processor is further configured to determine to use the second first electronic device as an input device for an augmented reality service when an object related to the second electronic device is not included in at least one object corresponding to a field of view of a user wearing the first electronic device.

[0147] According to various embodiments, the processor is further configured to determine not to use the second first electronic device as an input device for an augmented reality service when the user's gaze points to an object related to the second electronic device.

[0148] According to various embodiments, the processor is further configured to, when determining to use the second electronic device as an input device for an augmented reality service, send information related to an operation corresponding to input information obtained by the second electronic device to the first electronic device.

[0149] According to various embodiments, the input information obtained by the second electronic device may include at least one of a touch input, a physical button input, or information about the movement of the second electronic device.

[0150] According to various embodiments, the processor is further configured to, when detecting the input information, identify an object corresponding to the user's gaze based on the information of at least one object, and send a control menu related to the object corresponding to the user's gaze to the first electronic device.

[0151] Figure 5 is a flowchart of the interaction operation between a first electronic device and a second electronic device according to an embodiment of the present disclosure. In the following embodiments, each operation may be performed sequentially, but it is not necessary to be performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. As an example, the first electronic device may be Figure 1 the electronic device 101 shown in Figure 3 or the first electronic device 200 shown in Figure 6A , Figure 6B , Figure 6C and Figure 6D to describe Figure 5 at least some of the elements shown in

[0152] Figure 6A shows a screen configuration related to an augmented reality service displayed on a first electronic device according to an embodiment of the present disclosure.

[0153] Figure 6B shows a screen configuration related to an augmented reality service displayed on a first electronic device according to an embodiment of the present disclosure.

[0154] Figure 6C shows a screen configuration related to an augmented reality service displayed on a first electronic device according to an embodiment of the present disclosure.

[0155] Figure 6D shows a screen configuration related to an augmented reality service displayed on a first electronic device according to an embodiment of the present disclosure.

[0156] Referring to Figure 5 , according to various embodiments, in method 500, in operation 501, a first electronic device (e.g., Figure 1 the processor 120 in Figure 1 the communication module 190 in Figure 3 the processor 300 in Figure 3 or the communication circuit 330 in Figure 2 ) may establish a communication link (or communication channel) with a second electronic device (e.g.,

[0157] According to various embodiments, in operation 503, a first electronic device (e.g., processor 120 or 300) may output information related to an augmented reality service. According to an embodiment, the processor 300 may identify at least one object existing in a real space corresponding to a field of view of a user wearing the first electronic device 200 through at least one of a camera module 310 or a sensor module 340 (e.g., a space sensor). The processor 300 may control a display device 320 to display at least one virtual object related to the at least one object existing in the real space.

[0158] Referring to Figure 6A , when the user gazes at the real space through the display device 320, the display device 320 may display at least one virtual object 614 and / or 616 in an area 610 corresponding to a field of view of a user wearing the first electronic device 200. As an example, the first virtual object 614 may include information (e.g., status information) related to an object 612 (e.g., a television set) in the real space. As an example, the second virtual object 616 may include information (e.g., a playback screen) related to an application (e.g., a music application) executed in the first electronic device 200. According to an embodiment, the processor 300 may control the display device 320 to display an augmented reality image in which at least one virtual object is added to at least a part of a live image (e.g., a preview image) obtained through the camera module 310. According to an embodiment, the processor 300 may configure a method of displaying a virtual object based on a direction of a second electronic device 210 connected to the first electronic device 200 through a communication link.

[0159] Referring to Figure 6B , if the second electronic device 210 is in a vertical direction (see 620), the processor 300 may control the display device 320 to display a plurality of virtual objects 632 in the area 610 corresponding to a field of view of a user wearing the first electronic device 200 in the vertical direction.

[0160] Referring to Figure 6C , if the second electronic device 210 is in a horizontal direction (see 640), the processor 300 may control the display device 320 to display a plurality of virtual objects 652 in the area 610 corresponding to a field of view of a user wearing the first electronic device 200 in the horizontal direction.

[0161] According to various embodiments, in operation 505, a first electronic device (e.g., processor 120 or 300) may determine whether, while providing an augmented reality service, it is with a second electronic device (e.g., Figure 2interact with the second electronic device 210). According to an embodiment, the processor 300 may determine whether to interact with the second electronic device based on information about an object included in a region corresponding to the field of view of a user wearing the first electronic device 200. As an example, the information about an object included in a region corresponding to the field of view of the user may include at least one of: a dwell time of the user's gaze corresponding to an object included in a region corresponding to the field of view of a user wearing the first electronic device 200, whether an object related to the second electronic device 210 is included in a region corresponding to the field of view of a user wearing the first electronic device 200, or information about the user's gaze corresponding to an object related to the second electronic device 210. According to an embodiment, the processor 300 may determine whether to interact with the second electronic device based on information about the positions of the first electronic device 200 and the second electronic device 210. As an example, the information about the interaction operation with the second electronic device may include an operation in which the first electronic device uses the second electronic device as an input device for an augmented reality service.

[0162] According to various embodiments, if the first electronic device (e.g., the processor 120 or 300) determines to interact with the second electronic device (e.g., Figure 2 the second electronic device 210 in ) for an augmented reality service (e.g., "Yes" in operation 505), then in operation 507, the first electronic device may send interaction operation information related to the augmented reality service to the second electronic device. As an example, the interaction operation information includes information related to an input device for the augmented reality service of the second electronic device 210. According to an embodiment, the processor 300 may control the communication circuit 330 to send information indicating an interaction operation with the second electronic device 210 for an augmented reality service to the second electronic device 210.

[0163] According to various embodiments, if the first electronic device (e.g., the processor 120 or 300) determines not to interact with the second electronic device (e.g., Figure 2 the second electronic device 210 in ) for an augmented reality service (e.g., "No" in operation 505), then the first electronic device may send non-interaction operation information for the augmented reality service to the second electronic device. According to an embodiment, if the processor 300 determines not to interact with the second electronic device 210 for an augmented reality service, the processor 300 may control the communication circuit 330 to send information indicating a non-interaction operation for an augmented reality service to the second electronic device 210.

[0164] According to various embodiments, in the case of providing an augmented reality service, the first electronic device 200 may display an object corresponding to the user's gaze to be different from other objects.

[0165] Refer again toFigure 6A , the first electronic device 200 may display a virtual object related to the user's gaze around an object 612 corresponding to the user's gaze. As an example, if the object 612 corresponding to the user's gaze is an active object, the virtual object related to the user's gaze may be displayed around the corresponding object 612.

[0166] Refer to Figure 6D , the first electronic device 200 may display an object 660 (e.g., a virtual object) corresponding to the user's gaze as relatively larger than other objects. As an example, the active object may include at least one of an object in the real space or a virtual object defined as being able to provide various information based on an augmented reality service among a plurality of objects included in a region corresponding to the field of view of the user wearing the first electronic device 200.

[0167] According to various embodiments, the first electronic device 200 may confirm whether to interact with the second electronic device 210 for an augmented reality service based on a request from the second electronic device 210. According to an embodiment, if input information is detected, the second electronic device 210 may send a signal to the first electronic device 200 to confirm whether an interaction operation for an augmented reality service is required. According to an embodiment, the first electronic device 200 may determine whether to interact with the second electronic device 210 for an augmented reality service based on information on objects included in a region corresponding to the field of view of the user wearing the first electronic device 200 when receiving the signal for confirming whether an interaction operation for an augmented reality service is required. According to an embodiment, the first electronic device 200 may determine whether to interact with the second electronic device 210 for an augmented reality service based on information on the positions of the first electronic device 200 and the second electronic device 210 when receiving the signal for confirming whether an interaction operation for an augmented reality service is required.

[0168] Figure 7 is a flowchart in which a first electronic device interacts with a second electronic device based on information on a gazed object according to an embodiment of the present disclosure. According to an embodiment, Figure 7 The operations in Figure 5 may be details of operation 505 in Figure 1 In the following embodiments, the respective operations may be executed sequentially, but are not necessarily executed sequentially. For example, the order of the respective operations may be changed, and at least two operations may be executed in parallel. As an example, the first electronic device may be Figure 3 the electronic device 101 shown in Figure 8A or Figure 8B the first electronic device 200 shown in Figure 8C and Figure 8D will be described with reference to Figure 7At least some of the elements shown in

[0169] Figure 8A The screen configuration of a first electronic device for interacting with a second electronic device based on information on a gaze at an object according to an embodiment of the present disclosure is shown.

[0170] Figure 8B The screen configuration of a first electronic device for interacting with a second electronic device based on information on a gaze at an object according to an embodiment of the present disclosure is shown.

[0171] Figure 8C The screen configuration of a first electronic device for interacting with a second electronic device based on information on a gaze at an object according to an embodiment of the present disclosure is shown.

[0172] Figure 8D The screen configuration of a first electronic device for interacting with a second electronic device based on information on a gaze at an object according to an embodiment of the present disclosure is shown.

[0173] Referring to Figure 7 , according to various embodiments, in method 700, in operation 701, a first electronic device (e.g., Figure 1 the processor 120 in Figure 3 or the processor 300 in Figure 2 ) may identify an object corresponding to the gaze of a user wearing the first electronic device (e.g.,

[0174] Referring to Figure 8A , a user wearing the first electronic device 200 may identify a plurality of objects 810 and 820 through a region 800 corresponding to the field of view of the user wearing the first electronic device 200. As an example, in Figure 8A , an object corresponding to the user's gaze may include a first object 810 on which the user's gaze 830 is displayed. As an example, the first object 810 may include a virtual object displayed on the display device 320, and the second object 820 may include an object (e.g., an IoT device) existing in the real space corresponding to the field of view of the user. As an example, the region 800 corresponding to the field of view of the user wearing the first electronic device 200 may be configured to be equal to or smaller than the display area of the display device 320.

[0175] According to various embodiments, in operation 703, a first electronic device (e.g., the processor 120 or 300) may identify an object corresponding to the gaze of a user wearing the first electronic device (e.g., Figure 2whether the object corresponding to the user's gaze of the first electronic device 200) is an active object. As an example, the active object may include at least one of a real-space object or a virtual object defined as being able to provide various information based on an augmented reality service among a plurality of objects included in an area corresponding to the field of view of the user wearing the first electronic device 200. For example, the active object may be configured or removed based on a user input.

[0176] According to various embodiments, if the object corresponding to the user's gaze of the first electronic device (e.g., Figure 2 the first electronic device 200) is an active object (e.g., "yes" in operation 703), then in operation 705, the first electronic device (e.g., the processor 120 or 300) may confirm whether the user's gaze on the active object has been continuously maintained for a reference time. According to an embodiment, the processor 300 may confirm whether Figure 8A the user's gaze 830 corresponding to the first object 810 has been continuously maintained for a reference time. As an example, the user's gaze may be configured based on at least one of a head gaze or an eye gaze.

[0177] According to various embodiments, if the user's gaze on the active object has been continuously maintained for a reference time (e.g., "yes" in operation 705), then in operation 707, the first electronic device (e.g., the processor 120 or 300) may determine to perform an interactive operation with the second electronic device (e.g., Figure 2 the second electronic device 210) for an augmented reality service. According to an embodiment, if the user's gaze on the active object included in the area corresponding to the field of view of the user wearing the first electronic device 200 has been maintained for a reference time, the processor 300 may determine to configure the second electronic device 210 as an input device for the augmented reality service.

[0178] According to various embodiments, if the object corresponding to the user's gaze of the first electronic device (e.g., Figure 2 the first electronic device 200) is not an active object (e.g., "no" in operation 703), or if the user's gaze on the active object has not been maintained for the reference time (e.g., "no" in operation 705), then the first electronic device (e.g., the processor 120 or 300) may determine not to perform an interactive operation with the second electronic device (e.g., the second electronic device 210) for the augmented reality service.

[0179] According to various embodiments, the first electronic device 200 may confirm whether to interact with the second electronic device 210 for an augmented reality service based on a request from the second electronic device 210. According to an embodiment, if input information is detected, the second electronic device 210 may send a signal to the first electronic device 200 to confirm whether an interaction operation for the augmented reality service is required. If the first electronic device 200 receives from the second electronic device 210 a signal to confirm whether an interaction operation for the augmented reality service is required while the user's line of sight remains on the active object, the first electronic device 200 may determine to interact with the second electronic device 210 for the augmented reality service. In this case, the first electronic device 200 may determine to interact with the second electronic device 210 for the augmented reality service regardless of the dwell time of the user's gaze on the active object.

[0180] According to various embodiments, if the first electronic device 200 determines to interact with the second electronic device 210 for the augmented reality service, the first electronic device 200 may control the augmented reality service based on the input information provided from the second electronic device 210.

[0181] Referring to Figure 8B , if the processor 300 receives information on a click input 841 from the second electronic device 210, the processor 300 may control the display device 320 to display a function UI 850 of the first object 810 corresponding to the user's gaze. As an example, the function UI 850 may be displayed to at least partially overlap the first object 810, or may be displayed adjacent thereto. As an example, the display device 320 may display a cursor on a first menu 851 (e.g., a cancel menu) among a plurality of menus corresponding to the first object 810 included in the function UI 850. As an example, if the processor 300 receives an additional input (e.g., a click input) from the second electronic device 210, the processor 300 may control the display device 320 to remove the display of the function UI 850 based on the first menu 851 on which the cursor is displayed.

[0182] Referring to Figure 8C, if the processor 300 receives information about a touch movement input 843 (e.g., drag) in a first direction (e.g., left direction) from the second electronic device 210, the processor 300 may move the cursor displayed on the function UI 850 of the first object 810 in the first direction (e.g., left direction). As an example, as a result of the cursor moving in the first direction (e.g., left direction), the display device 320 may display the cursor on a second menu 852 (e.g., time menu) among a plurality of menus corresponding to the first object 810 included in the function UI 850. As an example, if the processor 300 receives an additional input (e.g., label input) from the second electronic device 210, the processor 300 may control the display device 320 to display time information based on the second menu 852 on which the cursor is displayed.

[0183] Referring to Figure 8D , if the processor 300 receives information about a touch movement input 845 (e.g., drag) in a second direction (e.g., right direction) from the second electronic device 210, the processor 300 may move the cursor displayed on the function UI 850 of the first object 810 in the second direction (e.g., right direction). As an example, as a result of the cursor moving in the second direction (e.g., right direction), the display device 320 may display the cursor on a third menu 854 (e.g., location menu) among a plurality of menus corresponding to the first object 810 included in the function UI 850. As an example, if the processor 300 receives an additional input (e.g., label input) from the second electronic device 210, the processor 300 may control the display device 320 to display location information based on the third menu 854 on which the cursor is displayed.

[0184] Figure 9 is a flowchart of an operation in which a first electronic device interacts with a second electronic device based on information about a gaze on an object related to the second device according to an embodiment of the present disclosure. According to an embodiment, Figure 9 The operations in Figure 5 may be details of operation 505 in Figure 1 shown. In the following embodiments, the respective operations may be executed sequentially, but it is not necessary to execute them sequentially. For example, the order of the respective operations may be changed, and at least two operations may be executed in parallel. As an example, the first electronic device may be Figure 3 the electronic device 101 shown in Figure 10A or Figure 10B the first electronic device 200 shown in Figure 10C and Figure 10D As an example, reference will be made to Figure 11A , Figure 11B and Figure 11C will be described Figure 9At least some of the components shown in

[0185] Figure 10A The screen configuration of a first electronic device for interacting with a second electronic device based on user gaze information according to an embodiment of the present disclosure is shown.

[0186] Figure 10B The screen configuration of a first electronic device for interacting with a second electronic device based on user gaze information according to an embodiment of the present disclosure is shown.

[0187] Figure 10C The screen configuration of a first electronic device for interacting with a second electronic device based on user gaze information according to an embodiment of the present disclosure is shown.

[0188] Figure 10D The screen configuration of a first electronic device for interacting with a second electronic device based on user gaze information according to an embodiment of the present disclosure is shown.

[0189] Figure 11A The screen configuration of a first electronic device for interacting with a second electronic device based on information about an object related to a second device with respect to a gaze according to an embodiment of the present disclosure is shown.

[0190] Figure 11B The screen configuration of a first electronic device for interacting with a second electronic device based on information about an object related to a second device with respect to a gaze according to an embodiment of the present disclosure is shown.

[0191] Figure 11C The screen configuration of a first electronic device for interacting with a second electronic device based on information about an object related to a second device with respect to a gaze according to an embodiment of the present disclosure is shown.

[0192] Referring to Figure 9 According to various embodiments, in method 900, in operation 901, a first electronic device (e.g., Figure 1 the processor 120 in Figure 3 or the processor 300 in Figure 2 may identify an object corresponding to a user's gaze while wearing the first electronic device (the first electronic device 200 in

[0193] Referring to Figure 10A, a user wearing the first electronic device 200 can confirm an object 1010 (e.g., an IoT device) existing in the real space through an area 1000 corresponding to the field of view of the user wearing the first electronic device 200. The processor 300 can identify the object 1010 located in the real space corresponding to the field of view of the user through at least one of the camera module 310 or the sensor module 340 (e.g., a space sensor). As an example, in Figure 10A the object corresponding to the user's gaze may include the object 1010 on which the user's gaze 1020 is displayed.

[0194] According to various embodiments, in operation 903, the first electronic device (e.g., the processor 120 or 300) can confirm whether the object corresponding to the user's gaze of the user wearing the first electronic device (e.g., Figure 2 the first electronic device 200 in Figure 2 is an object related to the second electronic device (e.g.,

[0195] the second electronic device 210 in Figure 2 ). As an example, an object related to the second electronic device 210 may include an object related to the content displayed on the display device 440 of the second electronic device 210, which is detected by analyzing the image obtained through the camera module 310. As an example, an object related to the second electronic device 210 may include an object related to the outside of the second electronic device 210 identified by at least one of the camera module 310 or the sensor module 340 (e.g., a space sensor). Figure 2 Figure 2 Figure 10A According to various embodiments, if the object corresponding to the user's gaze of the user wearing the first electronic device (e.g., Figure 10A the first electronic device 200 in

[0196] is not an object related to the second electronic device (e.g., Figure 2 the second electronic device 210 in Figure 2If the object related to the second electronic device 210 (e.g., "Yes" in operation 903), the first electronic device (e.g., processor 120 or 300) may determine not to interact with the second electronic device (e.g., Figure 2 the second electronic device 210 in) for augmented reality services. According to an embodiment, if it is determined that the user wearing the first electronic device 200 is gazing at an object 1110 related to the second electronic device 210, and the object 1110 is included in the region 1100 corresponding to the user's field of view, as Figure 11A shown, the processor 300 may determine not to interact with the second electronic device 210 for augmented reality services.

[0197] According to various embodiments, if the first electronic device 200 determines to interact with the second electronic device 210 for augmented reality services, the first electronic device 200 may control the augmented reality service based on information about the input provided by the second electronic device 210.

[0198] Referring to Figure 10B , if the processor 300 receives information about the click input 1030 from the second electronic device 210, the processor 300 may control the display device 320 to display the function UI 1040 of the object 1010 corresponding to the user's gaze. As an example, the function UI 1040 may be displayed to at least partially overlap with the object 1010. As an example, the display device 320 may display a cursor on the first menu 1041 (e.g., cancel menu) among the multiple menus related to the object 1010 included in the function UI 1040.

[0199] Referring to Figure 10C , if the processor 300 receives information about the touch movement input 1032 (e.g., drag) in the second direction (e.g., right direction) from the second electronic device 210, the processor 300 may move the cursor displayed on the function UI 1040 in the second direction (e.g., right direction). As an example, as a result of the cursor moving in the second direction (e.g., right direction), the display device 320 may display the cursor on the third menu 1042 (e.g., position menu) among the multiple menus related to the object 1010 included in the function UI 1040.

[0200] Referring to Figure 10D, if the processor 300 receives information about a touch movement input 1034 (e.g., a drag) in a first direction (e.g., the left direction) from the second electronic device 210, the processor 300 may move the cursor displayed on the functional UI 1040 in the first direction (e.g., the left direction). As an example, as a result of the cursor moving in the first direction (e.g., the left direction), the display device 320 may display the cursor on a fourth menu 1044 (e.g., a switching menu) among a plurality of menus related to the object 1010 included in the functional UI 1040. According to an embodiment, if the processor 300 receives an additional input (e.g., a click input) from the second electronic device 840, the processor 300 may perform an operation corresponding to the menu 1041, 1042, or 1044 on which the cursor is displayed. As an example, if the processor 300 receives an additional input (e.g., a click input) from the second electronic device 840 when the cursor is displayed on the fourth menu 1044, the processor 300 may control the communication circuit 330 to send an enabling signal (or a disabling signal) to the IoT device corresponding to the object 1010.

[0201] According to various embodiments, if the first electronic device 200 determines not to perform interaction operation information for an augmented reality service, the second electronic device 210 may process the input information.

[0202] Referring to Figure 11A , the processor 400 may control the display device 440 to display a list of images stored in the second electronic device 210 based on the execution of the gallery application.

[0203] Referring to Figure 11B , if a tap input 1120 is detected through the input device 420 (e.g., a touch circuit).

[0204] Referring to Figure 11C , the processor 400 may enlarge and display a first image (see 1130) corresponding to the click input 1120.

[0205] Figure 12 is a flowchart in which a first electronic device interacts with a second electronic device based on FOV information according to an embodiment of the present disclosure. According to an embodiment, Figure 12 The operations in Figure 5 may be details of operation 505 in Figure 1 In the following embodiments, the respective operations may be executed sequentially, but it is not necessary to execute them sequentially. For example, the order of the respective operations may be changed, and at least two operations may be executed in parallel. As an example, the first electronic device may be Figure 3 the electronic device 101 shown in Figure 13 or the first electronic device 200 shown inFigure 12 at least some of the components shown in

[0206] Figure 13 A screen configuration of a first electronic device for interoperating with a second electronic device based on FOV information on the first electronic device according to an embodiment of the present disclosure is shown.

[0207] Referring to Figure 12 , according to various embodiments, in method 1200, in operation 1201, a first electronic device (e.g., Figure 1 processor 120 in Figure 3 or processor 300 in Figure 2 ) may confirm whether an object related to a second electronic device (e.g., Figure 2 second electronic device 210 in

[0208] According to various embodiments, if an object related to a second electronic device (e.g., Figure 2 second electronic device 210 in Figure 2 ) is not included in a region corresponding to the field of view of a user wearing a first electronic device (e.g., Figure 2 first electronic device 200 in Figure 10A ), the first electronic device (e.g., processor 120 or 300) may determine to interoperate with the second electronic device (e.g.,

[0209] According to various embodiments, if an object related to a second electronic device (e.g., Figure 2 second electronic device 210 in Figure 2in a region corresponding to the field of view of the user of the first electronic device 200) (e.g., "Yes" in operation 1201), the first electronic device (e.g., processor 120 or 300) may determine not to interact with the second electronic device (e.g., Figure 2 the second electronic device 210 in) for augmented reality services.

[0210] Referring to Figure 13 , if the object 1310 associated with the second electronic device 210 exists in the region 1300 corresponding to the field of view of the user wearing the first electronic device 200, the processor 300 may determine not to interact with the second electronic device 210 for augmented reality services.

[0211] Figure 14 is a flowchart in which a first electronic device interacts with a second electronic device based on location information according to an embodiment of the present disclosure. According to an embodiment, Figure 14 The operations in may be Figure 5 Details of operation 505 in. In the following embodiments, the respective operations may be performed sequentially, but not necessarily sequentially. For example, the order of the respective operations may be changed, and at least two operations may be performed in parallel. As an example, the first electronic device may be Figure 1 the electronic device 101 shown in or Figure 3 the first electronic device 200 shown in.

[0212] Referring to Figure 14 , according to various embodiments, in method 1400, in operation 1401, the first electronic device (e.g., Figure 1 the processor 120 in or Figure 3 the processor 300 in) may confirm information about the location of a second electronic device (e.g., Figure 2 the second electronic device 210 in) connected to the first electronic device (e.g., Figure 2 the first electronic device 200 in) via a communication link (or communication channel). According to an embodiment, the processor 300 may periodically receive information about the location of the second electronic device 210 from the second electronic device 210 through the communication circuit 330. According to an embodiment, when the location of the second electronic device 210 changes, the processor 300 may receive information about the changed location of the second electronic device 210 from the second electronic device 210 through the communication circuit 330.

[0213] According to various embodiments, in operation 1403, the first electronic device (e.g., processor 120 or 300) may confirm whether the second electronic device (e.g., Figure 2 the second electronic device 210 in) is within the first electronic device (e.g., Figure 2within the reference range of the first electronic device 200). As an example, the reference range can be configured based on information about the position of the first electronic device 200.

[0214] According to various embodiments, if the second electronic device (e.g., Figure 2 the second electronic device 210 in Figure 2 is within the reference range of the first electronic device (e.g., the first electronic device 200 in Figure 2 ), then in operation 1405, the first electronic device (e.g., the processor 120 or 300) can determine to perform an interactive operation with the second electronic device (e.g.,

[0215] the second electronic device 210 in Figure 2 for an augmented reality service. Figure 2 According to various embodiments, if the second electronic device (e.g., Figure 2 the second electronic device 210 in

[0216] Figure 15 is not within the reference range of the first electronic device (e.g., the first electronic device 200 in Figure 1 ), then in operation 1407, the first electronic device (e.g., the processor 120 or 300) can determine not to perform an interactive operation with the second electronic device (e.g., Figure 3 the second electronic device 210 in Figure 16A for an augmented reality service. Figure 16B and Figure 16C 、 Figure 17A 、 Figure 17B and Figure 17C 、 Figure 18A and Figure 18B 、 Figure 19A and Figure 19B as well as Figure 20A and Figure 20B describe Figure 15 at least some of the elements shown in

[0217] Figure 16A shows a screen configuration of a first electronic device that displays a function UI based on input information provided by a second electronic device according to an embodiment of the present disclosure.

[0218] Figure 16B Shows a screen configuration of a first electronic device that displays a function UI based on input information provided from a second electronic device according to an embodiment of the present disclosure.

[0219] Figure 16C Shows a screen configuration of a first electronic device that displays a function UI based on input information provided from a second electronic device according to an embodiment of the present disclosure.

[0220] Figure 17A Shows a screen configuration of a first electronic device that displays a control UI based on input information provided from a second electronic device according to an embodiment of the present disclosure.

[0221] Figure 17B Shows a screen configuration of a first electronic device that displays a control UI based on input information provided from a second electronic device according to an embodiment of the present disclosure.

[0222] Figure 17C Shows a screen configuration of a first electronic device that displays a control UI based on input information provided from a second electronic device according to an embodiment of the present disclosure.

[0223] Figure 18A Shows a screen configuration of a first electronic device that controls video content based on input information provided from a second electronic device according to an embodiment of the present disclosure.

[0224] Figure 18B Shows a screen configuration of a first electronic device that controls video content based on input information provided from a second electronic device according to an embodiment of the present disclosure.

[0225] Figure 19A Shows a screen configuration of a first electronic device for providing guide information according to an embodiment of the present disclosure.

[0226] Figure 19B Shows a screen configuration of a first electronic device for providing guide information according to an embodiment of the present disclosure.

[0227] Figure 20A Shows a screen configuration of a first electronic device for displaying a function UI based on the form of a second electronic device according to an embodiment of the present disclosure.

[0228] Figure 20B Shows a screen configuration of a first electronic device for displaying a function UI based on the form of a second electronic device according to an embodiment of the present disclosure.

[0229] Refer to Figure 15 , according to various embodiments, in method 1500, in operation 1501, a first electronic device (e.g., Figure 1the processor 120 in or Figure 3 the processor 300 in) may determine whether to interoperate with a second electronic device connected via a communication link (or communication channel) (e.g., Figure 2 the second electronic device 210 in) for an augmented reality service. According to an embodiment, as Figure 7 described in operations 701 to 709 in, the processor 300 may determine whether to interoperate with the second electronic device 210 for an augmented reality service based on the dwell time of the user's gaze corresponding to the active object. According to an embodiment, as Figure 9 described in operations 901 to 907 in, the processor 300 may determine whether to interoperate with the second electronic device 210 for an augmented reality service based on whether the user's gaze points to an object related to the second electronic device 210. For example, in the case of providing an augmented reality service, as Figure 16A shown.

[0230] Referring to Figure 16A , the processor 300 may control the display device 320 to display a virtual object 1610 (e.g., a playback screen) related to an application (e.g., a music application) executed in the first electronic device 200 on at least a part of the area 1600 corresponding to the field of view of the user wearing the first electronic device 200. If the user's gaze points to the virtual object 1610, the processor 300 may configure the second electronic device 210 as an input device for the augmented reality service. According to an embodiment, as Figure 12 described in operations 1201 to 1205 in, the processor 300 may determine whether to interoperate with the second electronic device 210 for an augmented reality service based on whether an object related to the second electronic device 210 is included in the area corresponding to the field of view of the user wearing the first electronic device 200. According to an embodiment, as Figure 14 described in operations 1401 to 1407 in, the processor may determine whether to interoperate with the second electronic device 210 for an augmented reality service based on the information about the positions of the first electronic device 200 and the second electronic device 210.

[0231] According to various embodiments, if the first electronic device (e.g., the processor 120 or 300) determines to interoperate with the second electronic device (e.g., Figure 2 the second electronic device 210 in) for an augmented reality service (e.g., "Yes" in operation 1501), then in operation 1503, the first electronic device may confirm whether it has received input information from the second electronic device (e.g., Figure 2 the second electronic device 210 in).

[0232] According to various embodiments, if a first electronic device (e.g., processor 120 or 300) receives input information (e.g., "yes" in operation 1503) from a second electronic device (e.g., Figure 2 the second electronic device 210 in Figure 2 ), then in operation 1505, the first electronic device may perform a function corresponding to the input information received from the second electronic device (e.g.,

[0233] Refer to Figure 16B . If the processor 300 receives information related to a click input 1622 from the second electronic device 210, the processor 300 may control the display device 320 to display a function UI 1630 of the virtual object 1610 corresponding to the user's gaze. As an example, the function UI 1630 may be displayed at least partially overlapping the virtual object 1610, or may be displayed adjacent thereto. As an example, the function UI 1630 may display a cursor on a first menu 1631 (e.g., cancel menu) among a plurality of menus corresponding to the virtual object 1610.

[0234] Refer to Figure 16C . If the processor 300 receives information about a touch movement input 1624 (e.g., drag) in a third direction (e.g., the left diagonal direction) from the second electronic device 210 while the function UI 1630 is being displayed, the processor 300 may move the cursor displayed on the function UI 1630 in the third direction (e.g., the left diagonal direction). As an example, as a result of moving in the third direction (e.g., the left diagonal direction), the cursor of the function UI 1630 may be displayed on a fifth menu 1632 (e.g., settings menu) among a plurality of menus corresponding to the virtual object 1610 included in the function UI 1630. According to an embodiment, if the processor 300 receives a selection input (e.g., a tag input) from the second electronic device 210 while the function UI 1630 is being displayed, as Figure 16B or Figure 16C shown, the processor 300 may perform an operation corresponding to the menu 1631 or 1632 on which the cursor is displayed (e.g., removing the function UI or displaying the settings menu).

[0235] According to various embodiments, if a first electronic device (e.g., processor 120 or 300) determines not to interact with a second electronic device (e.g., Figure 2 the second electronic device 210 in Figure 2When the second electronic device 210) receives the input information (e.g., "No" in operation 1503), the first electronic device may control (or manage) the augmented reality service based on the input information detected in the first electronic device (e.g., Figure 2 the first electronic device 200) therein.

[0236] According to various embodiments, if the first electronic device 200 determines to interact with the second electronic device 210 for an augmented reality service, the first electronic device 200 may control the display form (e.g., size, position, or angle) of the virtual object displayed on the display device 320 based on the input information provided by the second electronic device 210. In an embodiment, in the case of providing an augmented reality service, the processor 300 may control the display device 320 to display at least one virtual object 1710 on at least a part of the area 1700 corresponding to the field of view of the user wearing the first electronic device 200, as Figure 17A shown.

[0237] Referring to Figure 17A , the user wearing the first electronic device 200 may confirm the image object 1710 displayed on at least a part of the real space through the area 1700 corresponding to the field of view of the user wearing the first electronic device 200. As an example, if the user gazes at the image object 1710, the gaze information 1702 may be displayed on the image object 1710. According to an embodiment, if the processor 300 receives information about multiple click inputs 1722 from the second electronic device 210 while the user gazes at the image object 1710, as [[ID=28 shown.

[0238] Referring to ​ , the processor 300 may control the display device 320 to magnify the image object 1710 (see 1730). As an example, the image object 1710 may be magnified based on a predefined ratio or a ratio corresponding to the multiple tap inputs 1722.

[0239] Referring to ​ , if the processor 300 receives information 1724 about the movement of the second electronic device 210 in a first direction (e.g., the left direction) from the second electronic device 210 while the user gazes at the image object 1710, the processor 300 may control the display device 320 such that the image object 1710 moves in the first direction (e.g., the left direction) (see 1740). As an example, the moving distance of the image object 17

[0240] According to various embodiments, if the first electronic device 200 determines to interact with the second electronic device 210 for an augmented reality service, the first electronic device 200 may control the driving of an application (e.g., a video application) executed in the first electronic device 200 based on input information provided from the second electronic device 210. In an embodiment, in the case of providing an augmented reality service, as ​ shown.

[0241] Referring to ​ , the processor 300 may control the display device 320 to display an execution screen of an application executed in the first electronic device 200 on a region 1800 corresponding to the field of view of the user wearing the first electronic device 200. According to an embodiment, if the processor 300 receives information on a click input 1812 from the second electronic device 210 while an execution screen of an application is being displayed on the display device 320, as ​ shown, the processor 300 may control the display device 320 to display a function UI (or a control menu) 1820 of the application. As an example, a cursor may be displayed on a sixth menu (e.g., a play menu) 1822 among a plurality of menus included in the function UI 1820.

[0242] Referring to ​ , if the processor 300 receives information on a touch movement input 1814 (e.g., a drag) in a first direction (e.g., a left direction) from the second electronic device 210 while the function UI (or the control menu) 1820 of the application is being displayed, as ​ shown, the processor 300 may move the cursor displayed on the function UI 1820 in the first direction (e.g., the left direction), as ​ shown. As an example, the cursor of the function UI 1820 may move in the first direction (e.g., to the left) and then may be displayed on a seventh menu 1824 (e.g., a back menu) included in the function UI 1820. According to an embodiment, if, as <s ​ shown, while the function UI (or the control menu) 1820 of the application is being displayed, the user gazes at and points to the seventh menu 1824, the processor 300 may control the display device 320 to display the cursor displayed on the function UI 1820 on the seventh menu 1824 (e.g., the back menu).

[0243] According to an embodiment, if, as ​ or ​As shown, when the processor 300 receives information about a selection input (e.g., a click input) while the functional UI 1820 is being displayed, the processor 300 may perform an operation (e.g., play or rewind) corresponding to the menu 1822 or 1824 on which the cursor is displayed.

[0244] According to various embodiments, if the first electronic device 200 determines to interact with the second electronic device 210 for an augmented reality service, the first electronic device 200 may provide guidance information corresponding to a user input.

[0245] Referring to ​ and ​ , in the case of providing an augmented reality service, the processor 300 may control the display device 320 to display at least one virtual object in the area 1900 corresponding to the field of view of the user wearing the first electronic device 200. For example, as ​ shown, the display device 320 may display one or more virtual objects 1910 arranged horizontally. For example, as ​ shown, the display device 320 may display one or more virtual objects 1930 arranged in a grid form. According to an embodiment, the processor 300 may control the display device 320 to display user gaze information in a form corresponding to the form of displaying one or more virtual objects. For example, in the case where one or more objects 1910 are arranged horizontally as ​ shown, the display device 320 may display the user gaze information 1920 in a form that can move left / right. For example, in the case where one or more objects 1930 are arranged in a grid form as ​ shown, the display device 320 may display the user gaze information 1940 in a form that can move up / down / left / right. As an example, among one or more virtual objects displayed in the area 1900 corresponding to the field of view of the user wearing the first electronic device 200, the display device 320 may display an object corresponding to the cursor to be relatively larger or have a different depth.

[0246] According to various embodiments, the first electronic device 200 may configure the functional UI differently based on the motion information (e.g., direction) of the second electronic device 210.

[0247] Referring to ​, the user wearing the first electronic device 200 can confirm the object 2002 in the real space through the area 2000 corresponding to the field of view of the user wearing the first electronic device 200. As an example, the object 2002 may correspond to an external device that supports IoT services. If the processor 300 receives input information (e.g., click input) from the second electronic device 210 when the user's gaze points to the object 2002, the processor 300 may control the display device 320 to display a control menu (e.g., function UI) corresponding to the object 2002. For example, if the second electronic device 210 is in a vertical direction (see 2010), as shown ​ As shown, the display device 320 may display a relatively simple control menu 2020 corresponding to the object 2002. As an example, the control menu 2020 may include a switch menu and / or a brightness adjustment menu.

[0248] Reference ​ If the second electronic device 210 is in a horizontal direction (see 2030), the display device 320 may display a relatively detailed control menu 2040 corresponding to the object 2002. As an example, the control menu 2040 may include a menu for adjusting the color of the object 2002.

[0249] ​ is a flowchart of the interaction operation between the second electronic device and the first electronic device according to an embodiment of the present disclosure. In the following embodiments, the various operations can be performed sequentially, but not necessarily. For example, the order of the various operations can be changed, and at least two operations can be performed in parallel. As an example, the first electronic device can be ​ The electronic device 101 or ​ The second electronic device 210 is shown.

[0250] Reference ​ According to various embodiments, in method 2100, in operation 2101, a second electronic device (eg, ​ The processor 120 or the communication module 190 in ​ The processor 400 or the communication circuit 410 in the embodiment may communicate with the first electronic device (eg, ​ According to an embodiment, if the user authentication for establishing a communication link between the first electronic device 200 and the second electronic device 210 is successful, the processor 400 may control the communication circuit 410 to establish a communication link (or communication channel) with the first electronic device 200. For example, the communication link (or communication channel) may be established based on a short-range communication scheme such as Bluetooth, BLE, or a wireless LAN (e.g., Wi-Fi direct).

[0251] According to various embodiments, in operation 2103, a second electronic device (e.g., processor 120 or 400) may confirm whether input information is detected. As an example, the input information may include at least one of user input information detected by the input device 420 and / or an operation state (e.g., motion information) of the second electronic device 210 detected by the sensor module 430.

[0252] According to various embodiments, if no input information is detected (e.g., "No" in operation 2103), the second electronic device (e.g., processor 120 or 400) may confirm whether input information is detected.

[0253] According to various embodiments, if input information is detected (e.g., "Yes" in operation 2103), then in operation 2105, the second electronic device (e.g., processor 120 or 400) may determine whether an input processing method of the second electronic device (e.g., ​ the second electronic device 210 in ) is configured as a first method. As an example, the first method may include an input processing method for sharing input information with the first electronic device 200.

[0254] According to various embodiments, if the first method is configured (e.g., "Yes" in operation 2105), then in operation 2107, the second electronic device (e.g., processor 120 or 400) may send the input information to the first electronic device (e.g., ​ the first electronic device 200 in ) that performs an interactive operation for an augmented reality service. As an example, the input information sent to the first electronic device 200 may include command information (e.g., tap, multiple taps, drag, long touch, or double tap) corresponding to the input detected in the second electronic device 210 or row data detected for identifying command information.

[0255] According to various embodiments, if the first method is not configured (e.g., "No" in operation 2105), then in operation 2109, the second electronic device (e.g., processor 120 or 400) may execute a function (or operation) corresponding to the input information. According to an embodiment, if, as ​ shown, a click input 1120 is detected, the processor 400 may control the display device 440 to enlarge and display an image object selected by the click input 1120 (see 1130), as ​ shown.

[0256] ​is a flowchart of controlling a second electronic device in interaction with a first electronic device according to an embodiment of the present disclosure. In the following embodiments, each operation may be performed sequentially, but it is not necessary to be performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. As an example, the first electronic device may be ​ the electronic device 101 shown in ​ or the second electronic device 210 shown in ​ described ​ at least some of the elements shown in

[0257] ​ shows an example of changing a display method in a second electronic device based on an interaction operation with a first electronic device according to an embodiment of the present disclosure.

[0258] ​ shows an example of changing a display method in a second electronic device based on an interaction operation with a first electronic device according to an embodiment of the present disclosure.

[0259] ​ shows an example of changing content to be displayed on a display device in a second electronic device based on an interaction operation with a first electronic device according to an embodiment of the present disclosure.

[0260] ​ shows an example of changing content to be displayed on a display device in a second electronic device based on an interaction operation with a first electronic device according to an embodiment of the present disclosure.

[0261] Referring to ​ , according to various embodiments, in method 2200, in operation 2201, the second electronic device (e.g., ​ the processor 120 or the communication module 190 in ​ , or the processor 400 or the communication circuit 410 in ​ ) may confirm whether it has received interaction operation information from a first electronic device (e.g., ​ the first electronic device 200 in) connected through an established communication link (or communication channel). According to an embodiment, as described in operation 501 or 507 in

[0262] , if the first electronic device 200 determines to interact with the second electronic device 210 for an augmented reality service, the first electronic device 200 may send interaction operation information related to the augmented reality service to the second electronic device 210. Accordingly, the processor 400 may confirm whether it has received, through the communication circuit 410, interaction operation information related to the augmented reality service from the first electronic device 200.

[0262] According to various embodiments, if from the first electronic device (e.g.,​ If the first electronic device 200 in ​ receives interaction operation information (e.g., "Yes" in operation 2201), then in operation 2203, the second electronic device (e.g., the processor 120 or 400) may switch the input processing method of the second electronic device (e.g.,

[0263] the second electronic device 210 in ​ to the first method. As an example, the first method may include an input processing method for sharing the input in the second electronic device 210 with the first electronic device 200 for interaction operations of an augmented reality service.

[0264] Referring to ​ , the display device 440 may display an execution screen (e.g., an image list) 2300 of an application (e.g., a gallery application) executed in the second electronic device 210.

[0265] Referring to ​ , if the input processing method of the second electronic device 201 is switched to the first method, the processor 400 may control the display device 440 such that the brightness of the image list 2310 of the display device 440 is relatively reduced. For example, if the input processing method of the second electronic device 201 is switched from the first method to the second method, the processor 400 may control the display device 440 such that the brightness of the display device 440 is restored, as ​ shown.

[0266] Referring to ​ , the display device 440 may display a standby screen 2400 of the second electronic device 210.

[0267] Referring to ​ , if the input processing method of the second electronic device 201 is switched to the first method, the processor 400 may control the display device 440 to display content 2410 (e.g., game information) corresponding to the augmented reality service. For example, if the input processing method of the second electronic device 201 is switched from the first input processing method to the second method, the processor 400 may control the display device 440 to display as​ The standby screen 2400 shown.

[0268] According to various embodiments, if no interaction operation information (e.g., "No" in operation 2201) is received from the first electronic device (e.g., ​ the first electronic device 200 in ​ ), then in operation 2207, the second electronic device (e.g., the processor 120 or 400) may maintain the second method of the second electronic device (e.g.,

[0269] ​ is a flowchart for determining an interaction operation with a first electronic device in a second electronic device according to an embodiment of the present disclosure. In the following embodiments, the respective operations may be executed sequentially, but it is not necessary to execute them sequentially. For example, the order of the respective operations may be changed, and at least two operations may be executed in parallel. As an example, the first electronic device may be ​ the electronic device 101 shown in ​ or the second electronic device 210 shown in

[0270] Referring to ​ , according to various embodiments, in method 2500, in operation 2501, the second electronic device (e.g., ​ the processor 120 or the communication module 190 in ​ , or ​ the processor 400 or the communication circuit 410 in

[0271] According to various embodiments, in operation 2503, the second electronic device (e.g., the processor 120 or 400) may determine whether it is for an augmented reality service with the first electronic device (e.g.,Figure 2 interact with the first electronic device 200 therein. According to an embodiment, the processor 400 may determine whether to interact with the first electronic device based on at least one of the dwell time of the user's gaze corresponding to an object included in a region corresponding to the field of view of the user wearing the first electronic device 200, whether an object related to the second electronic device 210 is included in a region corresponding to the field of view of the user wearing the first electronic device 200, or the user gaze information corresponding to the object related to the second electronic device 210. According to an embodiment, based on a comparison result of position information between the first electronic device 200 and the second electronic device 210, the processor 400 may determine whether to interact with the first electronic device 200 for an augmented reality service.

[0272] According to various embodiments, if the second electronic device (e.g., the processor 120 or 400) determines to interact with the first electronic device (e.g., Figure 2 the first electronic device 200 therein) for an augmented reality service (e.g., "yes" in operation 2503), then in operation 2505, the second electronic device may switch the input processing method of the second electronic device (e.g., Figure 2 the second electronic device 210 therein) to a first method.

[0273] According to various embodiments, in operation 2507, the second electronic device (e.g., the processor 120 or 400) may change the display information and / or display method of the display device (e.g., Figure 4 the display device 440 therein) based on the switch to the first method. According to an embodiment, if the input processing method of the second electronic device 210 is switched to the first method, the processor 400 may control the display device 440 to switch to a deactivated mode so that the user can focus on information related to the augmented reality service.

[0274] According to various embodiments, if the second electronic device (e.g., the processor 120 or 400) determines not to interact with the first electronic device (e.g., Figure 2 the first electronic device 200 therein) for an augmented reality service (e.g., "no" in operation 2503), then in operation 2509, the second electronic device may maintain the second method of the second electronic device (e.g., Figure 2 the second electronic device 210 therein).

[0275] According to various embodiments, when input information is detected, the second electronic device 210 may configure an input processing method of the second electronic device 210 based on information about the gaze of a user wearing the first electronic device 200. According to an embodiment, if input information is detected, the second electronic device 210 may confirm information about the gaze of the user wearing the first electronic device 200. If it is determined that the gaze of the user wearing the first electronic device 200 points to an active object, the second electronic device 210 may configure the input processing method of the second electronic device 210 as a first method. As an example, the second electronic device 210 may confirm information about the gaze of the user wearing the first electronic device 200 based on information related to an augmented reality service periodically received from the first electronic device 200. As an example, if input information is detected, the second electronic device 210 may send a request for user gaze information to the first electronic device 200 to confirm the user gaze information of the user wearing the first electronic device 200.

[0276] Figure 26 is a flowchart for providing an IoT service in a first electronic device according to an embodiment of the present disclosure. In the following embodiments, each operation may be executed sequentially, but it is not necessary to execute sequentially. For example, the order of each operation may be changed, and at least two operations may be executed in parallel. As an example, the first electronic device may be Figure 1 the electronic device 101 shown or Figure 3 the first electronic device 200 shown. As an example, the first electronic device may be Figure 1 the electronic device 101 shown or Figure 3 the first electronic device 200 shown. As an example, reference will be made to Figure 27A and Figure 27B to describe Figure 26 at least some of the elements shown therein.

[0277] Figure 27A shows a screen configuration of a first electronic device for registering an IoT device according to an embodiment of the present disclosure.

[0278] Figure 27B shows a screen configuration of a first electronic device for registering an IoT device according to an embodiment of the present disclosure.

[0279] Referring to Figure 26 , according to various embodiments, in method 2600, in operation 2601, the first electronic device (e.g., Figure 1 the processor 120 or the communication module 190 in Figure 3 , or the processor 300 or the communication circuit 330 in Figure 2The second electronic device 210) in establishes a communication link (or communication channel). As an example, the user identification information may be the same between the first electronic device 200 and the second electronic device 210. As an example, the first electronic device 200 and the second electronic device 210 may successfully authenticate the user associated with the establishment of the communication link.

[0280] According to various embodiments, in operation 2603, the first electronic device (e.g., the processor 120 or 300) may output information related to the augmented reality service. According to an embodiment, the processor 300 may control the display device 320 to display at least one virtual object in an area corresponding to the field of view of the user wearing the first electronic device 200.

[0281] Referring to Figure 27A , the user wearing the first electronic device 200 may identify an object such as the IoT device 2710 related to the augmented reality service through the area 2700 corresponding to the field of view of the user wearing the first electronic device 200.

[0282] According to various embodiments, in operation 2605, the first electronic device (e.g., the processor 120 or 300) may confirm whether an unregistered device is recognized in an area corresponding to the field of view of the user wearing the first electronic device (e.g., Figure 2 the first electronic device 200) in. According to an embodiment, the processor 300 may confirm whether an object related to the IoT device exists in an area corresponding to the field of view of the user wearing the first electronic device 200 through at least one of the camera module 310 or the sensor module 340 (e.g., a spatial sensor). If an object related to the IoT device 2710 is detected in an area corresponding to the field of view of the user wearing the first electronic device 200, the processor 300 may confirm whether the IoT device 2710 is registered in the first electronic device 200. As an example, the processor 300 may confirm whether the registration information about the IoT device 2710 is stored in the memory (e.g., Figure 1 the memory 130) in.

[0283] According to various embodiments, if no unregistered device is recognized (e.g., "No" in operation 2605), the first electronic device (e.g., the processor 120 or 300) may provide the augmented reality service.

[0284] According to various embodiments, if an unregistered device is recognized (e.g., "Yes" in operation 2605), then in operation 2607, the first electronic device (e.g., the processor 120 or 300) may send information related to the unregistered device to the second electronic device (e.g., Figure 2the second electronic device 210 in). As an example, information related to an unregistered device may include a device type obtained through image analysis (e.g., a television set) and / or information about the location of the first electronic device 200 (e.g., a living room).

[0285] According to various embodiments, in operation 2609, a first electronic device (e.g., the processor 120 or 300) may confirm whether a registration request signal is received from a second electronic device (e.g., Figure 2 the second electronic device 210 in). According to an embodiment, the second electronic device 210 may confirm an unregistered device identified by the first electronic device 200 based on information related to the unregistered device. The second electronic device 210 may confirm whether the unregistered IoT device 2710 of the first electronic device 200 is registered in the second electronic device 210. If the unregistered device 2710 of the first electronic device 200 is not even registered in the second electronic device 210, the second electronic device 210 may send a registration request signal to the first electronic device 200. Accordingly, the processor 300 may confirm whether a registration request signal is received from the second electronic device 210 through the communication circuit 410.

[0286] According to various embodiments, if a first electronic device (e.g., the processor 120 or 300) receives a registration request signal from a second electronic device (e.g., Figure 2 the second electronic device 210 in) (e.g., “Yes” in operation 2609), then in operation 2611, the first electronic device may display a registration screen related to the unregistered device.

[0287] Referring to Figure 27B , if a registration request signal is received from the second electronic device 210, the processor may control the display device 320 to display a registration screen 2720 corresponding to the object 2710 related to the IoT device.

[0288] According to various embodiments, in operation 2613, a first electronic device (e.g., the processor 1@0 or 300) may determine whether to register an unregistered device in the first electronic device (e.g., Figure 2 the first electronic device 200 in) based on an input corresponding to the registration screen related to the unregistered device.

[0289] According to various embodiments, if a first electronic device (e.g., the processor 120 or 300) determines that an unregistered device is registered in the first electronic device (e.g., Figure 2 the first electronic device 200 in) (e.g., “Yes” in operation 2613), then in operation 2615, the first electronic device may register the unregistered device in the first electronic device (e.g., Figure 2register information related to registered and unregistered devices in (the first electronic device). According to an embodiment, if an input for selecting the menu "Yes" 2722 in the registration screen 2720 is detected in Figure 27B Figure 27B , the processor 300 may determine to register the IoT device 2710. In this case, the processor 300 may register information related to the IoT device 2710 in the memory of the first electronic device 200 (e.g., Figure 1 the memory 130 in Figure 1 ). As an example, if the IoT device 2710 is detected through the camera module 310, the processor 300 may control the display device 320 to display a virtual object related to the IoT device (e.g., a control menu or status information). According to an embodiment, if an input for selecting the menu "No" 2724 in the registration screen 2720 is detected in Figure 27B Figure 27B , the processor 300 does not register the suggestion.

[0290] According to various embodiments, if a registration request signal (e.g., "No" in operation 2609) is not received from the second electronic device (e.g., Figure 2 the second electronic device 210 in Figure 2 ), in operation 2617, the first electronic device (e.g., the processor 120 or 300) may confirm whether registration information related to an unregistered device is received. According to an embodiment, if the unregistered device 2710 of the first electronic device 200 is registered in the second electronic device 210, the second electronic device 210 may send the registration information related to the unregistered device 2710 to the first electronic device 200. Accordingly, the processor 300 may determine whether registration information related to the unregistered device 2710 is received from the second electronic device 210 through the communication circuit 410.

[0291] According to various embodiments, if registration information related to an unregistered device is not received (e.g., "No" in operation 2617), the first electronic device (e.g., the processor 120 or 300) may determine whether a registration request signal related to the unregistered device is received.

[0292] According to various embodiments, if registration information related to an unregistered device is received (e.g., "Yes" in operation 2617), then in operation 2619, the first electronic device (e.g., processor 120 or 300) may display information related to the unregistered device. According to an embodiment, if registration information related to the IoT device 2710 is received from the second electronic device 210, the processor 300 may register the IoT device 2710 at the location of the first electronic device 200 (e.g., the living room). As an example, if the IoT device 2710 is detected through the camera module 310, the processor 300 may control the display device 320 to display a virtual object related to the IoT device (e.g., a control menu or status information).

[0293] Figure 28 FIG. is a flowchart of a second electronic device providing an IoT service based on a first electronic device according to an embodiment of the present disclosure. In the following embodiments, the respective operations may be performed sequentially, but it is not necessary to perform them sequentially. For example, the order of the respective operations may be changed, and at least two operations may be performed in parallel. As an example, the first electronic device may be Figure 1 the electronic device 101 shown in Figure 4 or the second electronic device 210 shown in

[0294] Referring to Figure 28 , according to various embodiments, in method 2800, in operation 2801, the second electronic device (e.g., Figure 1 the processor 120 or the communication module 190 in Figure 4 or the processor 400 or the communication circuit 410 in Figure 2 ) may establish a communication link (or communication channel) with the first electronic device (e.g., Figure 2 the first electronic device 200 in Figure 2 ). According to an embodiment, the first electronic device 200 and the second electronic device 210 may establish a communication link (or communication channel) based on a short-range communication scheme such as Bluetooth, BLE, or wireless LAN (e.g., Wi-Fi Direct).

[0295] According to various embodiments, in operation 2803, the second electronic device (e.g., the processor 120 or 400) may confirm whether it has received information related to an unregistered device from the first electronic device (e.g., Figure 2 the first electronic device 200 in Figure 2 ) connected through the established communication link (or communication channel).

[0296] According to various embodiments, if the second electronic device (e.g., the processor 120 or 400) receives information from the first electronic device (e.g., Figure 2If the first electronic device 200) receives information related to an unregistered device (e.g., "Yes" in operation 2803), then in operation 2805, the second electronic device can determine whether the unregistered device is registered in the second electronic device (e.g., Figure 2 the second electronic device 210) in. According to an embodiment, the processor 400 can confirm the type of the unregistered device identified by the first electronic device 200 and information about the location of the first electronic device 200 based on the information related to the unregistered device provided by the first electronic device 200. The processor 400 can confirm whether the unregistered device of the first electronic device 200 is included in the device list registered at the location of the first electronic device 200.

[0297] According to various embodiments, if the unregistered device is not registered in the second electronic device (e.g., Figure 2 the second electronic device 210) in (e.g., "No" in operation 2805), then in operation 2807, the second electronic device (e.g., the processor 120 or 400) can send a registration request signal corresponding to the unregistered device to the first electronic device (e.g., Figure 2 the first electronic device 200) in.

[0298] According to various embodiments, in operation 2809, the second electronic device (e.g., the processor 120 or 400) can confirm whether it has received registration information related to the unregistered device from the first electronic device (e.g., Figure 2 the first electronic device 200) in.

[0299] According to various embodiments, if the second electronic device (e.g., the processor 120 or 400) receives registration information related to the unregistered device from the first electronic device (e.g., Figure 2 the first electronic device 200) in (e.g., "Yes" in operation 2809), then in operation 2811, the second electronic device can register the unregistered device in the second electronic device (e.g., Figure 2 the second electronic device 210) in. According to an embodiment, if registration information related to an IoT device that has not yet been registered is received from the first electronic device 200, the processor 400 can register the IoT device (e.g., a television) at the location of the first electronic device 200 (e.g., the living room).

[0300] According to various embodiments, if the unregistered device is registered in the second electronic device (e.g., Figure 2 the second electronic device 210) in (e.g., "Yes" in operation 2805), then in operation 2813, the second electronic device (e.g., the processor 120 or 400) can send information related to the unregistered device to the first electronic device (e.g., Figure 2in the first electronic device 200).

[0301] According to various embodiments, if the second electronic device 210 receives information related to an unregistered device, the second electronic device 210 may confirm the information related to the unregistered device. According to an embodiment, if information related to an unregistered device is received from the first electronic device 200 through the communication circuit 410, the processor 400 may confirm information about the unregistered device through the memory of the second electronic device 210 (e.g., Figure 1 the memory 130) or through an external server (e.g., Figure 1 the server 108 in). The processor 400 may control the communication circuit 410 to send information about the unregistered device to the first electronic device 200. According to an embodiment, if information about the unregistered device is received from the second electronic device 210, the first electronic device 200 may control the display device 320 to display the information about the unregistered device.

[0302] According to various embodiments, a first electronic device (e.g., an electronic device providing an augmented reality service) may selectively interoperate with a second electronic device (e.g., a mobile device) based on at least one of information about an object included in an area corresponding to a user's field of view or information about the position of the first electronic device, thereby providing various control (or manipulation) methods corresponding to the augmented reality service.

[0303] According to various embodiments, a second electronic device (e.g., a mobile device) may selectively provide input information obtained in the second electronic device to a first electronic device (e.g., an electronic device providing an augmented reality service) based on an input processing method corresponding to interoperability information or non-interoperability information with the first electronic device (e.g., an electronic device providing an augmented reality service) for an augmented reality service, so that a user can easily control (or manipulate) the augmented reality service.

[0304] The embodiments disclosed in the specification and drawings only show specific examples to easily describe the subject matter according to the embodiments and help understand the embodiments, and are not intended to limit the scope of the embodiments. Therefore, the scope of various embodiments should be interpreted to include all changes or modifications based on the subject matter of various embodiments and the embodiments disclosed herein.

[0305] Although the present invention has been shown and described with reference to various embodiments of the present invention, those skilled in the art will understand that various changes in form and details may be made therein without departing from the spirit and scope of the present invention defined by the appended claims and their equivalents.

Claims

1. A first electronic device, the first electronic device comprising: A communication circuit; A camera; A sensor; A display; A processor operably connected to the communication circuit, the display, the sensor, and the camera; And A memory storing instructions, Wherein, when the instructions are run by the processor, the first electronic device is caused to: Establish communication with a second electronic device through the communication circuit, Obtain information about at least one object corresponding to the field of view of a user wearing the first electronic device through at least one of the camera or the sensor, Obtain information about the user's gaze, Based on the information about the user's gaze and the information about the at least one object, determine whether to use the second electronic device as an input device for an augmented reality service, and When it is determined to use the second electronic device as an input device for the augmented reality service, send information related to the input device for the augmented reality service to the second electronic device through the communication circuit.

2. The first electronic device according to claim 1, wherein, The processor is further configured to obtain information about the user's gaze based on at least one of head gaze or eye gaze of the first electronic device.

3. The first electronic device according to claim 1, wherein, The processor is further configured to control the display to display at least one virtual object on at least a part of an area corresponding to the field of view of a user wearing the first electronic device.

4. The first electronic device according to claim 3, wherein, The processor is further configured to: When the user's gaze corresponding to any one of at least one object corresponding to the field of view of a user wearing the first electronic device or the at least one virtual object has continuously maintained a reference time, determine to use the second electronic device as the input device for the augmented reality service.

5. The first electronic device according to claim 1, wherein, The processor is further configured to: When an object related to the second electronic device exists in an area corresponding to the field of view of a user wearing the first electronic device, confirm whether the user's gaze points to the object related to the second electronic device, and When the user's gaze points to an object other than the object related to the second electronic device, determine to use the second electronic device as the input device for the augmented reality service.

6. The first electronic device according to claim 1, wherein, The processor is further configured to: When it is determined to use the second electronic device as an input device for the augmented reality service, confirm whether input information is received from the second electronic device through the communication circuit, and When the input information is received from the second electronic device, perform an operation corresponding to the input information.

7. The first electronic device according to claim 6, wherein, The input information received from the second electronic device includes at least one of a touch input, a physical button input, or information about the movement of the second electronic device.

8. The first electronic device according to claim 6, wherein, The processor is further configured to: When the input information is received from the second electronic device, confirm the object corresponding to the user's gaze, and Control the display to display a control menu related to the object corresponding to the user's gaze.

9. A second electronic device, the second electronic device comprising: A communication circuit; An input unit; And A processor operably connected to the communication circuit and the input unit; And A memory storing instructions, Wherein, when the instructions are run by the processor, the second electronic device is caused to: Establish communication with a first electronic device through the communication circuit, When information related to an input device of an augmented reality service is received from the first electronic device through the communication circuit, configure the input processing method of the second electronic device as a first method, and When information related to the input device of the augmented reality service is not received from the first electronic device through the communication circuit, or when non-interactive operation information is received from the first electronic device through the communication circuit, configure the input processing method of the second electronic device as a second method, When a user input is detected through the input unit, confirm the input processing method of the second electronic device, When the input processing method is the first method for sharing input, send information about the user input to the first electronic device through the communication circuit, and When the input processing method is the second method with limited input sharing, perform an operation corresponding to the information about the user input.

10. The second electronic device according to claim 9, wherein the second electronic device further comprises: A display, Wherein the processor is further configured to, when information related to the input device of the augmented reality service is received from the first electronic device through the communication circuit, control the display to perform one of changing display information or changing a display method of the display.

11. The second electronic device according to claim 10, wherein, The processor is further configured to, when the input processing method of the second electronic device is the first method, control the display to display content corresponding to the augmented reality service.

12. The second electronic device according to claim 11, wherein, The processor is further configured to, when the input processing method of the second electronic device is the second method, control the display to display a standby screen.

13. The second electronic device according to claim 10, wherein, The processor is further configured to, when the input processing method of the second electronic device is the first method, adjust the brightness of the display based on the first method.

14. The second electronic device according to claim 9, wherein, The input unit includes at least one of a touch circuit or a physical button.

Images