Electronic device and method for displaying an object associated with an external electronic device based on the position and movement of the external electronic device

Abstract

According to various embodiments of the present disclosure, an electronic device includes one or more cameras, a display, a communication circuit, and a processor having a specified field of view. The processor may be configured to: confirm an external electronic device among one or more external objects included in the specified field of view through the camera, display a graphic object corresponding to the external electronic device through the display based on first position information of the external electronic device confirmed at least based on image information obtained through the camera, and if the external electronic device is outside the specified field of view, display the graphic object through the display based on second position information of the external electronic device confirmed by the camera before the external electronic device moves outside the specified field of view and information related to the movement of the external electronic device received from the external electronic device via the communication circuit after the external electronic device moves outside the specified field of view.

Description

Technical Field

[0001] Various embodiments relate to an electronic device and method for displaying an object related to the external electronic device using information about the external electronic device. Background Art

[0002] Due to the development of technology, various electronic devices for providing services related to virtual reality (VR) have been provided. The various electronic devices include electronic devices for displaying multimedia content related to virtual reality and external electronic devices for user input in multimedia content related to virtual reality. Summary of the invention

[0003] Technical issues

[0004] A user who receives a service related to virtual reality via an electronic device has a limited field of view (or viewing range). Due to the limitation of the field of view, a separate external electronic device is provided for displaying user input in the service related to VR. In the electronic device, a method for naturally displaying an external electronic device is required.

[0005] Technical problems to be achieved in this document are not limited to the above-mentioned technical problems, and other technical problems not mentioned are clearly understood by those skilled in the art from the following description.

[0006] Solution to the problem

[0007] According to various embodiments of the present disclosure, an electronic device includes one or more cameras with a specified field of view, a display, a communication circuit, and a processor, wherein the processor uses the camera to identify an external electronic device among one or more external objects included in the specified field of view, displays a graphic (visual) object corresponding to the external electronic device on the display based on first position information of the external electronic device identified at least based on image information obtained by the camera, and when the external electronic device deviates from the specified field of view, displays the graphic object on the display based on second position information of the external electronic device identified by the camera before the external electronic device deviates from the specified field of view and information related to the movement of the external electronic device received from the external electronic device through the communication circuit after the external electronic device deviates from the specified field of view.

[0008] The method according to various embodiments of the present disclosure includes the following operations: using one or more cameras with a specified field of view to identify an external electronic device among one or more external objects included in the specified field of view, displaying a graphic object corresponding to the external electronic device through a display based on first position information of the external electronic device identified at least based on image information obtained by the camera, and when the external electronic device deviates from the specified field of view, displaying the graphic object on the display based on second position information of the external electronic device identified by the camera before the external electronic device deviates from the specified field of view and information related to the movement of the external electronic device received from the external electronic device through a communication circuit after the external electronic device deviates from the specified field of view.

[0009] Advantageous Effects of the Invention

[0010] An electronic device and method according to various embodiments of the present disclosure provide an enhanced user experience (UX) by displaying a graphic object representing an external electronic device and multimedia content for providing virtual reality (VR) using at least one sensor included in the electronic device and at least one sensor of an external electronic device interlocked with the electronic device.

[0011] Effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned are clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an electronic device that displays an object related to an external electronic device based on a location and movement of the external electronic device in a network environment according to various embodiments.

[0013] Figure 2 Examples of functional configurations of the electronic device and external electronic devices according to various embodiments are shown.

[0014] Figure 3 Examples of functional configurations of the electronic device, the first external electronic device, and the second external electronic device according to various embodiments are shown.

[0015] Figure 4a An example of a case where an external electronic device deviates from the field of view of a camera of the electronic device according to various embodiments is shown.

[0016] Figure 4b Examples for comparing a camera's field of view and a user's field of view are shown according to various embodiments.

[0017] Figure 5a Examples of operations of electronic devices according to various embodiments are shown.

[0018] Figure 5b Examples of operations of electronic devices according to various embodiments are shown.

[0019] Figure 6 Examples of an external electronic device and a graphic object displayed corresponding to the external electronic device according to various embodiments are illustrated.

[0020] Figure 7 A time flow of a display displayed according to an operation of an electronic device according to various embodiments is shown.

[0021] Figure 8 An example for comparing the effect of correction using sensor data according to various embodiments is shown.

[0022] Figure 9a Examples of operations of electronic devices according to various embodiments are shown.

[0023] Figure 9b Examples of operations of electronic devices according to various embodiments are shown.

[0024] Fig.10 An example of a graphic object displayed on a display of an electronic device for guiding a location of an external electronic device according to various embodiments is illustrated. DETAILED DESCRIPTION

[0025] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.

[0026] Reference Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit.For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented to be embedded in the display device 160 (eg, a display).

[0027] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 coupled to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may load a command or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that can operate independently or in combination with the main processor 121. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a specified function. The auxiliary processor 123 may be implemented separately from the main processor 121 , or may be implemented as a part of the main processor 121 .

[0028] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) among the components of the electronic device 101, or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 180 or the communication module 190) that is functionally related to the auxiliary processor 123.

[0029] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0030] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0031] The input device 150 may receive a command or data to be used by another component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., the user). The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus).

[0032] The sound output device 155 can output a sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or playing records, and the receiver may be used for incoming calls. Depending on the embodiment, the receiver may be implemented as a separate part from the speaker, or the receiver may be implemented as a part of the speaker.

[0033] The display device 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0034] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0035] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0036] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0037] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0039] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0040] The power management module 188 may manage power supply to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as, for example, at least a portion of a power management integrated circuit (PMIC).

[0041] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate via a first network 198 (e.g., a short-range communication network such as Bluetooth TM The wireless communication module 192 may communicate with an external electronic device through a wireless communication network 198 (e.g., a wireless fidelity (Wi-Fi) direct connection, or an infrared data association (IrDA)) or a second network 199 (e.g., a long-distance communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as a first network 198 or a second network 199) using user information (e.g., an international mobile subscriber identity (IMSI)) stored in the user identification module 196.

[0043] The antenna module 197 may transmit a signal or power to the outside of the electronic device 101 (e.g., an external electronic device) or receive a signal or power from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna, wherein the antenna includes a radiation element composed of a conductive material or a conductive pattern formed in a substrate (e.g., a printed circuit board (PCB)) or on a substrate (e.g., a PCB). According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). The signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiation element may be additionally formed as a part of the antenna module 197.

[0044] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0045] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed at the electronic device 101 may be executed at one or more of the external electronic devices 102, 104, or 108. For example, if the electronic device 101 should automatically perform a function or service or should perform a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to perform at least a part of the function or service instead of running the function or service, or the electronic device 101 may request the one or more external electronic devices to perform at least a part of the function or service in addition to running the function or service. The one or more external electronic devices that receive the request may perform at least a part of the function or service requested, or perform another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request under the condition that the result is further processed or under the condition that the result is not further processed. To this end, for example, cloud computing technology, distributed computing technology or client-server computing technology may be used.

[0046] Figure 2 Examples of functional configurations of the electronic device and external electronic devices according to various embodiments are shown. Figure 2 At least a part of the functional configuration of the electronic device 201 and the external electronic device 202 is included in Figure 1 The electronic device 101 shown in Figure 1 The functional configuration of the electronic device 102 is shown in .

[0047] Reference Figure 2 , environment 200 includes an electronic device 201 and an external electronic device 202 .

[0048] The electronic device 201 interlocks with the external electronic device 202 and is configured to provide a user with content related to virtual reality (VR). The electronic device 201 includes a memory 210, a communication module 220, a display 230, a camera 240 or a processor 250.

[0049] In various embodiments, memory 210 includes Figure 1 The memory 130 shown in FIG. 1 and the communication module 220 include Figure 1 The communication module 190 shown in FIG. 1 and the display 230 include Figure 1The display device 160 shown in FIG. 1 , the camera 240 includes Figure 1 The camera 180 shown in FIG. 1 and the processor 250 include Figure 1 The processor 120 shown in .

[0050] In various embodiments, the processor 250 (eg, the processor 120 ) is operably or manipulably coupled to at least one of the memory 210 , the communication module 220 , and the display 230 .

[0051] In various embodiments, the processor 250 uses the camera 240 to acquire information about the location of the external electronic device 202 interlocked with the electronic device 201 .

[0052] In one embodiment, the processor 250 uses the camera 240 to acquire an image of the external environment of the electronic device 201. The image of the external environment includes images related to a plurality of external objects including the external electronic device 202. In one embodiment, the camera 240 is visually exposed through at least a portion of a first surface (first side) of a housing for mounting components of the electronic device 201 (e.g., the memory 210, the communication module 220, the display 230, or the processor 250). The housing includes a second surface on the opposite side of the first surface, and the display 230 is visually exposed through at least a portion of the second surface.

[0053] In one embodiment, the camera 240 has a specified field of view. The field of view is referred to by various terms such as field of view, viewing angle, viewing angle, FOV (field of view), horizontal FOV (hFOV), and vertical FOV (vFOV). For example, the camera 240 acquires an image corresponding to an area of ​​a predetermined size. The area of ​​the predetermined size includes an area formed at a predetermined angle relative to the upper side, the lower side, the left side, and the right side. For example, referring to Figure 6 , the area of ​​the predetermined size corresponds to the area included in the image 610. The user's field of view 401 corresponds to the area of ​​the field of view (FOV) displayed on the display. The camera field of view 403 is determined based on the capability of the lens included in the camera 240. For example, the camera 240 includes a wide-angle lens, and the camera field of view 403 has a large value angle corresponding to the wide-angle lens. Figure 4b , the camera field of view 403 corresponds to areas 412, 414 and 416 indicated by dashed lines.

[0054] In another embodiment, the camera 240 acquires images for an area of ​​a predefined size. The size of the area is different based on the field of view of the camera 240. For example, if the camera 240 corresponds to a wide-angle lens camera, the size of the area is increased. Figure 4b, the camera field of view (FOV) corresponds to the user field of view 401 and corresponds to a field of view 403 that is larger than the image area 410 displayed on the display.

[0055] Hereinafter, the image 410 displayed in the display 230 includes an image corresponding to the field of view of the user. In various embodiments, the field of view of the camera 240 is the same as the field of view of the display area of ​​the image 410 displayed on the display 230. However, the field of view of the camera 240 is not limited thereto, and as Figure 4b As shown in , it is smaller or larger than the user's field of view.

[0056] In another embodiment, the processor 250 is linked or interlocked with the external electronic device 202. For example, the interlocking includes a connection established by a wired manner or a wireless manner. The interlocking is performed by the communication module 220 included in the electronic device 201.

[0057] In another embodiment, the processor 250 detects the external electronic device 202 by detecting a signal output from the external electronic device 202. For example, the external electronic device 202 further includes a light emitting portion. The light emitting portion includes at least one light emitting diode (LED). For example, the external electronic device 202 outputs light corresponding to an illumination exceeding or equal to a threshold range (e.g., a threshold) through the LED. The processor 250 of the electronic device 201 identifies the external electronic device 202 by receiving light corresponding to an illumination exceeding or equal to the threshold. In one embodiment, when the processor 250 detects a signal output from the LED of the external electronic device in an image acquired by the camera 240, it is determined that the external electronic device 202 is located within the field of view of the camera 240. For another example, if the processor 250 does not detect a signal output from the LED of the external electronic device in the image acquired by the camera 240, it is determined that the external electronic device 202 is not located within the field of view of the camera 240. That is, when an area corresponding to an illumination exceeding the threshold is not detected in the acquired image, the processor 250 determines that the external electronic device 202 deviates from the field of view of the camera 240.

[0058] According to an embodiment, an LED included in the external electronic device 202 is turned on or off at a predetermined time interval. The electronic device 201 identifies information about the predetermined time interval for turning on or off the LED included in the external electronic device 202 by identifying that the external electronic device 202 is turned on or off at a predetermined time interval to detect the external electronic device 202. For example, the external electronic device 202 controls the LED to be turned on or off repeatedly every 0.1 seconds. When the external electronic device 201 controls the LED to be turned on or off at a predetermined time interval, the external electronic device 202 sends cycle information for providing the predetermined time interval to the electronic device 201. The electronic device 201 receives the cycle information and obtains information about the predetermined time interval for turning on or off the LED included in the external electronic device 202. The electronic device 201 detects the external electronic device 202 by identifying the LED turned on or off in the same manner as the predetermined time interval based on the cycle information. In one embodiment, the predetermined time interval is randomly determined. For example, the predetermined time interval is determined based on the unique ID of the external electronic device 202. Since the on or off time interval is determined based on the unique ID of the external electronic device 202 interlocked with the electronic device 201, even if the electronic device 201 identifies through the camera 240 an LED output from another external electronic device interlocked with another electronic device (for example, a VR controller of another user), the electronic device 201 accurately identifies the LED output from the external device 202 interlocked with the electronic device 201.

[0059] According to an embodiment, the external electronic device 202 measures the brightness of the external environment through an illumination sensor (not shown) included in the external electronic device 202. When the brightness of the external environment is greater than or equal to a predetermined value, the external electronic device 202 changes the LED output intensity. For example, the external electronic device 202 increases the LED output intensity in response to detecting that the measured brightness of the external environment is greater than a predetermined threshold. The external electronic device 202 is more easily recognized by the electronic device 201 by increasing the LED output intensity. The external electronic device 202 turns on or off the LED according to a predetermined time interval based on the increased LED output intensity.

[0060] In another embodiment, the electronic device 201 acquires information about the location of the external electronic device 202. The location information includes information determined based on an image obtained from the camera 240 of the external electronic device 202. For example, the location information includes information about the coordinates at which the external electronic device 202 is located in the acquired image. The location information includes an x-coordinate value, a y-coordinate value, or a z-coordinate value. The information about the coordinates is acquired by a location tracking module 252 included in the processor 250.

[0061] In another embodiment, the processor 250 includes a position tracking module 252. The position tracking module 252 obtains information about the position of the external electronic device 202 included in the image acquired by the camera 240. For example, the position tracking module 252 divides the acquired image into coordinates equally spaced on the x-axis and y-axis, and obtains an x-coordinate or a y-coordinate corresponding to the area where the external electronic device 202 is detected. However, in various embodiments, the x-axis coordinate and the y-axis coordinate are not limited to being obtained based on the division of the same interval, but are also obtained based on a technique for obtaining information about multiple positions. For another example, the position tracking module 252 obtains depth information based on the acquired image. The depth information is obtained based on an image operation for calculating depth information from a two-dimensional image. According to another embodiment, the position tracking module 252 obtains depth information based on a difference value for two images. For example, the electronic device 201 includes two cameras, and the two cameras respectively acquire images of the external electronic device 202. The two cameras are independently arranged side by side at a specific distance. The position tracking module 252 obtains depth information based on the difference between the respective images obtained from the two cameras. The position tracking module 252 obtains a z coordinate value according to the obtained depth information.

[0062] In various embodiments, the processor 250 receives information about the tilt of the external electronic device 202 from the external electronic device 202. The processor 250 receives the information about the tilt via the communication module 220.

[0063] According to an embodiment, the tilt information includes information obtained by at least one sensor included in the external electronic device 202. For example, the at least one sensor includes a gyroscope sensor. The gyroscope sensor acquires sensor data (hereinafter, first sensor data) for the tilt of the external electronic device 202. The first sensor data includes sensor data related to at least one of pitch, roll, or yaw.

[0064] According to an embodiment, tilt information is sent from the external electronic device 202 to the electronic device 201. For example, the tilt information is sent to the electronic device 201 based on a state change of the external electronic device 202. The external electronic device 202 sends the tilt information to the electronic device 201 in response to a power-on, on-state, powered-on, or activated state of the external electronic device 202. The tilt information is periodically sent from the external electronic device 202 to the electronic device 201. In one embodiment, the external electronic device 202 also includes a grip sensor (not shown). The grip sensor is configured to detect that the external electronic device 202 has been gripped by a user. The processor 295 (e.g., processor 250) receives a signal from the grip sensor indicating that the external electronic device 202 has been gripped by a user, and in response to receiving the indication signal, the processor 295 activates the gyroscope sensor. The processor 295 obtains information about the tilt via the activated gyroscope sensor and sends it to the electronic device 201.

[0065] For another example, the external electronic device 202 transmits information about the tilt to the electronic device 201 in response to receiving a control signal from the electronic device 201. The electronic device 201 detects that the interlocking with the external electronic device 202 is completed, and in response to the detection operation, transmits a control signal requesting the external electronic device 202 to transmit information about the tilt. The control signal includes a signal indicating activation of a gyro sensor and / or an acceleration sensor. The external electronic device 202 activates the gyro sensor in response to the control signal and acquires first sensor data. The external electronic device 202 activates the acceleration sensor in response to the control signal and acquires second sensor data.

[0066] In various embodiments, the processor 250 provides a graphic object (visual object) for representing the external electronic device 202 and multimedia content related to virtual reality (VR). A graphic object refers to an object displayed on the display 230 to indicate a user input in multimedia content related to virtual reality. The user input includes information about the movement of the external electronic device 202. Graphic objects are referred to as various terms, such as visual objects, graphic objects, and controller objects. According to an embodiment, the graphic object includes images of various shapes. For example, the graphic object includes an image that is the same or similar to the shape of the external electronic device 202. For another example, the external electronic device 202 includes an image determined based on the type of multimedia content. The image determined based on the type of content includes images for various objects (including tennis rackets, steering wheels, knives or swords, etc.).

[0067] In one embodiment, the processor 250 displays multimedia content related to virtual reality. The multimedia content includes at least one of a three-dimensional image or a panoramic image for providing a virtual reality service to a user.

[0068] In another embodiment, the processor 250 displays a graphic object representing the external electronic device 202 on the display 230. For example, the graphic object is displayed to overlap with multimedia content related to VR. The processor 250 renders multimedia content related to VR and renders the graphic object at a first time point. The graphic object is displayed overlappingly on the multimedia content.

[0069] In another embodiment, the graphic object is identified based on information about the position of the external electronic device 250 or information about the tilt of the external electronic device 202. For example, the processor 250 obtains information about the position of the external electronic device 202 and identifies the position for displaying the graphic object. For example, the processor 250 determines to display the graphic object in three dimensions at the identified display position based on the information about the tilt. Since the processor 250 only identifies the position of the external electronic device 202 based on the illuminance value in the image acquired via the camera 240, the tilt information is required as information for rendering in three dimensions. For another example, the processor 250 obtains information about the tilt based on a pattern or mark of the external electronic device 202 included in the image acquired via the camera 240. In one embodiment, the external electronic device 202 includes at least one mark. Since the at least one mark is located in a fixed area of ​​the external electronic device 202, when the external electronic device 202 is tilted, the at least one mark is also tilted in the same manner. The processor 250 obtains an image of at least one tilted mark in the image acquired via the camera 240. In another embodiment, the external electronic device 202 outputs a predetermined pattern of light through an LED. For example, when the external electronic device 202 is tilted to form a vertical angle with the ground, the external electronic device 202 turns on or off the LED at intervals of 0.1 seconds. For another example, when the external electronic device 202 is tilted to be parallel to the ground, the external electronic device 202 turns on or off the LED at intervals of 0.5 seconds. The external electronic device 202 indirectly provides information about the tilt of the external electronic device 202 by sending information about a predetermined pattern to the electronic device 201.

[0070] In yet another embodiment, the graphic object is determined based on multimedia content related to VR. The multimedia content related to VR includes tennis matches, war games, car games, etc. For example, when the multimedia content corresponds to a tennis match, the graphic object corresponds to an image including the shape of a tennis racket. For another example, when the multimedia content corresponds to a war game, the graphic object corresponds to an image including the shape of a sword or a knife. As another example, when the multimedia content corresponds to a car game, the graphic object corresponds to an image including the shape of a steering wheel.

[0071] The processor 250 determines a point on the display 230 corresponding to the value of the x-coordinate and / or the y-coordinate by referring to the value of the x-coordinate and / or the y-coordinate in the information about the position. The processor 250 determines the size of the determined point by referring to the value of the z-coordinate in the information about the position. For example, when the value of the z-coordinate is larger (for example, the external electronic device 202 is far away from the user), the determined point for the perspective effect is displayed smaller. For another example, when the value of the z-coordinate is smaller (for example, the external electronic device 202 is located close to the user), the determined point for the perspective effect is displayed larger. The processor 250 determines the three-dimensional display of the graphic object based on the tilt information. For example, the processor 250 determines the tilt direction or angle of the three-dimensional form by referring to the gyroscope sensor data indicating at least one of the pitch, roll or yaw in the information about the tilt. The processor 250 displays the graphic object tilted according to the tilt direction or angle of the 3D form at the position determined according to the x-coordinate, the y-coordinate or the z-coordinate. For example, referring to Figure 6 , the processor 250 displays a display image 620 via the display 230. The display image 620 includes a background image or a graphic object 602 related to the multimedia content. The processor 250 displays the graphic object 602 on an area corresponding to the position of the external electronic device 102 recognized based on the camera image 610. The processor 250 displays the graphic object 602 to have the same inclination based on the information about the inclination of the external electronic device 202 received from the external electronic device 202. The image 620 displayed via the display 230 includes an image corresponding to the field of view of the user. In various embodiments, the field of view of the camera 240 is the same as the field of view of the display area of ​​the image 620 displayed via the display 230.

[0072] In various embodiments, the processor 250 identifies that the external electronic device 202 deviates from the field of view of the camera 240. For example, the processor 250 identifies whether the external electronic device 202 deviates from the field of view of the camera 240 based on the image acquired by the camera 240. When an area corresponding to a threshold illuminance or higher is not detected in the acquired image, the processor 250 determines that the external electronic device 202 has deviated from the field of view of the camera 240. As another example, if the processor 250 detects an area corresponding to a threshold illuminance or higher in the image acquired by the camera 240, the processor 250 determines that the external electronic device 202 is within the field of view of the camera 240. For another example, referring to Figure 4a , the user holds each of the external electronic devices 202-1 and 202-2 with both hands while wearing the electronic device 201. Each of the external electronic devices 202-1 and 202-2 corresponds to Figure 2 The external electronic device 202 shown in FIG. Figure 4a, when the user of the electronic device 201 faces the arm frontally (e.g., indicated by a solid line), the external electronic devices 202-1 and 202-2 are within the field of view (401) of the electronic device 201, and the electronic device 201 acquires an image including the external electronic devices 202-1 and 202-2. Since the processor 250 detects an area corresponding to an illumination of a threshold illuminance or higher emitted from the external electronic devices 202-1 and 202-2, the processor 250 recognizes that the external electronic devices 202-1 and 202-2 are within the field of view of the electronic device 201. Figure 4a , when the user of the electronic device 201 points the arm to the side (e.g., as in the example of the arm indicated by the dotted line), the external electronic devices 202-1 and 202-2 deviate from the viewing angle 401 of the electronic device 201, and the electronic device 201 does not acquire an image including the external electronic devices 202-1 and 202-2. The electronic device 201 does not detect an area corresponding to an illumination higher than a threshold illumination or higher emitted from the external electronic devices 202-1 and 202-2. The electronic device 201 recognizes that the external electronic devices 202-1 and 202-2 deviate from the field of view of the camera 240 or are outside the field of view of the camera 240.

[0073] According to an embodiment, the processor 250 stores information about the position of the external electronic device 202 corresponding to the deviation time point. For example, the processor 250 identifies the position of the external electronic device 202 in response to identifying the deviation. The identified position is located at one of the boundary surfaces of the image acquired by the camera 240. The processor 250 stores the information about the identified position as second position information. The second position information is obtained by the latest image including the external electronic device 202 among the multiple images acquired by the camera 240. When the image acquired by the camera 240 is divided into an x-axis along the horizontal axis and a y-axis along the vertical axis, the second position information includes information about one of the points having only one of the x-coordinate value or the y-coordinate value.

[0074] In various embodiments, in response to the above recognition operation, the processor 250 sends a request to the external electronic device 202 to send information about the change in the position of the external electronic device 202. When the external electronic device 202 deviates from the field of view of the camera 240, the camera 240 does not acquire an image including the external electronic device 202, and therefore, the processor 250 does not acquire information about the position of the external electronic device 202. Therefore, when the processor 250 detects the deviation of the external electronic device 202, the processor 250 sends a request for information about the change in the position of the external electronic device 202 to the external electronic device 202.

[0075] In one embodiment, the information about the change in position includes information obtained by at least one sensor of the external electronic device 202. For example, the at least one sensor included in the external electronic device 202 includes a first sensor 292 or a second sensor 294. The first sensor 292 corresponds to a gyro sensor, and the second sensor 294 corresponds to an acceleration sensor. The first sensor 292 acquires information about the tilt of the external electronic device 202, and the second sensor 294 acquires information about the change in speed of the external electronic device 202. The external electronic device 202 transmits the information acquired from the first sensor 292 or the second sensor 294 to the electronic device 201 in response to a request for information about the change in position. That is, the external electronic device 202 transmits only the information about the tilt through the first sensor 292 to the electronic device 201, and transmits the information about the tilt through the first sensor 292 and the information about the change in speed through the second sensor 294 together in response to the request. The processor 250 of the electronic device 201 predicts the position of the external electronic device 202 at a point in time after the external electronic device 202 deviates using the tilt information or the information about the change in speed transmitted thereto.

[0076] In another embodiment, the information about the position change includes data indicating the amount of position change of the external electronic device 202. For example, the data indicating the amount of position change includes vector information. The data indicating the amount of position change is obtained by the external electronic device 202. The external electronic device 202 obtains the vector information based on the information obtained by the first sensor 292 or the second sensor 294. The external electronic device 202 transmits the vector information obtained based on the sensor data instead of transmitting the sensor data obtained by the first sensor 292 or the second sensor 294.

[0077] In one embodiment, the processor 295 predicts the moving direction of the external electronic device 202 based on the information about the tilt. For example, the processor 295 predicts the movement of the external electronic device 202 in the direction corresponding to the rotation direction. This is because, when the user holds the external electronic device 202 with his hand, when the data is detected by the gyro sensor, the expected value for the case where the data is detected by the rotation movement at the original position is very low. Therefore, when the processor 295 obtains data indicating that the external electronic device 202 has been rotated in the x+ direction, the processor 295 predicts that the external electronic device 202 has moved in the x+ direction. For another example, the processor 295 stores information about the position change for a repeated operation. In another embodiment, the processor 295 compares the data obtained from the first sensor 292 or the second sensor 294 with the stored information. When the obtained data matches the stored information, the processor 295 determines that the user holding the external electronic device 202 is performing a repeated operation. The processor 250 displays a graphic object on the display 230 based on the determined repeated operation.

[0078] In another embodiment, the request signal for sending the information about the position change includes a control signal indicating activation of the second sensor 294 of the external electronic device 202. The external electronic device 202 activates the second sensor 294 in response to receiving the control signal included in the request signal, and obtains information about the speed change. According to an embodiment, when the processor 250 recognizes the external electronic device 202 based on the image acquired by the camera 240 (for example, the external electronic device 202 is within the camera field of view 403), a control signal indicating activation of the first sensor 292 (for example, a gyro sensor) is sent to the external electronic device 202. The processor 250 receives information about the direction and / or angle in which the external electronic device 202 is tilted from the first sensor 292. The processor 250 renders the external electronic device 202 according to the direction and / or angle tilted at the position of the external electronic device 202 in the image acquired by the camera 240. According to another embodiment, when the processor 250 does not recognize the external electronic device 202 based on the image acquired by the camera 240 (for example, the external electronic device 202 is detected to be deviated from the camera field of view 403), a control signal indicating activation of both the first sensor 292 (for example, a gyro sensor) and / or the second sensor 294 (for example, an acceleration sensor) is sent to the external electronic device 202. The processor 250 receives information about the change in speed of the external electronic device 202 outside the camera field of view 403 from the second sensor 294. The processor 250 acquires the position to which the external electronic device 202 is predicted to have moved based on the information about the change in speed. The processor 250 receives information about the tilt direction and / or tilt angle of the external electronic device 202 from the gyro sensor. The processor 250 renders a graphic object corresponding to the external electronic device 202 at the position to which the external electronic device 202 is predicted to have moved. The processor 250 displays the graphic object based on the tilt direction and / or tilt angle.

[0079] According to various embodiments, the processor 250 changes the display of the graphic object based at least on information about the position of the external electronic device 202 at a deviation time obtained using the camera 240, information about the tilt of the external electronic device 202 received from the external electronic device 202, or information about a change in the position of the external electronic device 202.

[0080] According to an embodiment, the processor 250 predicts the position of the external electronic device 202 corresponding to the current time point. For example, the processor 250 generates information about the position change of the external electronic device 202 based on the information about the tilt or speed change of the external electronic device 202 received from the external electronic device 202. The information about the generated position change includes vector information. The vector information includes information about the displacement of the external electronic device 202. For another example, the processor 250 receives information about the position change from the external electronic device 202. The information about the position change received from the external electronic device 202 corresponds to the information obtained by the processor 295 of the external electronic device 202 based on the information about the tilt or speed change. The processor 250 predicts the current position of the external electronic device 202 based on the information about the position change and / or the information about the position of the external electronic device 202 at the deviated time point. For example, the processor 250 determines the area where the external electronic device 202 deviates as the starting point, and identifies the change in position according to the movement of the external electronic device 202 after deviating from the field of view of the camera based on the starting point and / or the vector information. The processor 250 displays a visual effect and / or a graphic object (eg, an arrow) separate from the visual effect for indicating a direction in which the external electronic device is located based on the predicted position of the external electronic device 202 .

[0081] In various embodiments, the starting point and / or vector information changes based on the movement of the electronic device 201. After the external electronic device 202 is outside the camera field of view 403, the processor 250 acquires information about the movement of the electronic device 201. Figure 1 The sensor module 176 in the electronic device 201 obtains information about the movement of the electronic device 201. In one embodiment, after the external electronic device 202 deviates from the camera field of view 403, when the electronic device 201 is rotated in the left direction (for example, wearing a VR device and rotating the head to the left), the starting point changes based on the movement of the electronic device 201. For example, by reflecting the left rotation of the electronic device 201, the starting point moves from the area when the external electronic device 202 deviates to the left direction. In another embodiment, if the electronic device 201 rotates in the left direction after the external electronic device 202 deviates from the camera field of view 403, the vector information changes based on the movement of the electronic device 201. For example, by reflecting the rotation of the electronic device 201 in the left direction, the vector information also includes information for indicating an additional position change as a right direction.

[0082] According to an embodiment, the processor 250 displays a portion of a first graphic object. The first graphic object refers to a graphic object determined based on multimedia content. For example, when the first graphic object corresponds to a sword, the processor 250 displays a portion of the first graphic object with reference to the length of the sword. That is, when the external electronic device 202 deviates from the field of view of the camera 240 but does not move more than the length of the sword, the processor 250 displays a portion of the first graphic object. The processor 250 uses vector information of the position change and deviation position of the external electronic device 202 to predict the current position, and when the distance between the predicted current position and the camera 240 is less than the length of the sword, a portion of the sword (e.g., the sword edge) is displayed.

[0083] According to an embodiment, the processor 250 displays a second graphic object different from the first graphic object by omitting the display of the first graphic object. The second graphic object corresponds to an object for indicating to the user that the external electronic device 202 is located in the outer area of ​​the image displayed on the display 230. The second graphic object is an object different from the first graphic object. The second graphic object includes an image that is the same or similar in shape to the external electronic device 202. For another example, the second graphic object includes an arrow-shaped image for indicating that the external electronic device 202 is located outside the field of view of the camera 240. The processor 250 further displays a guide word indicating that the external electronic device 202 is located outside the field of view of the camera 240 together with the second graphic object. For example, when the distance between the predicted position of the external electronic device 202 and the camera 240 exceeds the length of the first graphic object, the processor 250 does not display the first graphic object. Because the first graphic object is not displayed, the processor 250 displays a second graphic object for indicating the current position of the external electronic device 202.

[0084] The external electronic device 202 is configured to interlock with the electronic device 201 and provide user input in the content related to virtual reality. The external electronic device 202 includes a memory 260, a communication module 270, a user input module 280, a sensor module 290 or a processor 295.

[0085] In various embodiments, memory 260 includes Figure 1 The memory 130 shown in FIG. 1 and the communication module 220 include Figure 1 The communication module 190 shown in FIG. 1 and the user input module 280 include Figure 1 The input device 150 shown in FIG. 1 and the sensor module 290 include Figure 1 The sensor module 176 shown in FIG.

[0086] In various embodiments, the user input module 280 receives a user input for controlling the electronic device 201 interlocked with the external electronic device 202. The user input module 280 is visually exposed through a portion of a housing (not shown) of the external electronic device 202. In various embodiments, the user input module 280 includes a touch pad for receiving a user's touch input, a physical button capable of receiving physical pressure, or a physical key.

[0087] In various embodiments, the processor 295 is operably or manipulably coupled to at least one of the memory 260 , the communication module 270 , the user input module 280 , or the sensor module 290 .

[0088] In various embodiments, the sensor module 290 includes a first sensor 292 or a second sensor 294. The first sensor 292 corresponds to a gyro sensor, and the second sensor 294 corresponds to an acceleration sensor.

[0089] In various embodiments, the processor 295 transmits information about the tilt of the external electronic device 202 to the electronic device 201. For example, the processor 295 acquires information about the tilt via at least one sensor. The at least one sensor includes a gyroscope sensor. The tilt information includes information indicating a change in the 3D angle of the external electronic device 202. The tilt information includes data about the amount of change in pitch, roll, and yaw.

[0090] In various embodiments, the processor 295 transmits information about the position change of the external electronic device 202 to the electronic device 201. The information about the position change includes information about the tilt or information about the speed change obtained by the first sensor 292 or the second sensor 294 of the external electronic device 202. For another example, the information about the position change includes information indicating the position change amount obtained based on the information about the tilt or speed change. The information indicating the position change amount corresponds to vector information.

[0091] In one embodiment, the processor 295 adaptively sends information about the position change to the electronic device 201. For example, the processor 295 sends information about the position change to the electronic device 201 at a predefined time interval. The predefined time interval is changed based on at least the remaining battery power of the external electronic device 202 and the quality of the wireless communication between the external electronic device 202 and the electronic device 201. For example, when the charge level of the battery of the external electronic device 202 is high, the predefined time interval is reduced to reflect the movement of the external electronic device 202 in real time. For another example, when the charge level of the battery of the external electronic device 202 is low, in order to increase the driving time of the external electronic device 202 and reduce power consumption, the predefined time interval is increased. For example, when the quality of wireless communication is poor, the processor 295 reduces the predefined time interval to ensure the reliability of data transmission. For another example, when the quality of wireless communication is good, the processor 295 increases the predefined time interval.

[0092] In another embodiment, the processor 295 sends the information about the position change together with different information sent to the electronic device 201. For example, the processor 295 receives a user's physical key input and sends the information about the position change together with the received input.

[0093] Although not shown, the external electronic device 202 further includes a light emitting portion. For example, the light emitting portion activates the light emitting portion in response to detecting a turned-on state of the external electronic device 202. For another example, the external electronic device 202 generates light in response to receiving a signal indicating light emission from the electronic device 201. The electronic device 201 is configured to obtain information about the location of the external electronic device 202 by capturing light emitted from the light emitting portion via the camera 240. In various embodiments, the light emitting portion includes a light source of a light emitting diode (LED).

[0094] Figure 3 Examples of functional configurations of the electronic device, the first external electronic device, and the second external electronic device according to various embodiments are shown. Figure 3 The functional configuration of the second external electronic device 303 shown in FIG. Figure 2 The functional configuration of the external electronic device 202 is shown in FIG. Figure 3 The functional configuration of the electronic device 301 shown in FIG. Figure 2 In the functional configuration of the electronic device 201 shown in .

[0095] Reference Figure 3 , the first external electronic device 302 includes a memory 321 , a communication module 322 , a display 323 , a camera 324 or a processor 325 .

[0096] In various embodiments, memory 321 includes Figure 2 The memory 210 shown in FIG. 1 and the communication module 322 include Figure 2 The communication module 220 shown in FIG. 3 includes a display 323 Figure 2 The display 230 shown in FIG. 1 , the camera 324 includes Figure 2 The camera 240 shown in FIG. 1 and the processor 325 include Figure 2 The processor 250 shown in .

[0097] In various embodiments, the processor 325 is operably coupled to at least one of the memory 321 , the communication module 322 , the display 323 , and the camera 324 .

[0098] In various embodiments, the processor 325 acquires information about the position of the second external electronic device 303. For example, when the second external electronic device 303 is located within the field of view of the camera 324, the processor 325 acquires an image including the second external electronic device 303 via the camera 324. According to one embodiment, the processor 325 further includes a position tracking module. The position tracking module acquires information about the position of the second external electronic device 303 within the acquired image. The position tracking module corresponds to Figure 2 In another embodiment, information about the position of the second external electronic device 303 is obtained by the electronic device 301. The processor 325 obtains an image including the second external electronic device 303 through the camera 324 and sends the obtained image to the electronic device 301. The electronic device 301 obtains information about the position of the second external electronic device 303 within the image including the second external electronic device 303 received from the first external electronic device 302.

[0099] In various embodiments, the processor 325 transmits information about the position corresponding to the deviation time point of the second external electronic device 303 to the electronic device 301. For example, the processor 325 determines whether the second external electronic device 303 has deviated from the field of view of the camera 324 based on the image acquired by the camera 324. When an area corresponding to an illuminance of a threshold value or more is detected in the acquired image, the processor 325 determines that the second external electronic device 303 is located in the detected area. For another example, when an area corresponding to an illuminance of a threshold value or more is not detected in the acquired image, the processor 325 determines that the second external electronic device 303 deviates from the field of view of the camera 324. The processor 325 temporarily stores an image at a time point immediately before a time point when an area corresponding to an illuminance greater than the threshold value or more is not detected in a buffer or the like. Therefore, when the deviation is recognized, the processor 325 recognizes an image including the second external electronic device 303 among a plurality of images stored in a buffer or the like. The processor 325 acquires the position information of the second external electronic device 303 at the deviation time from the recognized image using a position tracking module.

[0100] In various embodiments, the processor 325 sends a signal indicating the deviation to the electronic device 301. For example, the processor 325 sends a signal indicating the deviation to the electronic device 301 in response to recognizing the deviation. Since the electronic device 301 does not have a camera, it does not recognize the deviation of the second external electronic device 303. Therefore, the processor 325 sends the signal indicating the deviation to the electronic device 301. In various embodiments, the processor 325 sends information about the position of the second external electronic device 303 corresponding to the deviation time point together with the above signal to the electronic device 301.

[0101] In various embodiments, the processor 325 displays an image related to the multimedia content or a graphic object corresponding to the second external electronic device 303 on the display 323. For example, the processor 325 receives data for display from the electronic device 301. The processor 325 receives location information predicting the location of the second external electronic device 303 from the electronic device 301, and displays the graphic object based on the predicted location information. The predicted location information is obtained by the electronic device 301. The predicted location information is obtained based on information about the location change sent from the second external electronic device 303 to the electronic device 301 or information about the location of the second external electronic device 303 at a deviated time point sent from the first external electronic device 302 to the electronic device 301.

[0102] In various embodiments, the electronic device 301 generates a graphic object to be output on the display 323 of the first external electronic device 302. The graphic object includes multimedia content related to VR (e.g., VR operation) and / or a graphic object corresponding to the external electronic device 303. For example, the processor 313 of the electronic device 301 receives information about the position of the second external electronic device 303 obtained from the first external electronic device 302 and / or information about the position of the second external electronic device 303 obtained from the second external electronic device 303, and generates a graphic object based on the received position information of the second external electronic device 303. The graphic object is sent to the first external electronic device 302 and output via the display 323.

[0103] Figure 4a An example of a case where an external electronic device is deviated from the field of view of a camera according to various embodiments is shown. Figure 4a The electronic device 201 shown in FIG. Figure 2 The electronic device 201 shown in FIG. Figure 4a The external electronic devices 202-1 and 202-2 shown in FIG. Figure 2 The external electronic device 202 is shown in FIG.

[0104] Reference Figure 4a , the electronic device 201 corresponds to a state in which the electronic device 201 is worn on the user's body. The electronic device 201 includes a display 230 on one surface of a housing (not shown). The display 230 is arranged to face the user's eyes in the wearing state of the electronic device 201. The electronic device 201 includes a camera 240 on a surface opposite to one surface of the housing. The camera 240 is provided on the opposite surface to acquire an image of the external environment in a direction corresponding to the gaze of the user.

[0105] According to various embodiments, the external electronic device 202 corresponds to a state of being held by a user's hand. The external electronic device 202 includes two devices 202-1 and 202-2 (eg, controllers) such that they are held by the user's left and right hands, respectively.

[0106] According to various embodiments, the camera 240 included in the electronic device 201 acquires an image of a field of view corresponding to a specific angle 401 (e.g., a viewing angle of the camera 240). For example, when the external electronic device 202 is located within the angle 401 (e.g., when the user extends his arms in front), the camera 240 obtains an image of the external electronic device 202. In one embodiment, the image of the external electronic device 202 includes an external image or an image of a shell of the external electronic device 202. According to another embodiment, the image of the external electronic device 202 includes an LED, a pattern, or a mark. For example, the external electronic device 202 additionally includes an LED separated from the shell of the external electronic device so as to be easily recognized by the processor 250. As an alternative to identifying the external electronic device 202 by processing the image of the shell of the external electronic device 202, the processor 250 identifies the image of the external electronic device 202 within the image obtained by the camera 240 by identifying an LED that emits a brightness greater than or equal to a threshold illumination value. For another example, the external electronic device 202 includes at least one mark. Each of the at least one mark includes a mark in the form of a distinguishable pattern. The processor 250 recognizes the at least one marker from the image acquired by the camera 240, and recognizes the external electronic device 202 within the acquired image. For another example, when the external electronic device 202 deviates from the angle 401 (for example, when the user stretches his or her arms to both sides), the external electronic device 202 deviates from the field of view of the camera 240, and the camera 240 does not acquire an image 413 for the external electronic device 202.

[0107] Figure 4b An example of comparing a field of view corresponding to a camera of an electronic device and a field of view corresponding to a user's gaze according to various embodiments is shown. Figure 4b , the size of the area displayed on the display 230 is different from the area of ​​the image acquired by the camera 240. According to an embodiment, the field of view of the camera 240 includes an area larger than the field of view of the user (eg, the area displayed on the display 230). Figure 4b In the image (a) shown in FIG. 2 , an area 410 indicated by a solid line corresponds to an area displayed on the display 230. An area 412 indicated by a dotted line indicates the size of an area acquired by the camera 240. Therefore, when the external electronic device 202 is located within the field of view of the camera 240, the processor 250 acquires an image including the external electronic device 202 even if the external electronic device 202 is outside the field of view of the user. The processor 250 obtains information about the position of the external electronic device 202 based on the image including the external electronic device 202. Therefore, the processor 250 displays a graphic object 411 corresponding to the external electronic device 202 based on the obtained position information.

[0108] According to another embodiment, the field of view of the camera 240 is the same as the field of view of the user. Figure 4b In the image (b) shown in FIG. 1 , the size of the area 410 indicated by the solid line is the same as or similar to the size of the area 414 indicated by the dotted line. Therefore, when the external electronic device 202 is outside the field of view of the user, because it is outside the field of view of the camera 240, the processor 250 does not acquire an image of the external electronic device 202. When the external electronic device 202 is located in the field of view of the user, because it is located in the field of view of the camera 240, the processor 250 acquires an image of the external electronic device 202. Therefore, the processor 250 obtains information about the position of the external electronic device 202 based on the image, and displays a graphic object 411 corresponding to the external electronic device 202.

[0109] According to another embodiment, the field of view of camera 240 includes an area that is smaller than the field of view of the user. Figure 4b In the image (c) shown in FIG. 1 , the size of the area 416 indicated by the dotted line is smaller than the size of the area 410 indicated by the solid line. Therefore, when the external electronic device 202 is located outside the field of view of the camera 240, the processor 250 does not acquire an image including the external electronic device 202 even if the external electronic device 202 is located within the field of view of the user. The processor 250 transmits a request signal to the external electronic device 202 to transmit information about the position change of the external electronic device 202. The processor 250 acquires the position of the external electronic device 202 based on the information about the position change, and displays a graphic object 411 corresponding to the external electronic device 202 at the acquired position.

[0110] Figure 5a The operation of the electronic device according to various embodiments is shown. Such operation is performed by Figure 2 The electronic device 201 shown in FIG. 1 or the processor 250 included in the electronic device 201 is executed.

[0111] Reference Figure 5a In operation 501, the processor 250 uses a camera to identify an external electronic device among one or more external objects included in a specified field of view. The processor 250 acquires an image of the surrounding environment of the electronic device 201 through the camera. The image includes a plurality of objects located around the electronic device 201. The processor 250 identifies the external electronic device 202 among the plurality of objects. For example, the external electronic device 202 also includes a light emitting portion. The processor 250 identifies the external electronic device 202 by identifying a light source exceeding a predetermined illuminance in the image acquired through the camera.

[0112] According to an embodiment, the external electronic device 202 adjusts the brightness of the LED based on the ambient brightness. The external electronic device 202 also includes an illuminance sensor (not shown), and uses the illuminance sensor (not shown) to obtain the value of lux (illuminance) of the ambient brightness of the external electronic device 202. The external electronic device 202 adjusts the brightness of the light source based on the value of lux of the ambient brightness. For example, the external electronic device 202 outputs light having a first brightness corresponding to the first interval in response to recognizing that the value of lux of the ambient brightness is included in the first interval. For another example, in response to recognizing that the value of lux of the ambient brightness is included in the second interval, light of a second brightness corresponding to the second interval is output. The first brightness corresponds to an illuminance value that needs to be separately identified from the ambient brightness of the first interval. The second brightness corresponds to an illuminance value that needs to be separately identified from the ambient brightness of the second interval.

[0113] At operation 503, the processor 250 displays a graphic object corresponding to the external electronic device on the display based on the first position information of the external electronic device 202. The processor 250 displays the graphic object at the identified position of the external electronic device 202. The graphic object is determined based on the multimedia content. For example, the graphic object includes images of various shapes, such as a knife, a tennis racket, or a car steering wheel.

[0114] At operation 505, if the external electronic device 202 is outside the designated field of view, the processor 250 displays a graphic object on the display based on the second position information confirmed by the camera before the external electronic device 202 is outside the designated field of view and the information related to the movement of the external electronic device received from the external electronic device through the communication circuit after the external electronic device 202 is outside the designated field of view. According to an embodiment, the processor 250 stores the second position information at the time point when the external electronic device 202 deviates from the designated field of view. The second position information is located at a point of the boundary surface of the image displayed on the display 210. According to an embodiment, the processor 250 receives the information related to the movement of the external electronic device 202 from the external electronic device 202 starting from the time point when the external electronic device 202 is outside the designated field of view. The movement related information includes data obtained by the gyro sensor and / or data obtained by the acceleration sensor. The processor 250 generates a vector value for the movement of the external electronic device based on the information related to the movement. The vector value is obtained by determining the magnitude and / or direction of the vector according to the information related to the movement, taking the area where the external electronic device 202 deviates as a starting point.

[0115] Figure 5b The operation of the electronic device according to various embodiments is shown. Such operation is performed by Figure 2 The electronic device 201 shown in FIG. 1 or the processor 250 included in the electronic device 201 is executed.

[0116] At operation 511, the processor 250 obtains information about the position of the external electronic device 202 interlocked with the electronic device 201. The information about the position of the external electronic device 202 refers to information about coordinate values ​​recognized by the external electronic device 202 in the image acquired by the camera 240. In one embodiment, the processor 250 acquires an image via the camera 240. The processor 250 obtains an x-coordinate value and / or a y-coordinate value based on the image. For example, by analyzing the image, the processor 250 identifies a coordinate value corresponding to an area where the external electronic device 202 is located in the image. In another embodiment, the processor 250 obtains depth information based on an image analysis of the image. The processor 250 obtains a z-coordinate value based on the acquired depth information and / or the acquired x-coordinate value and y-coordinate value.

[0117] At operation 513, the processor 250 receives tilt information about the tilt of the external electronic device 202. The tilt information includes sensor data obtained by the first sensor 292 included in the external electronic device 202. For example, the tilt information includes information about at least one amount of change in pitch, roll, or yaw. According to an embodiment of the present disclosure, the tilt information is configured to be automatically sent from the external electronic device 202 to the electronic device 201 in response to a turned-on state or an activated state of the external electronic device 202. According to another embodiment, the tilt information is configured to be sent to the electronic device 201 in response to a request from the electronic device 201.

[0118] At operation 515, the processor 250 provides a graphic object for representing the external electronic device 202 and multimedia content. The processor 250 uses the information obtained by the camera 240 about the tilt of the external electronic device 202 and / or the position of the external electronic device 202, thereby allowing the graphic object to be displayed. For example, the processor 250 obtains coordinate values ​​for x, y, and z from the camera 240, and receives information about the tilt from the external electronic device 202, and determines the shape of the external electronic device 202 in three-dimensional space. The processor 250 displays the graphic object to correspond to the determined shape of the external electronic device 202. For example, the graphic object is displayed at the same coordinates as the external electronic device 202 with the same tilt. According to various embodiments, the graphic object is determined based on the multimedia content. For example, when the multimedia content is tennis-related content, the graphic object includes the shape of a tennis racket. For another example, when the multimedia content is content related to a fighting game, the graphic object includes the shape of a knife or a sword.

[0119] At operation 517, the processor 250 identifies that the external electronic device 202 is outside the field of view of the camera. The processor 250 determines that the external electronic device 202 does not exist based on the image acquired by the camera 240. For example, when the image acquired by the camera 240 does not include an image of light emitted from a light emitting portion (not shown) of the external electronic device 202, the processor 250 identifies the deviation of the electronic device 202. For another example, when an area brighter than a predetermined illumination is not detected in the acquired image, the processor 250 determines that the external electronic device 202 has deviated. According to an embodiment, the processor 250 stores information about the position of the external electronic device 202 at the time point of deviation. For example, the processor 250 stores information about the position where the external electronic device 202 deviates in response to detecting the deviation.

[0120] In operation 519, the processor 250 requests to transmit information about the change in position of the external electronic device 202. When the external electronic device 202 deviates from the field of view of the camera 240, the processor 250 does not acquire information about the position of the external electronic device 202. The processor 250 transmits a signal requesting the external electronic device 202 to transmit information about the change in position of the external electronic device 202. According to various embodiments, the request signal includes a control signal indicating activation of the second sensor 294 of the external electronic device 202. For example, when the external electronic device 202 is included in the field of view of the camera 240, the external electronic device 202 activates only the first sensor 292, and information about the tilt obtained from the first sensor 292 is transmitted to the electronic device 201. For another example, when the external electronic device 202 deviates from the field of view of the camera 240, the external electronic device 202 receives a control signal from the electronic device 201 and activates the second sensor 294. The external electronic device 202 transmits information about the tilt obtained by the first sensor 292 and / or information about the speed change of the external electronic device 202 obtained by the second sensor 294 to the electronic device 201. For another example, regardless of whether a control signal indicating activation of the second sensor 294 is received from the electronic device 201, the second sensor 294 transmits information about the speed change to the electronic device 201. The processor 295 of the external electronic device 202 confirms (e.g., detects) information related to driving the external electronic device 202 (e.g., powering on), and activates the second sensor 294 in response to the confirmation operation. In addition, the processor 295 also includes a grip sensor (not shown), and the grip sensor (not shown) detects that the external electronic device 202 is gripped by a user. The processor 295 receives a sensor value from the grip sensor (not shown), and activates the second sensor 294 in response to receiving the sensor value. When the second sensor 294 is activated in response to power-on or receiving a sensor value from a grip sensor (not shown) of the external electronic device 202, information on the tilt of the external electronic device 202 acquired by the first sensor 292 and information on the change in speed of the external electronic device 202 acquired by the second sensor 294 are sent together to the electronic device 201. According to various embodiments, the processor 250 obtains a vector value for the change in position of the external electronic device 202. The processor 250 receives information on the change in position from the external electronic device 202, and the received information includes data acquired by the first sensor 292 and / or the second sensor 294. The processor 250 obtains the vector value based on the data on the tilt of the external electronic device 202 acquired by the first sensor 292 and / or the data on the change in speed of the external electronic device 202 acquired by the second sensor 294. The vector value includes information indicating how far the external electronic device 202 has moved from the position information corresponding to the deviation time point relative to the field of view of the camera 290 and the direction of the movement.

[0121] In various embodiments, the processor 250 determines information about a change in position of the external electronic device 202 based on information about a change in position of the electronic device 201. For example, the electronic device 201 includes a sensor module (not shown). The sensor module (not shown) includes an acceleration sensor and / or a gyro sensor. The gyro sensor acquires information about the tilt of the electronic device 201, and the acceleration sensor acquires information about a change in speed of the electronic device 201. When the electronic device 201 does not receive information about a change in tilt or speed of the external electronic device 202 from the first sensor 292 or the second sensor 294 within a predetermined time or receives information indicating empty, the electronic device 201 obtains information about a change in position of the electronic device 201. When information indicating empty is received from the external electronic device 202, or if information is not received within a predetermined time, the processor 250 determines that the electronic device 201 deviates from the field of view of the camera 240 due to the movement of the electronic device 201 when the external electronic device 202 does not move. Therefore, the electronic device 201 acquires information about the position change of the electronic device 201, and generates information about the position change of the external electronic device 202 based on the information about the position change of the electronic device 201. For example, in the case where the electronic device 201 moves in the upper right direction in the field of view of the camera 240, it is determined that the external electronic device 202 moves in the opposite direction (e.g., the lower left direction). This is because the position change of the external electronic device 202 presents a direction opposite to the moving direction of the electronic device 201 based on the field of view of the user (e.g., the display 230).

[0122] At operation 521, the processor 250 changes the display of the graphic object. The processor 250 determines the position where the external electronic device 202 is predicted to move based on the received information about the position change of the external electronic device 202. The processor 250 changes the display of the graphic object to correspond to the predicted position. According to an embodiment, the processor 250 changes the display of the graphic object so that a part of the graphic object is displayed and the rest is not displayed. For example, referring to Figure 7 According to another embodiment, the processor 250 does not display the graphic object. For example, referring to Figure 7In the image 740 of the image 740, the processor 250 does not display the graphic object including the shape of the knife, but only displays the image related to the multimedia content. When the predicted position of the external electronic device 202 is outside the predefined distance based on the information about the tilt information of the external electronic device 202 and / or the information about the speed change of the external electronic device 202, the processor 250 stops displaying the graphic object. For example, when the graphic object includes a shape with a longer length, the processor 250 sets the predefined distance of a larger size. For another example, when the graphic object includes a shape with a shorter length, the processor 250 sets the predefined distance of a smaller size.

[0123] Figure 6 Examples of an external electronic device and a graphic object displayed to correspond to the external electronic device according to various embodiments are illustrated.

[0124] Reference Figure 6 (a) and (b), the processor 250 uses the camera 240 to obtain a camera image 610. For example, the camera image 610 includes an image of the external electronic device 202, a user's hand for holding the external electronic device 202, and an external environment where the user is located.

[0125] In various embodiments, the processor 250 displays the display image 620 on the display 230. For example, the display image 620 includes multimedia content and / or a graphic object 602 corresponding to the external electronic device 202. For example, the graphic object 602 includes an image corresponding to the shape of a knife or sword.

[0126] According to various embodiments, the processor 250 displays the graphic object 602 and the multimedia content in an overlapping pattern. The processor 250 renders or displays a background image related to the multimedia content. For example, when the content is related to tennis, the background image corresponds to an image related to a tennis court or a tennis stadium. For example, the processor 250 displays a tennis racket at a position corresponding to the external electronic device 202 in the background image associated with the tennis court or tennis stadium, so that the background image and / or the graphic object are overlapped. The electronic device 201 provides immersive services related to VR by displaying multimedia content and / or a graphic object 602 overlapped on the multimedia content on the display 230.

[0127] Figure 7 2 shows a display that changes over time according to various embodiments. Hereinafter, images 710 to 760 displayed on display 230 include images corresponding to the field of view of the user. In various embodiments, the field of view of camera 240 is the same as the field of view of the display area of ​​images 710 to 760 displayed on the display. However, the field of view of camera 240 is not limited thereto, and as Figure 4b As shown in , it is smaller or larger than the user's field of view.

[0128] Reference Figure 7 (a) to (f), according to various embodiments, the processor 250 displays an image 710. When the external electronic device 202 is located within the field of view of the camera 240, the processor 250 obtains information about the position of the external electronic device 202 based on the image acquired by the camera 240. The processor 250 displays the image 710 based on the information and / or position information about the tilt of the external electronic device 202 received from the external electronic device 202. The processor 250 identifies the position of the electronic device 201 located within the field of view of the camera 240, and displays the graphic object as a three-dimensional object (e.g., a knife or a sword). For example, referring to the image 710, the graphic object is displayed as an image including the shape of a knife. The image including the shape of a knife is displayed based on the information about the position of the electronic device identified using the camera 240 and the information about the tilt received from the external electronic device 202, and as shown in the image 710, the image including the shape of a knife is displayed in the three-dimensional space as tilted in the same manner as the external electronic device 202. According to one embodiment, the processor 250 identifies the external electronic device 202 within the image obtained by the camera 240, and displays the knife at a position corresponding to the identified external electronic device 202. The processor 250 receives information about the tilt of the external electronic device 202 from the external electronic device 202, and based on the received information about the tilt, displays the tilt of the handle and the tilt of the blade of the knife. For example, the processor 250 identifies the area brightly indicated by the LED in the image obtained by the camera 240, and based on the information about the tilt of the external electronic device 202, the processor 250 marks the handle portion of the knife and the blade portion of the knife tilted 45 degrees to the left. The handle portion of the knife is marked on the identified area.

[0129] According to various embodiments, the processor 250 displays the image 720. When the external electronic device 202 deviates from the field of view of the camera 240, the electronic device 201 stores the position information at the deviation time point. For example, when the processor 250 fails to detect the brightness of the threshold illumination value or higher in the image acquired by the camera 240, the processor 250 detects the deviation. For another example, the processor 250 recognizes the external image of the external electronic device 202 or the image of the mark attached to the external electronic device 202 within the acquired image based on the image processing of the image acquired by the camera 240. When the processor 250 fails to detect the external image or the image for the mark, the processor 250 determines that the external electronic device 202 is outside the field of view of the camera 240. The processor 250 requests the external electronic device 202 to send information about the change in the position of the external electronic device 202 in response to detecting the deviation. For example, referring to the image 720, the user's hand deviates from the field of view of the camera. According to an embodiment, the processor 250 displays a portion of a graphic object. For example, if the external electronic device 202 does not move away by a predetermined distance after the deviation time, the processor 250 displays only a portion of the graphic object. Referring to image 720, the external electronic device 202 does not move a distance exceeding the length of the knife after deviating from the field of view of the camera 240. Therefore, the processor 250 does not display the handle portion of the knife corresponding to the external electronic device 202, but displays a portion of the graphic object corresponding to the blade portion of the knife.

[0130] According to various embodiments, the processor 250 displays the image 730. The processor 250 receives information about the position change from the external electronic device 202. The processor 250 predicts that the external electronic device 202 has moved a certain distance in the upper right direction based on the received information about the speed change of the external electronic device 202 and / or the information about the tilt of the external electronic device 202. The processor 250 changes the display of the tilt angle of the graphic object as shown in the image 730 by reflecting the tilt information. The processor 250 receives the information about the tilt of the external electronic device 202 while not displaying the handle portion of the knife corresponding to the external electronic device 202, and displays the graphic object corresponding to the blade portion of the knife. For example, the processor 250 receives information about the tilt, wherein the information indicates that the external electronic device 202 is further tilted to the left after the external electronic device 202 is out of the field of view of the camera 240. Although the external electronic device 202 is not included in the image acquired by the camera 240, the processor 250 changes the angle of the displayed blade portion based on the tilt information. The processor 250 displays a graphic object corresponding to the blade of the knife to be tilted to the left based on the received tilt information.

[0131] According to various embodiments, the processor 250 displays the image 740. The processor 250 receives information about the change in position of the external electronic device 202, and does not display a graphic object corresponding to the external electronic device 202 on the display 230. For example, the processor 250 obtains the predicted position to which the external electronic device 202 has moved based on data about the change in speed of the external electronic device 202 and / or data about the tilt. When the predicted position is located farther than a predetermined distance, the processor 250 does not display the graphic object including the shape of a knife. The predetermined distance is, for example, a distance equal to the length of the graphic object including the shape of a knife.

[0132] According to various embodiments, the processor 250 displays the image 750 and / or the image 760. The processor 250 detects that the external electronic device 202 has moved into the field of view of the camera 240. Therefore, the processor 250 acquires information about the position of the external electronic device 202 through the camera 240 and displays a graphic object. According to an embodiment, the processor 250 displays the image 750 corresponding to the external electronic device 202 at a time point when the external electronic device 202 enters the field of view of the camera 240. The processor 250 displays the graphic object at the time of re-entry based on the information about the tilt of the external electronic device 202. For example, the processor 250 identifies the entry position of the external electronic device 202 based on the image acquired at the entry time point. The processor 250 determines the tilt of the graphic object to be displayed at the entry position based on the information about the tilt of the external electronic device 202 in the received information about the position change. The processor 250 displays the graphic object based on the determined tilt and / or entry position.

[0133] Figure 8 An example for comparing effects of correction based on information on a position change of an external electronic device according to various embodiments is illustrated.

[0134] Hereinafter, the image 810 and the image 820 displayed on the display 230 include images corresponding to the field of view of the user. In various embodiments, the field of view of the camera 240 is the same as the field of view of the display area of ​​the image 810 and the image 820 displayed on the display 230. However, the field of view of the camera 240 is not limited thereto, and as Figure 4b As shown in , it is smaller or larger than the user's field of view.

[0135] Reference Figure 8(a), image 810 corresponds to an image in which the display of the graphic object is not corrected by using information about the position change. For example, the processor 250 stores information corresponding to the deviation time point of the external electronic device 202. The processor 250 stores the position (hereinafter, first position information) 815 and / or the slope (first sensor information) of the external electronic device relative to the deviation time point. Thereafter, when the external electronic device 202 moves into the field of view of the camera 240, the processor 250 detects the external electronic device 202. The processor 250 displays a graphic object 813 for the entry time point of the external electronic device 202. For example, the processor 250 does not receive information about the tilt from the external electronic device 202 after the deviation time point. The processor 250 displays the graphic object 813 according to the position (hereinafter, second position information) 817 of the external electronic device 202 relative to the entry time point of the external electronic device 202. Since the processor 250 has not received information about the tilt of the external electronic device 202 at the entry time point, the processor 250 displays the graphic object 813 based on the first sensor information and / or the second position information. Therefore, when the actual external electronic device 102 tilts toward the upper right side, the processor 250 displays the graphic object 813 according to the slope corresponding to the deviation time point of the external electronic device 202.

[0136] Reference Figure 8 , image 820 corresponds to an image in which the display of a graphic object is corrected by using information about a position change. For example, the processor 250 receives information about a position change of the external electronic device 202 from the external electronic device 202 during a time range from a deviation time point to an entry time point. The processor 250 identifies information about a tilt of the external electronic device 202 corresponding to the entry time point based on the received information about the position change. Therefore, the processor 250 displays a graphic object 823 based on the second position information 827 corresponding to the entry time point of the external electronic device 202 and / or the information about the tilt of the external electronic device 202 corresponding to the entry time point. For example, referring to Figure 8 In (a) and (b), the graphic objects 811 and 821 corresponding to the deviation time point of the external electronic device 202 are tilted toward the upper left at the deviation time point. Figure 8 (b), the graphic object 823 displayed at the entry time point of the external electronic device 202 is not facing the upper left, but is displayed as tilted to the upper right to match the tilt of the external electronic device 202. In the above embodiment, the processor 250 naturally displays the graphic object at the entry time point by receiving information about the position change of the external electronic device 202 from the external electronic device 202 during the time when the external electronic device 202 deviates from the field of view of the camera 240 and then enters the field of view of the camera 240.

[0137] Figure 9a An example of the operation of an electronic device according to various embodiments is shown. Such operation is performed by Figure 2 The process is performed by the electronic device 201 shown in FIG.

[0138] At operation 901, the processor 250 uses the camera 240 to identify the external electronic device 202. According to an embodiment, the processor 250 acquires an image of the external environment through the camera 240. The image of the external environment includes a plurality of external objects. The processor 250 identifies the external electronic device 202 among the plurality of external objects. For example, the external electronic device 202 also includes a light emitting portion for generating light outside the housing. The processor 250 identifies the external electronic device 202 by identifying an area brighter than a predetermined illuminance in the image acquired by the camera.

[0139] At operation 903, when the external electronic device 202 is out of the field of view of the camera, the processor 250 receives from the external electronic device 202 one or more position information acquired during a specified time range before the external electronic device 202 enters a low power mode. According to an embodiment, the low power mode corresponds to a mode for reducing power consumption of the external electronic device 202. The external electronic device 202 determines to enter the low power mode and controls the amount of power provided to a plurality of components. For example, the external electronic device 202 changes the operation cycle of the sensor module 290 to a long cycle to reduce power consumption, and reduces the number of operations of the sensor module 290. According to various embodiments, the low power mode is referred to as various terms including a dormant mode, a sleep mode, an inactive mode, an inactive state, a deactivated state, and the like. According to an embodiment, the external electronic device 202 executes a low power mode in which the external electronic device 202 does not perform a data acquisition operation performed by the sensor module 290 during a specified time range in the low power mode. When the external electronic device 202 acquires at least one of the information about the tilt acquired by the first sensor 292 and / or the information about the speed change acquired by the second sensor 294, the external electronic device 202 does not enter the low power mode. When the external electronic device 202 does not enter the low power mode, the external electronic device 202 monitors the acquisition of the at least one information during the above-mentioned specified time range from the time point corresponding to the acquisition of the at least one information. According to various embodiments, the external electronic device 202 acquires one or more position information during the specified time range. The sensor module 290 is activated from the specified time range (i.e., from the time when the user input was last received) to the time when the low power mode is executed. The sensor module 290 receives information about the tilt of the external electronic device 202 corresponding to the time range from the time when the user input was received to the time when the low power mode is executed.

[0140] At operation 905, the processor 250 receives a display request for displaying a graphic object associated with the external electronic device 202. The display request is based on a change in the operation mode of the external electronic device 202. For example, the processor 250 detects that the operation mode changes from the inactive mode to the active mode, and in response to the detection operation, receives a request signal for requesting the display of the graphic object. The request signal corresponds to a signal received from another electronic device (e.g., the second external electronic device 202). According to another embodiment, the request signal includes a control signal received from an element of the external electronic device 202. For another example, a display request for a graphic object is generated by detecting that a data value of the external electronic device 202 acquired by the first sensor 292 exceeds a threshold. For example, when the external electronic device 202 detects a motion exceeding a predetermined threshold, a display request for the graphic object is sent to the electronic device 201. For another example, when the change amount of the detected motion does not exceed a predetermined threshold (e.g., vibration caused by the user's carelessness is one case), the external electronic device 202 does not generate a display request. For another example, the external electronic device 202 also includes a grip sensor (not shown). When the grip sensor detects the user's grip, the external electronic device 202 generates a display request.

[0141] At operation 907, the processor 250 determines a position for displaying the graphic object based on one or more position information including information about a position of the external electronic device 202 deviating from the field of view of the camera 240, first sensor information corresponding to a time point when entering the low power mode, and second sensor information corresponding to a time point when releasing the low power mode.

[0142] Figure 9b An example of the operation of an electronic device according to various embodiments is shown. Such operation is performed by Figure 2 The process is performed by the electronic device 201 shown in FIG.

[0143] At operation 911, the processor 250 obtains information about the position of the external electronic device 202 interlocked with the electronic device 201. At operation 913, the processor 250 receives information about the tilt of the external electronic device 202. At operation 915, the processor 250 provides a graphic object (visual object) for representing the external electronic device 202 and multimedia content. Operations 911 to 915 correspond to Figure 5b Operations 511 to 515 shown in FIG.

[0144] At operation 917, the processor 250 determines that the sleep mode has been entered. The sleep mode corresponds to a mode for reducing power consumption of the external electronic device 202. According to an embodiment, when no user input is received for a predetermined length of time, the processor 250 determines that the entry into the sleep mode has been performed.

[0145] At operation 919, the processor 250 receives position information and / or tilt information corresponding to a time point of entering the sleep mode of the external electronic device 202. For example, when the processor 250 does not receive a user input within a predetermined length of time, the processor 250 determines that the low power mode has been entered, and controls receiving information about the external electronic device 202 corresponding to a time point of entering the low power mode.

[0146] At operation 921, the processor 250 recognizes that the sleep mode is released. The sleep mode is released by receiving a wake-up signal from the external electronic device 202. The wake-up signal is referred to in various terms such as a paging signal, an activation signal, etc. For example, when the movement of the external electronic device 202 is detected to exceed a predetermined threshold of the external electronic device 202, the wake-up signal is generated. For another example, the external electronic device 202 also includes a grip sensor (not shown). The external electronic device 202 generates a wake-up signal based on data obtained from the grip sensor indicating that the user has gripped the external electronic device 202.

[0147] At operation 923, the processor 250 predicts the position of the external electronic device 202 based on the information about the tilt of the external electronic device 202 corresponding to the time point of release of the sleep mode, the information about the tilt of the external electronic device 202 corresponding to the time point of entering the sleep mode, and / or the information about the position of the external electronic device 202 corresponding to the time point when entering the sleep mode. According to an embodiment, the processor 250 predicts the change in the position of the external electronic device 202 by comparing the difference between the information corresponding to the time of entering the sleep mode and / or the information corresponding to the release time among the information about the tilt of the external electronic device 202. The processor 250 obtains the position predicted to be at the time of releasing the sleep mode by applying the information about the change in the predicted position to the position information at the time of entering the sleep mode.

[0148] According to various embodiments, the processor 250 predicts the position based on the information about the tilt of the electronic device 201. For example, the electronic device 201 moves between the time point of entering the sleep mode and the time point of releasing the sleep mode. In order to reflect the movement of the electronic device 201, the processor 250 compares the information about the tilt of the electronic device 201 when the external electronic device 202 enters the sleep mode and / or the information about the tilt of the electronic device 201 when the external electronic device 202 releases the sleep mode, so that the processor 250 obtains the information about the movement (motion) of the electronic device 201.

[0149] At operation 925, the processor 250 determines whether the predicted position of the external electronic device is included in the field of view of the camera. When the predicted position of the external electronic device 250 is within the area of ​​the field of view of the camera 240, at operation 929, the processor 250 displays a graphic object at the predicted position of the external electronic device 202. When the predicted position of the external electronic device is outside the area of ​​the field of view of the camera 240, at operation 927, the processor 250 displays a graphic object for guiding the predicted position. The graphic object is not a graphic object corresponding to the external electronic device 202. For example, the graphic object includes an image in the shape of an arrow. For another example, the graphic object includes an image that is the same or similar in shape to the external electronic device 202.

[0150] Fig.10 2 shows an example of a graphic object for guiding the location of an external electronic device according to various embodiments. Hereinafter, the image 1010 and the image 1020 displayed on the display 230 include images corresponding to the field of view of the user. In various embodiments, the field of view of the camera 240 is the same as the field of view of the image 1010 and the image 1020 displayed on the display 230. However, the field of view of the camera 240 is not limited thereto, and as Figure 4b As shown in , it is smaller or larger than the user's field of view.

[0151] Reference Fig.10 , the processor 250 displays an image 1010 related to the multimedia content on the display 230. The image 1010 related to the multimedia content includes at least one of a panoramic image or a 3D image. In various embodiments, the processor 250 displays a graphic object for directing the location of the external electronic device 202. For example, the processor 250 displays a graphic object 1012 in the shape of an arrow. The arrow-shaped graphic object 1012 is displayed as being overlapped on the image related to the displayed multimedia content. According to an embodiment, the processor 250 changes the display of the arrow-shaped graphic object 1012. For example, when the external electronic device 202 is away from the electronic device 201, the processor 250 reduces the thickness of the graphic object 1012 in the shape of an arrow. For another example, when the distance between the external electronic device 202 and the electronic device 201 decreases, the processor 250 increases the thickness of the graphic object 1012 in the form of an arrow. In another embodiment, the processor 250 changes the color of the graphic object. The processor 250 changes the color of the graphic object based on the color of the multimedia content displayed on the display 230. For example, the processor 250 determines the display area of ​​the graphic object 1012 in the shape of an arrow, and identifies some areas of the image related to the multimedia content corresponding to the determined area. In order to improve the recognition ability of the graphic object 1012, the processor 250 identifies the color of the identified area and displays the graphic object 1012 in the shape of an arrow in a complementary color of the identified color.

[0152] In various embodiments, the processor 250 displays the visual effect 1022. The processor 250 displays the image 1020 associated with the multimedia content superimposed with the visual effect 1022. For example, the processor 250 displays the visual effect 1022 in a boundary surface in which the direction in which the external electronic device 202 is located is predicted among the four boundary surfaces forming the boundary of the display 230. The visual effect 1022 includes an effect of highlighting the boundary surface. Although not shown, the processor 250 includes a visual effect related to blurring for blurring the boundary surface. In the above embodiment, the graphic object 1012 has been described as including an image in the shape of an arrow, but is not limited thereto. For example, the graphic object 1012 includes an image of the same shape or a similar shape as the external electronic device 202.

[0153] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0154] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalents or replacements for corresponding embodiments. For the description of the accompanying drawings, similar figure numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each phrase in the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together in the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., the second element)”, “combined to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” when the terms “operably” or “communicably” are used or when the terms “operably” or “communicably” are not used, this means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0155] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or the smallest unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0156] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without one or more other components. This enables the machine to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in a storage medium and data being temporarily stored in a storage medium.

[0157] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be published online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM), or may be published (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least a portion of the computer program product may be temporarily generated, or at least a portion of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0158] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

Claims

1. An electronic device, comprising: a housing including a first surface and a second surface on an opposite side of the first surface; one or more cameras having a specified field of view, wherein the one or more cameras are exposed through at least a portion of the first surface of the housing; a display, wherein the display is exposed through at least a portion of the second surface of the housing; communications circuits; and processor, The processor is configured as follows: using the one or more cameras to identify an external electronic device in one or more external objects included in the specified field of view, displaying a graphic object corresponding to the external electronic device on a display according to first position information of the external electronic device recognized based at least on image information obtained using the one or more cameras, and When the external electronic device is outside the designated field of view, the graphic object is displayed on the display based on second position information of the external electronic device identified using the one or more cameras before the external electronic device is outside the designated field of view and information related to the movement of the external electronic device received from the external electronic device via the communication circuit after the external electronic device is outside the designated field of view.

2. The electronic device according to claim 1, wherein: The external electronic device further includes: a first sensor including a gyro sensor; and a second sensor including an acceleration sensor, and The information related to the movement of the external electronic device includes at least one of data related to the tilt of the external electronic device obtained by the first sensor or data related to the speed change of the external electronic device obtained by the second sensor.

3. The electronic device according to claim 1, wherein: The external electronic device further comprises a light emitting portion, wherein the light emitting portion comprises one or more light emitting diodes LED, and The processor is configured to: use the one or more cameras to acquire an image including at least a portion of the light-emitting portion, estimate a position associated with the light-emitting portion based at least on image calculations for the image, and determine first position information associated with an external electronic device based at least on the estimated position.

4. The electronic device according to claim 2, wherein: The processor is also configured to: When the external electronic device is located within the specified field of view, identifying a display position of the graphic object based at least on the first position information and the data related to the tilt of the external electronic device, When the external electronic device is outside the specified field of view, identifying a display position of the graphic object based at least on the second position information and the third position information, and displaying the graphic object based on the identified display position, The third position information includes vector information obtained based on the data related to the tilt of the external electronic device and the data related to the speed change of the external electronic device.

5. The electronic device according to claim 1, wherein: The processor is configured to display the graphic object based on multimedia content related to virtual reality provided by the electronic device, the first position information, and the second position information.

6. The electronic device according to claim 1, wherein: The processor is configured to: if the external electronic device is not recognized within the specified field of view, receive from the external electronic device one or more third location information acquired during a specified time range before the external electronic device enters a low power mode, and When the display request of the graphic object is confirmed, a display position of the graphic object to be displayed on a display is determined based on at least the one or more third position information.

7. The electronic device according to claim 1, wherein: The processor is configured as: Identify that the external electronic device has entered sleep mode, receiving third position information of the external electronic device corresponding to a time point of entering the sleep mode and information related to a tilt of the external electronic device corresponding to the time point of entering the sleep mode, Recognize that sleep mode is released, receiving information related to the tilt of the external electronic device corresponding to a time point at which the sleep mode is released, A second graphic object for indicating the position of the external electronic device is displayed based at least on the third position information of the external electronic device and the information related to the tilt of the external electronic device corresponding to the entry time point and the information related to the tilt of the external electronic device corresponding to the release time point.

8. The electronic device according to claim 7, wherein: The second graphic object includes an image in the same or similar shape as that of the external electronic device or an image in an arrow shape.

9. A method of an electronic device, comprising: using one or more cameras having a specified field of view to identify an external electronic device in one or more external objects included in the specified field of view; displaying a graphic object corresponding to the external electronic device on a display according to first position information of the external electronic device identified based at least on image information obtained by the one or more cameras; and When the external electronic device is outside the designated field of view, the graphic object is displayed on the display based on second position information of the external electronic device identified using the one or more cameras before the external electronic device is outside the designated field of view and information related to the movement of the external electronic device received from the external electronic device via the communication circuit after the external electronic device is outside the designated field of view.

10. The method according to claim 9, wherein: The information related to the movement of the external electronic device includes at least one of data related to the tilt of the external electronic device obtained by a first sensor included in the external electronic device or data related to a speed change of the external electronic device obtained by a second sensor included in the external electronic device.

11. The method according to claim 10, further comprising: When the external electronic device is located within the specified field of view, identifying a display position of the graphic object based at least on the first position information and the data related to the tilt of the external electronic device; When the external electronic device is outside the specified field of view, identifying a display position of the graphic object based at least on the second position information and the third position information; and Based on the identified display position, displaying the graphic object, The third position information includes vector information obtained based on the data related to the tilt of the external electronic device and the data related to the speed change of the external electronic device.

12. The method according to claim 9, further comprising: using the one or more cameras to obtain an image including at least a portion of a light emitting portion included in an external electronic device; estimating a position relative to the light emitting portion based at least on image operations on the image; and First location information associated with the external electronic device is determined based at least on the estimated location.

13. The method according to claim 9, wherein: The step of displaying the graphic object on the display includes: displaying the graphic object based on multimedia content related to virtual reality, first position information and second position information.

14. The method according to claim 9, further comprising: When the external electronic device is not recognized within the designated field of view, receiving from the external electronic device one or more third location information acquired during a designated time range before the external electronic device enters a low power mode; and When the display request of the graphic object is confirmed, a display position of the graphic object to be displayed on a display is determined based on at least the one or more third position information.

15. The method according to claim 9, further comprising: Identify that the external electronic device has entered sleep mode; receiving third position information of the external electronic device corresponding to a time point of entering the sleep mode and information related to a tilt of the external electronic device corresponding to the time point of entering the sleep mode; Recognize that sleep mode has been released; receiving information related to a tilt of the external electronic device corresponding to a release time point of the sleep mode; and A second graphic object for indicating the position of the external electronic device is displayed based at least on the third position information of the external electronic device and the information related to the tilt of the external electronic device corresponding to the entry time point and the information related to the tilt of the external electronic device corresponding to the release time point.

16. An electronic device comprising: case; Memory; a camera having a designated field of view and visually exposed through at least a portion of the first surface of the housing; a display exposed through at least a portion of a second surface of the housing different from the first surface; Communication module; as well as processor, The processor is configured as follows: using a camera to obtain information related to a location of an external electronic device, wherein the external electronic device is interlocked with the electronic device, receiving information related to the tilt of the external electronic device from the external electronic device using the communication module, identifying a graphical object based on at least one of the position-related information or the tilt-related information, providing the graphic object for representing the external electronic device and multimedia content related to virtual reality using a display, using the camera to identify that the external electronic device is deviating from the designated field of view of the camera, In response to identifying the deviation, transmitting a signal for requesting the external electronic device to transmit information related to the change in position of the external electronic device, and Display of the graphic object is changed based on at least one of the information related to the position of the external electronic device at the offset time obtained using the camera, the information related to the tilt of the external electronic device, or the information related to the position change of the external electronic device.

17. The electronic device according to claim 16, wherein: The information related to the change in position of the external electronic device is determined based on data obtained by a sensor included in the external electronic device, and The sensor includes at least one of an acceleration sensor or a gyroscope sensor.

18. The electronic device according to claim 16, wherein: The processor is configured to activate the acceleration sensor in response to recognizing that the external electronic device is offset from the designated field of view of the camera.

19. The electronic device according to claim 16, wherein: The processor is configured to obtain a position to which the external electronic device is predicted to have moved based at least on the information related to the position of the external electronic device at the deviation time, the information related to the tilt of the external electronic device, and the information related to the position change of the external electronic device.

20. The electronic device according to claim 16, wherein: The graphic object is displayed overlaid on the multimedia content related to virtual reality, and The multimedia content includes content for providing virtual reality services.

Images