Electronic device for recognizing control signal based on motion information and control method thereof

By using inertial measurement sensors and processors in portable devices to identify user locations and gazes, combined with lookup tables, the problem of difficulty in identifying user locations in large screen environments is solved, and efficient user interaction control is achieved.

CN119096222APending Publication Date: 2024-12-06SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380036936.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-09-22
Filing Date
2023-05-15
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In large or immersive screen environments, there are difficulties in using optical sensors for user position and posture tracking, especially when the user is required to move freely, and traditional methods require complex equipment.

Method used

By measuring motion information using an inertial measurement sensor in a portable device worn by the user, and using a processor to identify the device orientation and user position, combined with a lookup table to identify distances, the identification and control of user position and gaze can be achieved without the need for an optical sensor.

Benefits of technology

It realizes accurate identification of user location and control signals without the need for optical sensors in large screens or immersive screen environments, improving the convenience and usability of user interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096222A_ABST
    Figure CN119096222A_ABST
Patent Text Reader

Abstract

An electronic device is disclosed. The electronic device includes a projection unit, a communication interface, and at least one processor connected to the projection unit and the communication interface and controlling the electronic device. The processor is used for controlling the projection unit to project the image onto the projection area; receiving first motion information from each of the first mobile device, the second mobile device, and the wearable device via the communication interface after the first mobile device, the second mobile device, and the wearable device worn by the user are arranged to correspond to the image; identifying an orientation of each of the first mobile device, the second mobile device, and the wearable device based on the first motion information; and a location of the user may be identified based on the identified orientation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that recognizes a control signal based on motion information and a control method thereof. Background Art

[0002] With the development of electronic technology, various types of electronic devices are being developed. In particular, electronic devices for providing content to users through large screens or immersive screens are currently being developed.

[0003] Users can control electronic devices through portable devices. Portable devices can use optical sensors, such as infrared radiation (IR) to estimate the pointing direction. However, in spaces including large screens, immersive screens, or multi-projector displays, where users can move freely, there are limitations to using optical sensors that require line of sight (LOS) to operate.

[0004] Furthermore, in order to interact with the content displayed in the wide space, the user's position (or posture) may be required. However, there is a problem that a camera or complex sensors are required to track the user's position. Summary of the invention

[0005] According to one or more embodiments, an electronic device includes: a projection circuit; a communication interface; and at least one processor connected to the projection circuit and the communication interface and configured to control the electronic device, wherein the processor is configured to: control the projection circuit to project an image onto a projection area; after a first portable device, a second portable device, and a wearable device worn by a user are arranged to correspond to the image, receive first motion information from each of the first portable device, the second portable device, and the wearable device through the communication interface; identify the orientation of each of the first portable device, the second portable device, and the wearable device based on the first motion information; and identify the user's position based on the identified orientation.

[0006] The processor is configured to, after identifying the location of the user, receive second motion information from each of the first portable device, the second portable device, and the wearable device through the communication interface, and update the location of the user based on the second motion information.

[0007] The electronic device also includes: a memory storing a lookup table, wherein the processor is configured to: identify a first angle and a second angle, the first angle being formed by a first straight line moving from the user to the front of the user and a second straight line parallel to the orientation of the first portable device, and the second angle being formed by a third straight line moving from the user to the front of the user and a fourth straight line parallel to the orientation of the second portable device; identify distances corresponding to the first angle and the second angle based on the lookup table; and identify the user's position based on the identified distances.

[0008] Each of the first portable device, the second portable device, and the wearable device includes an inertial measurement sensor, and the first motion information includes: information related to at least one of roll, pitch, or yaw measured by the inertial measurement sensor of each of the first portable device, the second portable device, and the wearable device.

[0009] The processor is configured to control the projection circuit to project the pointer toward a point of the projection area based on at least one orientation of the first portable device or the second portable device.

[0010] The processor is configured to: identify a gaze of the user based on first motion information received from the wearable device, and identify an area in the projection area corresponding to the user's field of view based on the user's gaze.

[0011] The processor is configured to control the projection circuit to project the pointer toward an edge of the one area based on a point of the projection area corresponding to at least one orientation of the first portable device or the second portable device falling outside the one area.

[0012] The processor is configured to change the size of the one area based on receiving a preset control signal from at least one of the first portable device or the second portable device through the communication interface while the pointer is in a state of being projected toward an edge of the one area.

[0013] The projection area is implemented as a screen having at least one projection surface whose size is greater than or equal to a preset size, or is implemented as an immersive screen including at least two projection surfaces.

[0014] The first portable device and the second portable device are each held by a user, and the wearable device is configured to be worn on a head of the user.

[0015] An electronic device comprises: a display; a communication interface; and at least one processor connected to the display and the communication interface and configured to control the electronic device, wherein the processor is configured to: control the display to display an image; after a first portable device, a second portable device, and a wearable device worn by a user are arranged to correspond to the image, receive first motion information from each of the first portable device, the second portable device, and the wearable device through the communication interface; identify the orientation of each of the first portable device, the second portable device, and the wearable device based on the first motion information; and identify the user's position based on the identified orientation.

[0016] A control method for an electronic device, the method comprising: projecting an image onto a projection area; receiving first motion information from each of a first portable device, a second portable device, and a wearable device worn by a user after the first portable device, the second portable device, and the wearable device are arranged to correspond to the image; identifying an orientation of each of the first portable device, the second portable device, and the wearable device based on the first motion information; and identifying a position of the user based on the identified orientation.

[0017] The method also includes: after identifying the location of the user, receiving second motion information from each of the first portable device, the second portable device, and the wearable device; and updating the location of the user based on the second motion information.

[0018] Identifying the location of a user includes: identifying a first angle and a second angle, the first angle being formed by a first straight line moving from the user to the front of the user and a second straight line parallel to the orientation of the first portable device, and the second angle being formed by a third straight line moving from the user to the front of the user and a fourth straight line parallel to the orientation of the second portable device; identifying distances corresponding to the first angle and the second angle based on a lookup table; and identifying the location of the user based on the identified distances.

[0019] The first motion information includes information about at least one of roll, pitch, or yaw measured by an inertial measurement sensor of each of the first portable device, the second portable device, and the wearable device.

[0020] The method also includes projecting a pointer to a point of the projection area based on at least one orientation of the first portable device or the second portable device.

[0021] The method also includes: identifying a gaze of the user based on first motion information received from the wearable device; and identifying an area in the projection area corresponding to the user's field of view based on the user's gaze.

[0022] The method further includes projecting the pointer toward an edge of the one area based on a point of the projection area corresponding to at least one orientation of the first portable device or the second portable device falling outside the one area.

[0023] The method further includes changing a size of the one area based on receiving a preset control signal from at least one of the first portable device or the second portable device while the pointer is in a state of being projected toward an edge of the one area.

[0024] The projection area is implemented as a screen having at least one projection surface whose size is greater than or equal to a preset size, or is implemented as an immersive screen including at least two projection surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a block diagram illustrating an electronic system according to one or more embodiments;

[0026] Figure 2 is a block diagram showing a configuration of an electronic device according to one or more embodiments;

[0027] Figure 3 is a block diagram showing a detailed configuration of an electronic device according to one or more embodiments;

[0028] Figure 4 is a diagram showing the overall operation of an electronic device according to one or more embodiments;

[0029] Figure 5 is a flowchart illustrating the overall operation of an electronic device according to one or more embodiments;

[0030] Figure 6 is a diagram showing a portable device according to one or more embodiments;

[0031] Figure 7 is a diagram illustrating a projection area near a user according to one or more embodiments;

[0032] Figure 8 and Fig. 9 is a diagram illustrating a method of identifying a user's location according to one or more embodiments;

[0033] Fig.10 and Fig.11 is a diagram showing an area corresponding to a user's field of view in a projection area according to one or more embodiments;

[0034] Fig.12 and Fig.13 is a diagram showing an area in a projection area corresponding to a field of view of a user and a projection position of a pointer according to one or more embodiments;

[0035] Fig.14 are diagrams showing detailed configurations of an electronic device and a portable apparatus according to one or more embodiments; and

[0036] Fig.15 is a flowchart illustrating a control method of an electronic device according to one or more embodiments. DETAILED DESCRIPTION

[0037] The exemplary embodiments of the present disclosure may be modified in various ways. Therefore, specific exemplary embodiments are shown in the drawings and described in detail in the specific embodiments. However, it should be understood that the present disclosure is not limited to specific exemplary embodiments, but includes all modifications, equivalents and substitutions without departing from the scope and spirit of the present disclosure. In addition, well-known functions or configurations are not described in detail because they would obscure the present disclosure with unnecessary details.

[0038] An aspect of the present disclosure is to provide a portable device / wearable device that transmits motion information without using an optical sensor, an electronic apparatus that recognizes a control signal based on motion information received therefrom, and a control method thereof.

[0039] The present disclosure will be described in detail below with reference to the accompanying drawings.

[0040] The terms used to describe one or more embodiments of the present disclosure are currently widely used general terms selected in consideration of their functions in this article. However, these terms may change according to the intentions of technicians in the relevant field, legal or technical interpretations, the emergence of new technologies, etc. In addition, in some cases, there may be arbitrarily selected terms, and in this case, the meaning of the term will be disclosed in more detail in the corresponding description. Therefore, the terms used herein should not be simply understood as their names, but should be understood based on the meaning of the term and the overall context of the present disclosure.

[0041] In the present disclosure, expressions such as “having”, “may have”, “including”, “may include”, etc. are used to specify the existence of corresponding characteristics (for example, elements such as values, functions, operations or components), but do not exclude the existence or possibility of additional characteristics.

[0042] The expression “at least one of A and / or B” should be understood to mean any one of “A” or “B” or “A and B”.

[0043] As used herein, expressions such as "first", "second", "1st", "2nd" and the like may be used to refer to various elements, regardless of their order and / or importance. In addition, it should be noted that these expressions are only used to distinguish an element from another element, without limiting the related elements.

[0044] Unless otherwise specified, singular expressions include plural expressions. It should be understood that terms such as "formed" or "comprising" are used herein to specify the presence of features, quantities, steps, operations, elements, components or combinations thereof, without excluding the possibility of the presence or addition of one or more other features, quantities, steps, operations, elements, components or combinations thereof.

[0045] In the present disclosure, the term “user” may refer to a person using an electronic device or an apparatus (eg, an artificial intelligence electronic device) using an electronic device.

[0046] Various embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0047] Figure 1 1 is a block diagram showing an electronic system 1000 according to one or more embodiments. Figure 1 As shown, the electronic system 1000 may include an electronic device 100, a first portable device 200-1, a second portable device 200-2, and a wearable device 300. However, the one or more embodiments are not limited thereto, and the electronic system 100 may include the electronic device 100, and may further include at least one of the first portable device 200-1, the second portable device 200-2, or the wearable device 300.

[0048] The electronic device 100 may receive motion information from at least one of the first portable device 200-1, the second portable device 200-2, or the wearable device 300. Based on the received motion information, the electronic device 100 may identify the orientation of the device corresponding to the motion information based on the motion information. In one or more examples, the orientation may be posture information. The electronic device 100 may identify the position of the user based on the orientation information of each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300.

[0049] The electronic apparatus 100 may recognize a control command based on the orientation information of each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300. For example, the electronic apparatus 100 may recognize a control command based on a change in the orientation information of each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300, and perform an operation corresponding to the recognized control command.

[0050] The electronic device 100 can recognize a control command based on one of the orientation information of the first portable device 200-1, the second portable device 200-2, and the wearable device 300, and perform an operation corresponding to the recognized control command. In one or more examples, the electronic device 100 can recognize a control command by combining the orientation information of at least two of the first portable device 200-1, the second portable device 200-2, and the wearable device 300.

[0051] The electronic device 100 may be a device that projects an image to a projection area through a projection portion. In one or more examples, the electronic device 100 may be a projector. In one or more examples, the electronic device 100 may be a device that displays an image through a display, and may be a television (TV) or any other electronic device known to those of ordinary skill in the art.

[0052] However, as understood by those of ordinary skill in the art, the embodiments are not limited to these configurations, and the electronic device 100 may receive motion information from at least one of the first portable device 200-1, the second portable device 200-2, or the wearable device 300, and may be any device configured to perform a corresponding operation. In one or more examples, the electronic device 100 may receive the motion information. In one or more examples, the device that provides a screen corresponding to the motion information to a projector, a TV, etc. may be implemented as a smart phone, a tablet computer, augmented reality (AR) glasses, a desktop personal computer (PC), a notebook, or any other device with a screen known to those of ordinary skill in the art.

[0053] To facilitate the following description, the electronic device 100 implemented as a projector will be described.

[0054] Each of the first portable device 200-1 and the second portable device 200-2 may obtain motion information of each of the first portable device 200-1 and the second portable device 200-2, respectively, through an inertial measurement sensor (or an inertial measurement unit (IMU) sensor). For example, each of the first portable device 200-1 and the second portable device 200-2 may obtain information about at least one of roll, pitch, or yaw of each of the first portable device 200-1 and the second portable device 200-2, respectively, through an inertial measurement sensor. Each of the first portable device 200-1 and the second portable device 200-2 may have a corresponding inertial measurement sensor mounted thereon.

[0055] The first portable device 200-1 and the second portable device 200-2 may transmit the motion information to the electronic apparatus 100. For example, the first portable device 200-1 and the second portable device 200-2 may transmit the motion information to the electronic apparatus 100 through Bluetooth communication or Wi-Fi communication.

[0056] The first portable device 200-1 may be implemented in a form held by the user's left hand, and the second portable device 200-2 may be implemented in a form held by the user's right hand. However, as will be appreciated by those of ordinary skill in the art, the embodiments are not limited to these configurations, and the first portable device 200-1 and the second portable device 200-2 may be configured in any form or shape that can be held by the user.

[0057] The first portable device 200-1 and the second portable device 200-2 may be implemented in a combinable or separable form. For example, a magnet may be provided at a combining portion of the first portable device 200-1 and the second portable device 200-2, where attachment and detachment of the devices may be performed.

[0058] The wearable device 300 may obtain motion information of the wearable device 300 through the inertial measurement sensor. For example, the wearable device 300 may obtain information related to at least one of roll, pitch, or yaw of the wearable device 300 through the inertial measurement sensor.

[0059] The wearable device 300 may send the motion information to the electronic apparatus 100. For example, the wearable device 300 may send the motion information to the electronic apparatus 100 through Bluetooth communication or Wi-Fi communication.

[0060] The wearable device 300 may be implemented in the form of a pair of earphones to be worn in the user's ears. However, as will be appreciated by those of ordinary skill in the art, the embodiments are not limited to these configurations, and the wearable device 300 may take any form or shape configured to be worn on the user's head. In addition, the wearable device 300 may be implemented in a form that can be worn on the user's body.

[0061] Figure 2 is a block diagram showing a configuration of an electronic device 100 according to one or more embodiments. Figure 2 , the electronic device 100 may include a projection part 110 , a communication interface 120 and a processor 130 .

[0062] The projection part 110 may be a circuit configured to project an image to a projection area. Specifically, the projection part 110 may use a light source such as a lamp or a light emitting diode (LED) and project an image including at least one of content received from a source device and pre-stored content at the projection area.

[0063] The communication interface 120 may be configured to perform communication with various types of external devices according to various types of communication methods. For example, the electronic apparatus 100 may perform communication with the first portable device 200-1, the second portable device 200-2, or the wearable device 300 through the communication interface 120.

[0064] The communication interface 120 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, or any other communication module known to those skilled in the art. In one or more examples, each communication module may be implemented in the form of at least one hardware chip.

[0065] The Wi-Fi module and the Bluetooth module can perform communication in the Wi-Fi method and the Bluetooth method, respectively. When using the Wi-Fi module or the Bluetooth module, various connection information such as a service set identifier (SSID), a session key, etc. can be sent and received first. After the communication connection is made using the connection information, various information can be sent and received. The infrared communication module can perform communication according to the infrared communication (Infrared Data Association (IrDA)) technology that wirelessly sends data within a short distance by using infrared rays existing between visible light and millimeter waves.

[0066] The wireless communication module may include at least one communication chip for performing communication according to various wireless communication standards in addition to the above-mentioned communication methods, and the various wireless communication standards are, for example, but not limited to, ZigBee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), advanced LTE (LTE-A), fourth generation (4G), fifth generation (5G), or any other communication standard known to a person of ordinary skill in the art.

[0067] In one or more examples, communication interface 120 may include a wired communication interface such as, but not limited to, HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, or any other communication interface known to those of ordinary skill in the art.

[0068] Besides, the communication interface 120 may further include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module to perform communication using a twisted pair cable, a coaxial cable, an optical fiber cable, or any other cable known to those skilled in the art.

[0069] The processor 130 may control the overall operation of the electronic device 100. For example, the processor 130 may control the overall operation of the electronic device 100 by connecting with one or more components of the electronic device 100. For example, the processor 130 may be connected with one or more components such as the projection part 110, the communication interface 120, and the memory, and control the operation of the electronic device 100.

[0070] At least one processor 130 may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a majority integrated core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. At least one processor 130 may control one of the elements of the electronic device 100 or a random combination thereof, and perform operations associated with communication or data processing. At least one processor 130 may execute one or more programs or instructions stored in a memory. For example, at least one processor 130 may execute a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0071] If the method according to one or more embodiments of the present disclosure is included in multiple operations, the multiple operations may be performed by one processor, or by multiple processors. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor), and the third operation may be performed by the second processor (e.g., an artificial intelligence dedicated processor).

[0072] At least one processor 130 may be implemented as a single-core processor including one core, or may be implemented as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). If at least one processor 130 is implemented as a multi-core processor, each core included in the multi-core processor may include an internal memory of the processor (such as a cache memory or an on-chip memory), and a common cache shared by multiple cores may be included in the multi-core processor. In addition, each core (or a portion of the multiple cores) included in the multi-core processor may independently read a program command for implementing the method according to one or more embodiments of the present disclosure and execute the program command, or read a program command for implementing the method according to one or more embodiments of the present disclosure in conjunction with all (or a portion) of the multiple cores and execute the program command.

[0073] If the method according to one or more embodiments of the present disclosure is included in multiple operations, the multiple operations may be performed by one of the multiple cores included in the multi-core processor, or by multiple cores. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first core included in the multi-core processor, or the first operation and the second operation may be performed by the first core included in the multi-core processor, and the third operation may be performed by the second core included in the multi-core processor.

[0074] According to one or more embodiments of the present disclosure, at least one processor 130 may represent one or more processors and a system on chip (SoC) integrated with other electronic components, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, and the core referred to herein may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, a machine learning accelerator, or any other core processor structure known to a person of ordinary skill in the art. However, the embodiments of the present disclosure are not limited to these configurations. However, for ease of description, the term processor 130 will be used to describe the operation of the electronic device 100.

[0075] The processor 130 can control the projection part 110 to project an image to the projection area, and after the first portable device 200-1, the second portable device 200-2, and the wearable device 300 worn by the user are arranged to correspond to the image, the processor 130 can also control the projection part 110 to receive the first motion information from each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 through the communication interface 120. For example, the processor 130 can control the projection part 110 to project an image including three points at the projection area. The user can focus on the center point among the three points, and arrange the first portable device 200-1 to face the left point, and arrange the second portable device 200-2 to face the right point. In addition, the first portable device 200-1, the second portable device 200-2, and the wearable device 300 can send the corresponding first motion information to the electronic device 100. In one or more examples, the first motion information may include information about at least one of roll, pitch, or yaw measured by an inertial measurement sensor of each of the first portable device 200 - 1 , the second portable device 200 - 2 , and the wearable device 300 .

[0076] The electronic device 100 may identify the orientation of each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 based on the first motion information, and identify the user's position based on the identified orientation. For example, the electronic device 100 may also include a memory storing a lookup table, and the processor 130 may identify a first angle and a second angle, the first angle being formed by a straight line moving from the user to the front of the user and a straight line parallel to the orientation of the first portable device 200-1, and the second angle being formed by a straight line moving from the user to the front of the user and a straight line parallel to the orientation of the second portable device 200-2. The processor 130 may identify the distance corresponding to the first angle and the second angle based on the lookup table, and identify the user's position based on the identified distance. For example, the lookup table may be a table including a first angle and a second angle for each distance from the projection area.

[0077] The processor 130 may receive second motion information from each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 through the communication interface 120 after identifying the location of the user, and update the location of the user based on the second motion information. For example, the processor 130 may update the location of the user based on the second motion information of the wearable device 300, and perform operations corresponding to the second motion information of the first portable device 200-1 and the second portable device 200-2. In one or more examples, the processor 130 may perform operations corresponding to the second motion information of the first portable device 200-1 and the second portable device 200-2 based on the updated location of the user.

[0078] The processor 130 may control the projection part 110 to project a pointer at a point of the projection area based on at least one orientation of the first portable device 200-1 or the second portable device 200-2. The processor 130 may recognize the user's gaze based on the first motion information received from the wearable device 300, and recognize an area in the projection area corresponding to the user's field of view based on the user's gaze. The processor 130 may control the projection part 110 to project the pointer to the edge of an area based on a point of the projection area corresponding to at least one orientation of the first portable device 200-1 or the second portable device 200-2 falling outside the area.

[0079] However, as understood by those of ordinary skill in the art, the embodiments are not limited to these configurations, and the processor 130 may control the projection part 110 to project a flashing pointer toward the edge of the one area based on that a point of the projection area corresponding to at least one orientation of one of the first portable device 200-1 or the second portable device 200-2 falls outside the one area. In one or more examples, the processor 130 may project the pointer toward the edge of the one area and provide a notification sound through a speaker.

[0080] The processor 130 may change the size of the one area based on receiving a preset control signal from at least one of the first portable device 200-1 or the second portable device 200-2 through the communication interface 120 while the pointer is in a state of being projected at the edge of the one area.

[0081] The projection area may be implemented as a screen having at least one projection surface greater than or equal to a preset size or an immersive screen including at least two projection surfaces.

[0082] If the electronic device 100 is implemented in a form including a display, the processor 130 may control the display to display an image. The processor 130 may receive first motion information from each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 worn by the user after being arranged to correspond to the image through the communication interface 120. The processor 130 may identify the orientation of each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 based on the first motion information. In addition, the processor 130 may identify the position of the user based on the identified orientation.

[0083] Figure 3 is a block diagram illustrating a detailed configuration of the electronic device 100 according to one or more embodiments.

[0084] Figure 3 1 is a block diagram showing a detailed configuration of an electronic device 100 according to one or more embodiments. The electronic device 100 may include a projection portion 110, a communication interface 120, and a processor 130. Figure 3 , the electronic device 100 may further include a memory 140, a display 150, a user interface 160, a microphone 170, and a speaker 180. Figure 3 The components shown are Figure 2 Detailed description of the overlapping parts of the components shown.

[0085] The memory 140 may refer to hardware that stores information such as data in an electrical or magnetic form for access to the data by the processor 130 or any other memory structure known to a person of ordinary skill in the art. For example, the memory 140 may be implemented as at least one of a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), a random access memory (RAM), a read-only memory (ROM), or any other memory structure known to a person of ordinary skill in the art.

[0086] In the memory 140, at least one instruction required for the operation of the electronic device 100 or the processor 130 may be stored. For example, the instruction may be a code unit that instructs the operation of the electronic device 100 or the processor 130, and may be prepared in a machine language, which is a language that a computer can understand. In one or more examples, the memory 140 may store a plurality of instructions for performing a specific job of the electronic device 100 or the processor 130 as an instruction set.

[0087] The memory 140 may store data as information in bits or bytes (which may represent characters, numbers, images), or any other type of data known to those of ordinary skill in the art. For example, the memory 140 may store a lookup table for identifying the location of a user, information about operations based on motion information, or any other suitable information known to those of ordinary skill in the art.

[0088] Memory 140 may be accessed by processor 130 , and reading, writing, modifying, deleting, updating, or any other suitable operation of instructions, instruction sets, or data may be performed by processor 130 .

[0089] The display 150 may be a configuration for displaying an image, and may be implemented as various forms of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, and a plasma display panel (PDP). In the display 150, a driving circuit, a backlight unit, or any other display structure known to a person skilled in the art may be included, and the driving circuit may be implemented in the form of an a-si TFT, a low temperature polycrystalline silicon (LTPS) TFT, an organic TFT (OTFT), or any other transistor structure known to a person skilled in the art. The display 150 may be implemented as a touch screen coupled to a touch sensor, a flexible display, a three-dimensional display (3D display), or any other display structure known to a person skilled in the art.

[0090] Even if the electronic apparatus 100 is implemented as a projector, a separate display 150 may be further included, and the processor 130 may provide information on the operating state of the electronic apparatus 100 through the display 150 .

[0091] The user interface 160 may be implemented as a button, a touch pad, a mouse, a keyboard, etc., or may be implemented as a touch screen configured to perform a display function and an operation input function together. In one or more examples, the button may be various types of buttons such as a mechanical button, a touch pad, or a scroll wheel formed at a random area of ​​a front surface portion or a side surface portion, a rear surface portion, or any other suitable position of the exterior of the body of the electronic device 100.

[0092] The microphone 170 may be a configuration for receiving sound and converting it into an audio signal. The microphone 170 may be electrically connected to the processor 130 and may receive sound by the control of the processor 130.

[0093] For example, the microphone 170 may be formed as an integrated type integrated into the upper side or the front surface direction, the side surface direction, etc. of the electronic device 100. In one or more examples, the microphone 170 may be provided in a separate remote controller or any other suitable controller from the electronic device 100. In this case, the remote controller may receive sound through the microphone 170 and provide the received sound to the electronic device 100.

[0094] Microphone 170 may include various configurations, such as a microphone that collects sound in analog form, an amplifier circuit that amplifies the collected sound, an A / D converter circuit that samples the amplified sound and converts it into a digital signal, a filter circuit that removes noise components from the converted digital signal, or any other microphone components known to a person of ordinary skill in the art.

[0095] The microphone 170 may be implemented in the form of a sound sensor and may be implemented by any method known to those of ordinary skill in the art for collecting sound.

[0096] The speaker 180 may be a component that outputs not only various audio data processed by the processor 130 but also various notification sounds, voice messages, or any other suitable sounds.

[0097] Each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 may include an inertial measurement sensor, and the first motion information may include information related to at least one of roll, pitch, or yaw measured by the inertial measurement sensor of each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300.

[0098] For example, each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 may include at least one of a gyro sensor, an acceleration sensor, or a magnetometer sensor.

[0099] A gyro sensor is a sensor for detecting the rotation angle of a device, and can measure changes in the orientation of an object using the following characteristics: a specific direction initially set is always maintained with high accuracy regardless of the rotation of the earth. A gyro sensor may be referred to as a gyroscope, and is implemented in a mechanical method or in an optical method using light.

[0100] The gyro sensor can measure angular velocity. Angular velocity may refer to the rotation angle per unit time, and the measurement principle of the gyro sensor is as follows. For example, the angular velocity in the horizontal state (stationary state) may be 0 degrees / second, and then, if the object tilts 50 degrees while moving for 10 seconds, the average angular velocity during 10 seconds may be 5 degrees / second. If a tilt angle of 50 degrees has been maintained in the stationary state, the angular velocity may be 0 degrees / second. Through this process, the angular velocity changes according to 0→5→0, and the angle increases from 0 degrees to 50 degrees. In order to obtain the angle from the angular velocity, the entire time may be integrated. Since the gyro sensor measures the angular velocity as described above, the tilt angle may be calculated by integrating the angular velocity over the entire time. However, the gyro sensor may generate an error due to the influence of temperature, and the error may accumulate during the integration process, resulting in a drift in the final value. Therefore, the device may also include a temperature sensor, and the error of the gyro sensor may be compensated by using the temperature sensor.

[0101] An acceleration sensor may be a sensor that measures the acceleration of a device or the intensity of an impact, and may be referred to as an accelerometer. An acceleration sensor may detect one or more dynamic forces, such as, but not limited to, acceleration, vibration, impact, or any other suitable force. Depending on the detection method, the acceleration sensor may be implemented in an inertial method, a gyroscope method, a silicon semiconductor method, or any other suitable method. For example, an acceleration sensor may be a sensor that uses gravity acceleration to sense the inclination of a device, and may be formed as a 2-axis fluxgate or a 3-axis fluxgate.

[0102] A magnetometer sensor may be a sensor that measures the strength and direction of the earth's magnetism. In a broader sense, a magnetometer sensor may include a sensor that measures the inherent magnetization of an object and may be referred to as a magnetometer. A magnetometer sensor may be implemented in the following manner: the strength of a magnetic field is measured by suspending a magnet horizontally in a magnetic field and measuring the direction in which the magnet moves or by rotating a coil in a magnetic field and measuring the induced electromotive force generated in the coil.

[0103] For example, a geomagnetic sensor (as a type of magnetometer sensor) that measures geomagnetic intensity can be implemented as a fluxgate-type geomagnetic sensor that uses a fluxgate to detect geomagnetism. A fluxgate-type geomagnetic sensor can be a device that measures the magnitude and direction of an external magnetic field by using a high magnetic permeability material such as permalloy as a magnetic core, applying an excitation magnetic field by winding a coil around the magnetic core, and measuring a second harmonic component that is proportional to the external magnetic field generated according to the magnetic saturation and nonlinear magnetic properties of the magnetic core. The current azimuth can be detected by measuring the magnitude and direction of the external magnetic field, and the degree of rotation is measured accordingly. The geomagnetic sensor can be formed by a 2-axis fluxgate or a 3-axis fluxgate. A 2-axis fluxgate sensor (i.e., a 2-axis sensor) can correspond to a sensor formed by an X-axis fluxgate and a Y-axis fluxgate that are orthogonal to each other, and a 3-axis fluxgate (i.e., a 3-axis sensor) can correspond to a sensor that adds a Z-axis fluxgate to an X-axis fluxgate and a Y-axis fluxgate.

[0104] When the geomagnetic sensor and the acceleration sensor are used as described above, the motion information of the device can be obtained, and the posture can be obtained through the motion information. For example, the motion information can be expressed as a change in roll, a change in pitch, and a change in yaw, and the posture information can be expressed as a roll angle, a pitch angle, and an azimuth angle.

[0105] The azimuth (yaw angle) may correspond to an angle changed in the left-right direction on the horizontal plane, and when the azimuth is calculated, the direction the device is facing may be known. For example, when a geomagnetic sensor is used, the azimuth may be measured by the following equation.

[0106] Equation 1: ψ = arctan(sinψ / cosψ)

[0107] In one or more examples, the variable ψ represents an azimuth angle, and cos ψ and sin ψ represent an X-axis fluxgate output value and a Y-axis fluxgate output value.

[0108] The roll angle may correspond to the angle at which the horizontal plane is tilted laterally (left and right), and when the roll angle is calculated, the left gradient or right gradient of the device may be known. The pitch angle may correspond to the angle at which the horizontal plane is tilted vertically (up and down), and when the pitch angle is calculated, the gradient angle at which the device is tilted upward or downward may be known. For example, when an acceleration sensor is used, the roll angle and the pitch angle may be measured using the following equations.

[0109] Equation 2: φ = arcsin(ay / g)

[0110] Equation 3: θ = arcsin(ax / g)

[0111] In one or more examples, the variable g represents gravity acceleration, φ represents a roll angle, θ represents a pitch angle, ax represents an X-axis acceleration sensor output value, and ay represents a Y-axis acceleration sensor output value.

[0112] For example, if each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 provides motion information to the electronic apparatus 100, the electronic apparatus 100 can obtain posture information of each of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 based on the received information.

[0113] However, as understood by one of ordinary skill in the art, embodiments are not limited to these configurations, and the inertial measurement sensor may be any sensor configured to obtain motion information or posture information of a device.

[0114] Furthermore, each of the first portable device 200 - 1 , the second portable device 200 - 2 , and the wearable device 300 may directly obtain an orientation and provide the orientation to the electronic apparatus 100 .

[0115] As described above, the electronic device 100 can recognize the user's position by obtaining the orientation of the portable device or any information indicating the orientation without an optical sensor such as an IR sensor or a camera, and perform a corresponding operation. In addition, the electronic device 100 can control the pointer to remain within the user's field of view, thereby improving usability and convenience in interacting with the display. In addition, the portable device can be attachable and detachable, thereby improving user convenience, and various types of interactions are feasible.

[0116] The following will be Figures 4 to 14 The operation of the electronic device 100 is described in more detail. Figures 4 to 14 In the present invention, for the convenience of description, a single embodiment will be described. However, Figures 4 to 14 The individual embodiments of the present invention may be implemented in combination.

[0117] Figure 4 is a diagram illustrating overall operations of the electronic device 100 according to one or more embodiments.

[0118] The processor 130 may perform an operation of identifying an initial position of a user ( 400 ) and an operation of identifying a position (or location) of a pointer.

[0119] First, the processor 130 may control the projection part 110 to project an image pattern to the projection area (400-1). The processor 130 may receive sensor data 420-3 of one or more handheld controllers 420-1 (such as the first portable device 200-1 and the second portable device 200-2) and wearable sensors 420-2 (such as the wearable device 300), and use the received sensor data to identify the user's position (400-3) through a lookup table (400-2). The position may be the user's initial position.

[0120] After obtaining the initial position, processor 130 may calculate the user's current position (410-1) and identify an area in the projection area corresponding to the user's field of view (eg, viewpoint) (410-2). In one or more examples, the calculated current position is a refinement of the obtained initial position.

[0121] The processor 130 may identify a point (410-3) to which a pointer of the projection area is to be projected based on the orientation of the handheld controller 420-1, and project the pointer toward an edge of an area in the projection area corresponding to the user's field of view based on the point of the projection area falling outside the area (410-4) in the projection area.

[0122] Processor 130 may control a multimedia object or multimedia content based on sensor data 420-3 (430). For example, processor 130 may change the size of a multimedia object or the position of a multimedia object based on sensor data 420-3. The multimedia object or multimedia content may be displayed on a projection surface such as a projection screen.

[0123] Figure 5 is a flowchart illustrating an overall operation of the electronic device 100 according to one or more embodiments.

[0124] In operation S510, the processor 130 may receive sensor data from the first portable device 200-1, the second portable device 200-2, and the wearable device 300, and process the sensor data. For example, the processor 130 may receive first motion information from the first portable device 200-1, the second portable device 200-2, and the wearable device 300, and recognize the orientations of the first portable device 200-1, the second portable device 200-2, and the wearable device 300 based on the first motion information.

[0125] In operation S520, the processor 130 may perform an initialization process (operation). For example, the processor 130 may project a preset image to the projection area and obtain various information by processing sensor data received after the user performs the process based thereon.

[0126] In operation S530, the processor 130 may identify at least one of the user's position, the user's field of view, or the projection position of the pointer based on the received sensor data. For example, the processor 130 may identify the user's position based on the orientations of the first portable device 200-1, the second portable device 200-2, and the wearable device 300. In addition, the processor 130 may identify the user's gaze based on the first motion information received from the wearable device 300, and identify an area in the projection area corresponding to the user's field of view based on the user's gaze. In addition, the processor 130 may identify the projection position of the pointer based on at least one orientation of the first portable device 200-1 or the second portable device 200-2.

[0127] In operation S540, the processor 130 may project the pointer toward an edge of an area corresponding to the user's field of view in the projection area based on the projection position of the pointer based on at least one orientation of the first portable device 200-1 or the second portable device 200-2 falling outside the area.

[0128] In operation S550 , the processor 130 may perform control of the multimedia object or multimedia content based on the sensor data received thereafter.

[0129] After performing the one-time initialization process (operation) as described above, the process of operation S550 may be continued. In addition, since an optical sensor is not used, an accurate control signal may be recognized even if there is movement of the user or other users.

[0130] Figure 6 is a diagram illustrating a portable device according to one or more embodiments.

[0131] like Figure 6 As shown, the first portable device 200-1 and the second portable device 200-2 may be attachable or detachable. For example, the first portable device 200-1 and the second portable device 200-2 may be provided with magnets at their contact surfaces and may be attachable or detachable.

[0132] Furthermore, the first portable device 200 - 1 and the second portable device 200 - 2 may provide different control signals to the electronic apparatus 100 when separated and when combined.

[0133] Figure 7 is a diagram illustrating a projection area near a user according to one or more embodiments.

[0134] like Figure 7As shown, the projection area can be implemented as a screen having at least one projection surface greater than or equal to a preset size or an immersive screen including at least two projection surfaces.

[0135] In one or more examples, Figure 7 The projection area in can be implemented as a device including multiple displays or a device including a display bent 90 degrees.

[0136] The user may interact through the first portable device 200 - 1 , the second portable device 200 - 2 , and the wearable device 300 .

[0137] Figure 8 and Fig. 9 is a diagram illustrating a method of identifying a user's location according to one or more embodiments.

[0138] In operation S910, a spatial image pattern may be displayed via a screen / display / projector device. Figure 8 As shown, the processor 130 can project an image including at least three points to the projection area. The image can be a spatial image pattern via a screen / display / projector device. In one or more examples, the image including three points can be an image for guiding user actions.

[0139] In operation S920, the user may be prompted to simultaneously point to the left and right parts of the spatial image pattern with the corresponding hands while holding the left unit (e.g., 200-1) and the right unit (e.g., 200-2) of the handheld controller. As a result of the prompt, the user may gaze at the center point among the three points, arrange the first portable device 200-1 to face the left point, and arrange the second portable device 200-2 to face the right point.

[0140] In operation S930, based on the fused IMU sensor orientation, when the user looks at and points to the displayed spatial image pattern, the initial position of the user is obtained from a pre-stored lookup table. For example, the processor 130 can be configured to cause the first portable device 200-1, the second portable device 200-2, and the wearable device 300 to receive the first motion information (S930), respectively, and the first portable device 200-1, the second portable device 200-2, and the wearable device 300 to identify the corresponding orientation based on the first motion information, and to identify the user's position (or location) based on the identified orientation. In operation S940, the user's position / location can be tracked or updated based on the change of the IMU sensor orientation data.

[0141] For example, Figure 8As shown, the processor 130 can obtain θ1 and θ2 based on the orientations of the first portable device 200-1 and the second portable device 200-2, and identify the position of the user by identifying the distance d1 corresponding to θ1 and θ2 based on the lookup table stored in the memory 140. In one or more examples, if θ1 and θ2 are 180 degrees, the distance may be theoretically 0, and if θ1 and θ2 are 0 degrees, the distance may be theoretically infinite.

[0142] exist Figure 8 In the example of an image including three dots, any method is feasible as long as it is a method that can guide the user.

[0143] Through the above process, the user's location can be identified without a camera or IR sensor.

[0144] Fig.10 and Fig.11 is a diagram illustrating an area corresponding to a user's field of view in a projection area according to one or more embodiments.

[0145] like Fig.10 As shown, a user can ensure a particular field of view, such as a field of view based on eye movement, a field of view based on neck movement, or a field of view based on any other movement of the user.

[0146] The processor 130 can recognize the user's gaze based on the first motion information received from the wearable device 300, and identify an area in the projection area corresponding to the user's field of view based on the user's gaze. In one or more examples, the one area of ​​the projection area can be enlarged or reduced according to the user.

[0147] However, as understood by those of ordinary skill in the art, embodiments are not limited to these configurations, and the processor 130 may identify an area in the projection area corresponding to the user's field of view based on the first motion information received from the wearable device 300. For example, the processor 130 may identify an area in the projection area corresponding to the user's field of view based on the user's head direction, regardless of the user's gaze.

[0148] like Fig.11 As shown in the upper left end of , if the user moves to the right, an area in the projection area corresponding to the user's field of view may also change from area 1110 to area 1120.

[0149] In one or more examples, such as Fig.11 As shown in the upper right end of , if the person moves toward the projection area, an area in the projection area corresponding to the user's field of view may also change from area 1130 to area 1140.

[0150] In one or more examples, such as Fig.11 As shown in the lower end of , if the user's gaze changes from left to right, an area in the projection area corresponding to the user's field of view may also change from area 1150 to area 1160.

[0151] In one or more examples, the projection area may be implemented as a screen having at least one projection surface greater than or equal to a preset size or as an immersive screen including at least two projection surfaces. Therefore, the area that the user can view is determined according to the movement of the user or the movement of the user's gaze, and the user's gaze may not reach the remaining area. Therefore, it may be difficult for the user to identify the position of the pointer.

[0152] Fig.12 and Fig.13 is a diagram illustrating one area of ​​a projection area corresponding to a field of view of a user and a projection position of a pointer according to one or more embodiments.

[0153] In operation S1310, the initial position of the user may be determined by using IMU sensor data from a handheld controller pointed at the displayed image pattern. For example, the processor 130 may identify the position of the user. In operation S1320, the position of the user is updated based on continuously received sensor data (such as IMU sensor data).

[0154] In operation S1330, based on the current position information and the head orientation (e.g., from the ear IMU sensor), the coordinates of the display area related to the field of view (FoV) of the user's eyes are calculated. For example, the processor 130 can identify an area in the projection area corresponding to the user's field of view based on the user's position and the sensor data received from the wearable device 300.

[0155] In operation S1340, mapping information between the handheld IMU sensor data and the 3D pointer movement is automatically adjusted based on the FoV area coordinates of the display so that the movement of the 3D pointer is limited within the FoV area. The processor 130 may project the pointer toward a point in the projection area corresponding to at least one orientation of the first portable device 200-1 or the second portable device 200-2 based on that a point in the projection area corresponding to at least one orientation of the first portable device 200-1 or the second portable device 200-2 is within an area corresponding to the field of view of the user in the projection area, and project the pointer toward an edge of an area corresponding to the field of view of the user in the projection area based on that a point in the projection area corresponding to at least one orientation of the first portable device 200-1 or the second portable device 200-2 falls outside the projection area.

[0156] In one or more examples, such as Fig.12 As shown on the left side of , according to the related art, the pointer may be projected to point 1210 or point 1220 regardless of the one area in the projection area corresponding to the user's field of view. For example, according to the present disclosure, the processor 130 may be configured such that: even if the portable device faces point 1210 or point 1220, the user is able to recognize the pointer by projecting the pointer at the edge of one area in the projection area corresponding to the user's field of view, such as Fig.12 as shown on the right.

[0157] Fig.14 is a diagram showing a detailed configuration of an electronic device and a portable device according to one or more embodiments. Fig.14 Description of the configurations shown that are repeated in the previous figure.

[0158] The electronic device 100 may further include a power module 191, an audio module 192, and a video module 193. The power module 191 may be a configuration for supplying power to the electronic device 100, the audio module 192 may be a configuration for processing audio content, and the video module 193 may be a configuration for processing video content.

[0159] The processor 130 may include an initialization module 131, an image pattern module 132, a lookup table module 133, a sensor data processing module 134, a pointer control module 135, and a control module 136. Each module may be implemented as hardware as a configuration of the processor 130. However, as understood by those of ordinary skill in the art, the embodiments are not limited to these configurations, and the operations of the various components may be implemented as software and implemented in a form stored in the memory 140.

[0160] The processor 130 may start an operation for identifying the position of the user, etc. through the initialization module 131. The processor 130 may project an image guiding the user's action at the projection area through the image pattern module 132. The processor 130 may identify the position of the user based on the lookup table module 133, identify the orientation of the portable device through the sensor data processing module 134, and limit the projection area of ​​the pointer to an area corresponding to the user's field of view in the projection area through the pointer control module 135. The processor 130 may perform an operation corresponding to the sensor data through the control module 136.

[0161] Fig.15 is a flowchart illustrating a control method of an electronic device according to one or more embodiments.

[0162] In operation S1510, an image may be projected onto a projection area. In operation S1520, after a first portable device, a second portable device, and a wearable device worn by a user are arranged to correspond to the provided image, the electronic device may receive first motion information from each of the first portable device, the second portable device, and the wearable device. In operation S1530, an orientation of each of the first portable device, the second portable device, and the wearable device may be identified based on the first motion information. In operation S1540, a position of the user may be identified based on the identified orientation.

[0163] In one or more examples, it may also include: after identifying the location of the user, receiving second motion information from each of the first portable device, the second portable device, and the wearable device, and updating the location of the user based on the second motion information.

[0164] Identifying the position of the user in operation S1540 may include: identifying a first angle and a second angle, the first angle being formed by a straight line moving from the user to the front of the user and a straight line parallel to the orientation of the first portable device, and the second angle being formed by a straight line moving from the user to the front of the user and a straight line parallel to the orientation of the second portable device; identifying distances corresponding to the first angle and the second angle based on a lookup table; and identifying the position of the user based on the identified distances.

[0165] In addition, the first motion information may include information about at least one of roll, pitch, or yaw measured by an inertial measurement sensor of each of the first portable device, the second portable device, and the wearable device.

[0166] The method may further include projecting the pointer to the one point of the projection area based on at least one orientation of the first portable device or the second portable device.

[0167] In one or more examples, it may also include: identifying a user's gaze based on first motion information received from the wearable device; and identifying an area in the projection area corresponding to the user's field of view based on the user's gaze.

[0168] Furthermore, projecting the pointer may include projecting the pointer at an edge of the one area based on a point of the projection area corresponding to at least one orientation of the first portable device or the second portable device falling outside the one area.

[0169] In one or more examples, it may further include: changing the size of the one area based on receiving a preset control signal from at least one of the first portable device or the second portable device while the pointer is in a state of being projected at the edge of the one area.

[0170] The projection area may be implemented as a screen greater than or equal to a preset size or as an immersive screen including at least two projection surfaces.

[0171] Furthermore, the first portable device and the second portable device may be held by a user, and the wearable device may be worn at a head of the user.

[0172] According to one or more embodiments of the present disclosure as described above, the electronic device can recognize the position of the user by obtaining the orientation of the portable device, etc., without an optical sensor such as an IR sensor or a camera, and perform an operation corresponding thereto.

[0173] Additionally, the electronic device may control the pointer to remain within the user's field of view, thereby improving usability and convenience in interacting with the display.

[0174] Furthermore, since the portable device is attachable or detachable and various interactions can be performed, user convenience can be improved.

[0175] In the above, the electronic device has been described assuming that it is a projector, but even if it is a device including a display, the same operations as those remaining except for operations associated with the display can be performed.

[0176] According to one or more embodiments of the present disclosure, the various embodiments described above may be implemented using software including instructions stored in a machine-readable storage medium (e.g., a computer). The machine may call instructions stored in the storage medium, and as a device capable of operating according to the called instructions, may include an electronic device according to the above-mentioned embodiment (e.g., electronic device (A)). Based on the execution of the instructions by the processor, the processor may use other elements directly or under the control of the processor to perform the function corresponding to the instruction. The instruction may include code generated by a compiler or executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. In this article, "non-transitory" only means that the storage medium is tangible and does not include signals, and the term does not distinguish whether the data is semi-permanently or temporarily stored in the storage medium.

[0177] In addition, according to one or more embodiments, a method according to the various embodiments described above may be provided, which is included in a computer program product. The computer program product may be exchanged between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or distributed online through an application store (e.g., PLAYSTORE™). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored in a storage medium (e.g., a server of a manufacturer, a server of an application store, or a memory of a relay server), or may be temporarily generated.

[0178] In addition, according to one or more embodiments of the present disclosure, the above-mentioned various embodiments can be implemented in a recordable medium readable by a computer or a device similar to a computer using software, hardware, or a combination of software and hardware. In some cases, the embodiments described herein can be implemented by the processor itself. According to software implementation, embodiments such as the processes and functions described herein can be implemented as separate software. Each software can perform one or more functions and operations described herein.

[0179] Computer instructions for performing processing operations in the device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in the non-transitory computer-readable medium are executed by a processor of a specific device, the specific device may perform the processing operations of the device according to the various embodiments described above. A non-transitory computer-readable medium may refer to a medium (such as a register, a cache, a memory, etc.) that stores data semi-permanently rather than for a very short time, and can be read by a device. Specific examples of non-transitory computer-readable media may include, but are not limited to, for example, a compact disk (CD), a digital versatile disk (DVD), a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, etc.

[0180] In addition, each element (e.g., module or program) according to the various embodiments described above may be formed by a single entity or multiple entities, and some of the sub-elements described above may be omitted, or other sub-elements may also be included in various embodiments. Alternatively or additionally, some elements (e.g., modules or programs) may be integrated into one entity and perform the same or similar functions performed by each corresponding element before integration. According to various embodiments, the operations performed by modules, programs, or other elements may be performed sequentially, in parallel, repeatedly, or in a heuristic manner, or at least some operations may be performed in a different order, omitted, or different operations may be added.

[0181] Although the present disclosure has been illustrated and described with reference to various embodiments of the present disclosure, it will be understood that the various embodiments are intended to be illustrative rather than restrictive. Those skilled in the art will appreciate that various changes in form and details may be made therein without departing from the true spirit and full scope of the present disclosure, including the appended claims and their equivalents.

Claims

1. An electronic device, include: Projection circuit; Communication interface; as well as at least one processor connected to the projection circuit and the communication interface and configured to control the electronic device, Wherein, the processor is configured to: controlling the projection circuit to project an image onto a projection area, After a first portable device, a second portable device, and a wearable device worn by a user are arranged to correspond to the image, receiving first motion information from each of the first portable device, the second portable device, and the wearable device through the communication interface, identifying an orientation of each of the first portable device, the second portable device, and the wearable device based on the first motion information, and Based on the identified orientation, a location of the user is identified.

2. The electronic device according to claim 1, in, The processor is configured to: After identifying the location of the user, receiving second motion information from each of the first portable device, the second portable device, and the wearable device through the communication interface, and Based on the second motion information, the position of the user is updated.

3. The electronic device according to claim 1, further comprising: include: A memory storing a lookup table, Wherein, the processor is configured to: identifying a first angle and a second angle, the first angle being formed by a first straight line moving from the user to the front of the user and a second straight line parallel to the orientation of the first portable device, and the second angle being formed by a third straight line moving from the user to the front of the user and a fourth straight line parallel to the orientation of the second portable device, Based on the lookup table, identifying distances corresponding to the first angle and the second angle, and Based on the identified distance, a location of the user is identified.

4. The electronic device according to claim 1, in, Each of the first portable device, the second portable device, and the wearable device includes an inertial measurement sensor, and The first motion information includes: Information related to at least one of roll, pitch, or yaw measured by an inertial measurement sensor of each of the first portable device, the second portable device, and the wearable device.

5. The electronic device according to claim 1, in, The processor is configured to: The projection circuit is controlled to project a pointer toward a point of the projection area based on at least one orientation of the first portable device or the second portable device.

6. The electronic device according to claim 5, in, The processor is configured to: recognizing a gaze of the user based on first motion information received from the wearable device, and Based on the gaze of the user, an area of ​​the projection area corresponding to the field of view of the user is identified.

7. The electronic device according to claim 6, in, The processor is configured to: Based on the fact that a point of the projection area corresponding to at least one orientation of the first portable device or the second portable device falls outside the area, the projection circuit is controlled to project the pointer toward an edge of the area.

8. The electronic device according to claim 7, in, The processor is configured to: The size of the one area is changed based on receiving a preset control signal from at least one of the first portable device or the second portable device through the communication interface while the pointer is in a state of being projected toward an edge of the one area.

9. The electronic device according to claim 1, in, The projection area is implemented as a screen having at least one projection surface whose size is greater than or equal to a preset size, or is implemented as an immersive screen including at least two projection surfaces.

10. The electronic device according to claim 1, in, The first portable device and the second portable device are each held by the user, and The wearable device is configured to be worn on the user's head.

11. An electronic device, include: monitor; Communication interface; as well as at least one processor connected to the display and the communication interface and configured to control the electronic device, Wherein, the processor is configured to: controlling the display to display an image, After a first portable device, a second portable device, and a wearable device worn by a user are arranged to correspond to the image, receiving first motion information from each of the first portable device, the second portable device, and the wearable device through the communication interface, identifying an orientation of each of the first portable device, the second portable device, and the wearable device based on the first motion information, and Based on the identified orientation, a location of the user is identified.

12. A control method for an electronic device, the control method include: projecting an image onto a projection area; receiving first motion information from each of a first portable device, a second portable device, and a wearable device worn by a user after the first portable device, the second portable device, and the wearable device are arranged to correspond to the image; identifying an orientation of each of the first portable device, the second portable device, and the wearable device based on the first motion information; as well as Based on the identified orientation, a location of the user is identified.

13. The control method according to claim 12, further comprising: include: receiving second motion information from each of the first portable device, the second portable device, and the wearable device after identifying the location of the user; as well as Based on the second motion information, the position of the user is updated.

14. The control method according to claim 12, in, Identifying the location of the user includes: identifying a first angle and a second angle, the first angle being formed by a first straight line moving from the user to the front of the user and a second straight line parallel to the orientation of the first portable device, and the second angle being formed by a third straight line moving from the user to the front of the user and a fourth straight line parallel to the orientation of the second portable device, Based on a lookup table, identifying distances corresponding to the first angle and the second angle, and Based on the identified distance, a location of the user is identified.

15. The control method according to claim 12, in, The first motion information includes information about at least one of roll, pitch, or yaw measured by an inertial measurement sensor of each of the first portable device, the second portable device, and the wearable device.