Electronic device and operating method thereof

Abstract

An operating method of an electronic device according to an embodiment may comprise an operation of displaying image data captured by a camera in a first area of a display. The operating method may comprise an operation of identifying objects included in the first area. The operating method may comprise an operation of, on the basis of a user input including at least one of a gaze input, a speech input, and a gesture input, determining a target object to be controlled by a user among the objects. The operating method may comprise an operation of displaying the target object in a second area of the display. The second area may be overlaid on the first area. In the second area, image data corresponding to the target object may be streamed. The image data streamed in the second area may be controlled on the basis of a user input for the target object to an object control interface.

Description

Electronic device and method of operation thereof

[0001] Embodiments of the present invention relate to an electronic device and a method of operating the same.

[0002] Augmented reality (AR) refers to the display of virtual images within the physical environment of the real world, or the simultaneous presentation of real objects and virtual images. An AR experience is when a user experiences augmented reality using AR technology. AR experiences expand user experiences beyond the boundaries of reality and the virtual world by enabling users to perceive and perceive worlds they have never experienced before. Furthermore, by sharing their AR experiences with others, users can extend their experiences to others without the constraints of time and space.

[0003] Virtual reality (VR) refers to a technology that completely immerses users in a virtual environment, replacing the real world. VR separates users from reality and transports them into a virtual space, allowing them to experience a 360-degree surrounding environment. Like augmented reality (AR), VR is being utilized in various fields along with advancements in modern technology, creating new, creative, and innovative experiences. The general term for augmented reality and VR is extended reality (XR).

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] A method of operating an electronic device according to one embodiment may include an operation of displaying image data captured by a camera in a first area of ​​a display. The method may include an operation of identifying objects included in the first area. The method may include an operation of determining a target object that the user wishes to control among the objects based on a user input including at least one of a gaze input, a voice input, and a gesture input. The method may include an operation of displaying the target object in a second area of ​​the display. The second area may be an overlay on the first area. The second area may be a area in which image data corresponding to the target object is streamed. The image data streamed on the second area may be controlled based on a user input to an object control interface of the target object.

[0006] An electronic device according to one embodiment may include a display. The electronic device may include one or more processors. The electronic device may include a memory that stores instructions. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to display image data captured by a camera in a first area of ​​the display. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to identify objects included in the first area. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to determine a target object that the user wishes to control among the objects based on a user input including at least one of a gaze input, a voice input, or a gesture input. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to display the target object in a second area of ​​the display. The second region may be overlaid on the first region. The second region may be a region in which image data corresponding to the target object is streamed. The image data streamed on the second region may be controlled based on a user input to an object control interface of the target object.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0008] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0009] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to one embodiment.

[0010] FIG. 4 is a schematic block diagram of an electronic device according to one embodiment.

[0011] FIG. 5 is a drawing for explaining object identification according to one embodiment.

[0012] FIG. 6A and FIG. 6B are diagrams for explaining target object determination according to one embodiment.

[0013] FIG. 7 is a diagram illustrating an object control interface for controlling a target object according to one embodiment.

[0014] FIGS. 8 and 9 are drawings for explaining a method of displaying a target object on a display based on a reference point according to one embodiment.

[0015] FIG. 10A and FIG. 10B are drawings for explaining a size selection guide of a target object according to one embodiment.

[0016] FIG. 11 is a diagram illustrating a case in which there are multiple target objects according to one embodiment.

[0017] FIG. 12 is a drawing for explaining a case in which there are multiple cameras according to one embodiment.

[0018] FIG. 13 is a drawing for explaining a case in which the environment in which a target object is displayed changes according to one embodiment.

[0019] FIG. 14 illustrates a flowchart of a method of operating an electronic device according to one embodiment.

[0020] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0021]

[0022] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment.

[0023] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0024] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134).

[0025] According to one embodiment, the processor (120) may be implemented as a circuit (e.g., a processing circuit) such as a system on chip (SoC) or an integrated circuit (IC). The processor (120) may include one or more processors. For example, the processor (120) may include a combination of one or more processors such as a CPU, a GPU, an MPU, an AP, and a CP.

[0026] According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0027] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0028] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto.

[0029] According to one embodiment, the memory (130) may include one or more memories. The instructions stored in the memory (130) may be stored in a single memory. The instructions stored in the memory (130) may be divided and stored in a plurality of memories. The instructions stored in the memory (130) may be individually or collectively executed by the processor (120) to cause the electronic device (101) (e.g., the electronic device (200) of FIG. 2 , the electronic device (300) of FIG. 3 , and the electronic device (400) of FIG. 4 ) to perform and / or control the method described with reference to FIGS. 2 to 14 . Instructions stored in the memory (130) may be individually or collectively executed by a plurality of processors to cause the electronic device (101) (e.g., the electronic device (200) of FIG. 2, the electronic device (300) of FIG. 3, and the electronic device (400) of FIG. 4) to perform and / or control the method described with reference to FIGS. 2 to 14. According to one embodiment, the memory (130) may include a volatile memory (132) or a non-volatile memory (134).

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a depth sensor, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048]

[0049] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0050] Referring to FIG. 2, a wearable electronic device (200) according to one embodiment may include at least one of a light output module (211), a display member (201), and a camera module (250). An electronic device (e.g., an electronic device (101) of FIG. 1) may be implemented in the form of a wearable electronic device (200). A wearable electronic device (200) (e.g., an electronic device (102) of FIG. 1) may be implemented separately from an electronic device (e.g., an electronic device (101) of FIG. 1). A wearable electronic device (200) (e.g., an electronic device (102) of FIG. 1) may be connected to an electronic device (e.g., an electronic device (101) of FIG. 1).

[0051] According to one embodiment, the light output module (211) may include a light source capable of outputting an image and a lens that guides the image to the display member (201). The output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0052] According to one embodiment, the display member (201) may include an optical waveguide (e.g., a waveguide). An output image of the optical output module (211) incident on one end of the optical waveguide may be propagated within the optical waveguide and provided to a user. The optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide the output image of the optical output module (211) to the user's eyes by using at least one diffractive element or reflective element.

[0053] According to one embodiment, the camera module (250) can capture still images and / or moving images. The camera module (250) is positioned within the lens frame and may be positioned around the display member (201). The camera module (250) may include a first camera module (251), a second camera module (253), and a third camera module (255).

[0054] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. The first camera module (251) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0055] According to one embodiment, the second camera module (253) can capture an external image.

[0056] According to one embodiment, the third camera module (255) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. The third camera module (255) can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. The second camera module (253) can also be used for hand detection and tracking, and user gesture recognition. At least one of the first camera module (551) to the third camera module (255) can be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module can include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0057]

[0058] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device (300) according to one embodiment.

[0059] Referring to FIGS. 3A and 3B, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1) may be implemented in the form of a wearable electronic device (300). Camera modules (311, 312, 313, 314, 315, 316) (e.g., camera module (180) of FIG. 1) and / or depth sensor (317) (e.g., sensor module (176) of FIG. 1) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on a first surface (310) of the housing.

[0060] According to one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0061] According to one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection, tracking, and recognition of user gestures (e.g., hand movements). The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. The camera modules (311, 312) can also be used for hand detection and tracking, and recognition of user gestures.

[0062] In one embodiment, a depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (317), camera modules (313, 314, 315, 316) may determine the distance to an object.

[0063] According to one embodiment, a camera module (325, 326) for facial recognition (e.g., camera module (180) of FIG. 1) and / or a display (321) (e.g., display module (160) of FIG. 1) (and / or a lens) may be disposed on the second side (320) of the housing.

[0064] In one embodiment, a face recognition camera module (325, 326) adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user.

[0065] According to one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). The wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .

[0066] As described above, according to one embodiment, the wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured to a body part of the user. The wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0067]

[0068] FIG. 4 is a schematic block diagram of an electronic device according to one embodiment.

[0069] Referring to FIG. 4, according to one embodiment, an electronic device (400) may include at least a part of the configuration of the electronic device (101) described with reference to FIG. 1. The electronic device (400) may be implemented in the form of a wearable electronic device (e.g., the wearable electronic device (200) of FIG. 2) (e.g., smart glasses including virtual reality glasses). The electronic device (400) may also be implemented in the form of a wearable electronic device (e.g., the wearable electronic device (300) of FIG. 3) (e.g., a head mounted display (HMD) including an augmented reality (AR) device, a virtual reality (VR) device, and / or a mixed reality (MR) device). The electronic device (400) may be configured to easily control user interface (UI) components provided in an AR environment and a VR environment.

[0070] According to one embodiment, the electronic device (400) can control a target object displayed on a display (410) (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, or the display (321) of FIG. 3). For example, the electronic device (400) can display (e.g., stream) a target object on a second area of ​​the display (410), and image data streamed (e.g., image display, video playback, real-time playback) from the second area can be controlled based on user input to an object control interface associated with displaying, recording, and sharing of the target object.

[0071] According to one embodiment, the electronic device (400) may include a display (410) (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, or the display (321) of FIG. 3), one or more processors (420) (e.g., the processor (120) of FIG. 1), and a memory (430) (e.g., the memory (130) of FIG. 1, or the memory (430) of FIG. 4). The memory (430) may store various data used by at least one component (e.g., the processor (420)) of the electronic device (400).

[0072] According to one embodiment, the processor (420) may be implemented as a circuit (e.g., a processing circuit) such as a system on chip (SoC) or an integrated circuit (IC). The processor (420) may include one or more processors. For example, the processor (420) may include a combination of one or more processors such as a CPU, a GPU, an MPU, an AP, and a CP.

[0073] According to one embodiment, the memory (430) may include one or more memories. The instructions stored in the memory (430) may be stored in a single memory. The instructions stored in the memory (430) may be divided and stored in multiple memories. The instructions stored in the memory (430) may be individually or collectively executed by the processor (420) to cause the electronic device (400) to perform and / or control the method described with reference to FIGS. 2 to 14. The instructions stored in the memory (430) may be individually or collectively executed by multiple processors to cause the electronic device (400) to perform and / or control the method described with reference to FIGS. 2 to 14.

[0074] According to one embodiment, the display (410) can visually provide information to an external device (e.g., a user) of the electronic device (400). The display (410) can be configured to display images or videos. The display (410) can display a graphical user interface (GUI) of a running app (or application program). The display (410) can include, for example, a display, a holographic device, or a projector and control circuitry for controlling the device. The display (410) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by a touch.

[0075] According to one embodiment, a processor (420) (e.g., an application processor) may access a memory (430) to execute instructions. The processor (420) may execute an image processing module (421), a gesture analysis module (422), a gaze analysis module (423), an automatic speech recognition (ASR) module (424), and / or a natural language understanding (NLU) module (425). For example, the image processing module (421), the gesture analysis module (422), the gaze analysis module (423), the ASR module (424), and / or the NLU module (425) may process user input.

[0076] According to one embodiment, the image processing module (421) may process data collected from a camera (e.g., the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, or the camera modules (311-316) of FIG. 3) to generate image data. The image data may be captured by the camera. The image processing module (421) may identify objects based on the image data. Object identification will be described in detail with reference to FIG. 5.

[0077] According to one embodiment, the gesture analysis module (422) can analyze a user's gesture input. For example, the user's gesture input can be input to the electronic device (400) by a glove or a joystick. The user's gesture input can be input to the electronic device (400) by a camera (e.g., the third camera module (255) of FIG. 2 or the camera modules (313 to 316) of FIG. 3). For example, the gesture analysis module (422) can perform user hand detection, user hand tracking, and user gesture (e.g., hand movement) recognition.

[0078] According to one embodiment, the gaze analysis module (423) can analyze a user's gaze input. The user's gaze input can be input to the electronic device (400) by a camera (e.g., the first camera module (251) of FIG. 2 or the camera modules (325, 326) of FIG. 3). For example, the gaze analysis module (423) can recognize the trajectory of the gaze. For example, the gaze analysis module (423) can periodically or aperiodically analyze information related to the trajectory of the user's eyes or gaze (e.g., trajectory information).

[0079] According to one embodiment, an automatic speech recognition module (ASR module) (424) and a natural language understanding module (NLU module) (425) can process a user's voice input. The ASR module (424) can convert data related to a voice input received from an electronic device (400) into text data. The NLU module (425) can analyze the converted text data to determine an action to be performed in response to a voice input (e.g., a user utterance) (e.g., a target object determination action and / or a target object control action).

[0080] In one embodiment, the user input may include first information used to determine the target object and second information associated with at least one of displaying, recording, or sharing the target object. For example, in the case of the voice input "Zoom in on the cat in front," the first information used to determine the target object may correspond to "the cat in front," and the second information associated with the target object control action (e.g., displaying, recording, and / or sharing) may correspond to "Zoom in."

[0081] In one embodiment, the user input providing the first information and the user input providing the second information may have different input types. For example, in the case of a gestural input (e.g., pointing at a cat in the line of sight with a finger) and a vocal input (e.g., "Zoom in on that"), the first information used to determine the target object may be derived from the gestural input, and the second information associated with the target object control action (e.g., displaying, recording, and / or sharing) may be derived from the vocal input.

[0082] According to one embodiment, if the first information and / or the second information is insufficient in the input received by the electronic device (400) from the user, the electronic device (400) may request additional input from the user. For example, if the electronic device (400) only receives a gesture input (e.g., a gesture for enlarging or reducing, such as a gesture between the thumb and index finger), the electronic device (400) may request additional input from the user regarding which object to enlarging or reducing. In this case, the additional input from the user may be, for example, a voice input (e.g., “that cat”), a gesture input (e.g., pointing at the cat with a finger), or a gaze input (e.g., staring at the cat for a certain period of time).

[0083]

[0084] FIG. 5 is a drawing for explaining object identification according to one embodiment.

[0085] Referring to FIG. 5, according to one embodiment, an electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (200) of FIG. 2, an electronic device (300) of FIG. 3, or an electronic device (400) of FIG. 4) may display an image captured by a camera (e.g., a camera module (180) of FIG. 1, a camera module (250) of FIG. 2, or camera modules (311-316) of FIG. 3) on a first area (e.g., the entire area) of a display (e.g., a display module (160) of FIG. 1, a display member (201) of FIG. 2, a display (321) of FIG. 3, or a display (410) of FIG. 4). For example, what is displayed on the display (410) of the electronic device (400) may be a video. The electronic device (400) may display a video by sequentially displaying a plurality of images over time.

[0086] According to one embodiment, the electronic device (400) (e.g., the image processing module (421)) can identify (e.g., 502) objects (e.g., 511, 512, and 513) included in the first area (501). The electronic device (400) can recognize objects (e.g., trees, cars, utility poles, and people) in the first area (501). The electronic device (400) can classify the recognized objects. For example, the electronic device (400) can classify the recognized cars as 'white car', 'black car', or 'car with number xxxx' (e.g., classification based on features). For example, the object recognition of the electronic device (400) may always be run in the background, or the object recognition of the electronic device (400) may be triggered based on a user's input. Additionally, for example, object recognition may be performed by a server external to the electronic device (e.g., server (108) of FIG. 1).

[0087]

[0088] FIG. 6A and FIG. 6B are diagrams for explaining target object determination according to one embodiment.

[0089] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3, or electronic device (400) of FIG. 4) can determine a target object that the user wishes to control among identified objects based on user input including gaze input, voice input, and / or gesture input.

[0090] Referring to FIG. 6A, according to one embodiment, the electronic device (400) may acquire image data (602) corresponding to a target object (e.g., two people) within image data (601). The target object is not limited to one, and if there are multiple target objects, the electronic device (400) may treat the multiple objects as one target object. For example, when acquiring image data (602) corresponding to a target object (e.g., two people), the electronic device (400) may acquire natural image data (602) by blurring the outline of the target object.

[0091] Referring to FIG. 6B, according to one embodiment, the electronic device (400) may obtain image data (604) corresponding to a target object (e.g., a child) within the image data (603). For example, if a portion of the target object (e.g., a child) is covered by another object, the electronic device (400) may infer the covered portion of the area and obtain natural image data (604). For example, the inference of the portion of the area of ​​the electronic device (400) may be performed through a generative model. The generative model may be built into the electronic device (400) or may be executed on a server external to the electronic device (e.g., the server (108) of FIG. 1).

[0092] According to one embodiment, the first information used to determine the target object may be obtained from different types of user input (e.g., gaze input, voice input, or gesture input). For example, if a voice input (e.g., "cat") and a gesture input (e.g., pointing to a striped cat) are input, the electronic device (400) may determine the striped cat as the target object.

[0093]

[0094] FIG. 7 is a diagram illustrating an object control interface for controlling a target object according to one embodiment.

[0095] Referring to FIG. 7, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3, or electronic device (400) of FIG. 4) may display an object control interface for controlling a target object.

[0096] According to one embodiment, the electronic device (400) can display image data captured by a camera (e.g., the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, or the camera modules (311-316) of FIG. 3) on a first area (e.g., the entire area of ​​the display (701)) of the display (701) (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, the display (321) of FIG. 3, or the display (410) of FIG. 4). For example, what is displayed on the display (410) of the electronic device (400) can be a video. The electronic device (400) can display a video by sequentially displaying a plurality of images over time. The electronic device (400) can identify objects included in the first area (e.g., the entire area of ​​the display (701)). The electronic device (400) can determine a target object (701-1) that the user wishes to control among the identified objects based on user input (e.g., user input including gaze input, voice input, and / or gesture input).

[0097] According to one embodiment, the electronic device (400) can display a target object (701-1) on a second area (702-1) of a display (702). The display (701) and the display (702) may refer to the same hardware (e.g., the display module (160) of FIG. 1 , the display member (201) of FIG. 2 , the display (321) of FIG. 3 , and the display (410) of FIG. 4 ). Distinguishing the displays (410) with different reference numbers (e.g., 701, 702, 703, and 704) may be to distinguish the displays (410) that display different data over time. For example, the second area (702-1) may be overlaid on the first area (e.g., the entire area of ​​the display (702)). For example, the second area (702-1) may be a place where image data corresponding to the target object (701-1) is streamed (e.g., real-time playback, dynamic transmission). Streaming may be a method of transmitting data (e.g., image data) in pieces. The electronic device (400) may improve user experience by transmitting image data in real time through streaming. In one embodiment, the image data streamed on the second area (702-1) may include only image data corresponding to the target object (701-1). What is displayed on the display (410) of the electronic device (400) may be a video. The electronic device (400) may display a video by sequentially displaying a plurality of images over time.

[0098] According to one embodiment, the electronic device (400) may display an object control interface (703-1) associated with displaying, recording, and / or sharing a target object (701-1) on the display (703). For example, image data streaming on the second area (702-1) of the display (703) may be controlled based on a user input to the object control interface (703-1). In one embodiment, the object control interface (703-1) may represent an element for performing a specific task or function in the user experience (UX) and user interface (UI) design. The object control interface (703-1) may be implemented in the form of a button, but is not limited thereto. For example, the touchability of the object control interface (703-1) may be visually implemented through a three-dimensional design and / or a visual shadow that appears to be pressable. For example, the object control interface (703-1) may be operated through interactions such as clicks and touches. For example, in response to a user input to the object control interface (703-1) (e.g., a gaze input that gazes at a certain area (e.g., a symbol) of the object control interface for a certain period of time, a voice input (e.g., speaking) corresponding to a symbol of the object control interface, or a gesture input (e.g., touching) to a symbol of the object control interface), the electronic device (400) may provide visual feedback (e.g., an indication that a clicked button has been pressed) to the user. For example, the user may perform a specific task through interaction with the object control interface (703-1).

[0099] According to one embodiment, the object control interface (703-1) may include a symbol (e.g., a start recording symbol or a stop recording symbol) associated with recording a target object (e.g., image data corresponding to the target object). For example, the object control interface (703-1) may include a symbol (e.g., a cancel symbol) associated with releasing the target object (701-1). The object control interface (703-1) may include, for example, a symbol that triggers an additional interface (e.g., a more symbol).

[0100] According to one embodiment, the electronic device (400) may display an object control interface (703-2) on the display (704) based on a user input for a symbol (e.g., a more symbol) that causes an additional interface. For example, the object control interface (703-2) may include a symbol (e.g., a zoom in symbol or a zoom out symbol) associated with adjusting the size of a target object (e.g., a second area into which image data corresponding to the target object is streamed). For example, the object control interface (703-2) may include a symbol (e.g., a move symbol) associated with adjusting the position of a target object (e.g., a second area into which image data corresponding to the target object is streamed). For example, the object control interface (703-2) may include a symbol (e.g., an edit symbol) associated with editing a target object (e.g., image data corresponding to the target object) (e.g., editing via a camera app or editing via an editing app). For example, the object control interface (703-2) may include a symbol (e.g., a sharing symbol) associated with sharing (e.g., sharing via a messenger app) of a target object (e.g., image data corresponding to the target object).

[0101] According to one embodiment, the electronic device (400) may update whether to record a target object (e.g., image data corresponding to the target object) based on a user's input (e.g., a subsequent input) (e.g., an input for a start recording symbol or a stop recording symbol) to the object control interface (703-1). For example, the electronic device (400) may adjust the size of the target object (e.g., a second area into which image data corresponding to the target object is streamed) based on a user's input (e.g., a subsequent input) (e.g., an input for a zoom in symbol or a zoom out symbol) to the object control interface (703-2). For example, the electronic device (400) may adjust the position of the target object (e.g., a second area into which image data corresponding to the target object is streamed) based on a user's input (e.g., a subsequent input) (e.g., an input for a move symbol) to the object control interface (703-2). For example, the electronic device (400) may perform editing or sharing of a target object (e.g., image data corresponding to the target object) based on a user's input (e.g., a subsequent input) (e.g., an input for an edit symbol or a share symbol) to the object control interface (703-2).

[0102] According to one embodiment, the electronic device (400) may release the target object (701-1) based on a user input to the object control interface (703-1) (e.g., an input for a cancel symbol). For example, image data of the target object (701-1) streamed on the second area (702-1) may be recorded within the electronic device (400) (e.g., memory (430)) from the time the target object (701-1) is determined. When the target object (701-1) is released, recording of the target object (e.g., image data corresponding to the target object) may be stopped.

[0103] In one embodiment, the second information associated with controlling (e.g., displaying (e.g., moving, enlarging, or reducing), recording, sharing, or editing) the target object may be obtained from different types of user input (e.g., gaze input, voice input, or gesture input). For example, when a voice input (e.g., “Zoom in”) and a gesture input (e.g., spreading the index finger and thumb apart) are input, the electronic device (400) may enlarg the target object (e.g., the second region).

[0104]

[0105] FIGS. 8 and 9 are drawings for explaining a method of displaying a target object on a display based on a reference point according to one embodiment.

[0106] Referring to FIG. 8, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, the electronic device (300) of FIG. 3, or the electronic device (400) of FIG. 4) can control a target object (802) on a display (801) based on a reference point (803) of the target object (802).

[0107] In one embodiment, the reference point (e.g., 803) may be a reference point for controlling the target object (e.g., 802). For example, the reference point (e.g., 803) may be set to the coordinates of the center pixel located at the bottom of the pixels constituting the target object (e.g., 802) so as to be consistent with the reality in which gravity exists. For example, the target object may be enlarged or reduced based on the reference point. If the reference point is located at the exact center of the target object, when the target object standing on the ground is reduced, the target object may be displayed as if it is floating above the ground. If the reference point is located at the exact center of the target object, when the target object standing on the ground is enlarged, a part of the target object may be displayed as if it is located below the ground. The electronic device (400) sets the reference point of the target object not to the coordinates of the exact center of the pixels forming the target object, but to the coordinates of the central pixel located at the bottom of the pixels forming the target object, so that even when the target object standing on the ground is enlarged or reduced, the target object can be displayed as standing correctly on the ground. However, the setting of the reference point may vary depending on the environment (e.g., AR environment, VR environment, or MR environment) provided by the electronic device (400). In one embodiment, the setting of the reference point may be changeable according to the user's setting. For example, the reference point may be set to the coordinates corresponding to the center of gravity of the object.

[0108] According to one embodiment, the electronic device (400) can determine a reference point (803) of a target object (802). The electronic device (400) can determine a size and position of a second area based on the reference point and second information (e.g., second information associated with control (e.g., display (e.g., moving, enlarging, or reducing), recording, sharing, or editing) of the target object). The electronic device (400) can display the target object (802) in the second area, the size and position of which have been determined (e.g., streaming image data corresponding to the target object).

[0109] Referring to FIG. 9, according to one embodiment, the location of the second area (902-1) determined by the electronic device (400) based on the reference point of the target object (901-1) may be a location that does not overlap with another object (903).

[0110] According to one embodiment, the electronic device (400) can display image data captured by a camera (e.g., the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, or the camera modules (311-316) of FIG. 3) on a first area (e.g., the entire area of ​​the display (901)) of a display (901) (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, the display (321) of FIG. 3, or the display (410) of FIG. 4). The electronic device (400) can identify objects included in the first area (e.g., the entire area of ​​the display (901)). The electronic device (400) can determine a target object (901-1) that the user wants to control among the identified objects based on a user input (e.g., a user input including a gaze input, a voice input, and / or a gesture input). The electronic device (400) can determine the location of the second area (902-1) based on the reference point of the target object (901-1). For example, the location of the second area (902-1) may be a location that does not overlap with another object (903).

[0111] According to one embodiment, when the input received from the user by the electronic device (400) includes a voice input (e.g., “Show me on the secondary screen”), the electronic device (400) may determine the location of the second area (902-1) so that the target object (901-1) and the second area (902-1) are displayed together. The user may track the original of the target object (e.g., 901-1) and the separately extracted target object (e.g., the target object streamed from the second area (902-1)) together on the display (902). The display (901) and the display (902) may refer to the same hardware (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, the display (321) of FIG. 3, and the display (410) of FIG. 4). Distinguishing displays (410) with different reference numbers (e.g., 901 and 902) may be to distinguish displays (410) that display different data over time.

[0112]

[0113] FIG. 10A and FIG. 10B are drawings for explaining a size selection guide of a target object according to one embodiment.

[0114] Referring to FIG. 10A, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3, electronic device (400) of FIG. 4) can provide a size selection guide (1003) of a target object (1000-1) to a user.

[0115] According to one embodiment, the electronic device (400) can display image data captured by a camera (e.g., the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, or the camera modules (311-316) of FIG. 3) on a first area (e.g., the entire area of ​​the display (1001)) of the display (1001) (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, the display (321) of FIG. 3, or the display (410) of FIG. 4). The electronic device (400) can identify objects included in the first area (e.g., the entire area of ​​the display (1001)). The electronic device (400) can determine a target object (1001-1) that the user wants to control among the identified objects based on a user input (e.g., a user input including a gaze input, a voice input, and / or a gesture input). For example, the electronic device (400) can receive an input for enlarging or reducing the size of a target object (1001-1) from the user.

[0116] According to one embodiment, the electronic device (400) may display a size selection guide (1003) of a target object (1001-1) on a first area (e.g., the entire area of ​​the display (1002)). The size selection guide (1003) may provide the user with an example of the size of the target object (1001-1) (e.g., the size of the second area). The size selection guide (1003) may include objects (1003-1 to 1003-3) having the same or similar shape as the target object (1001-1). The objects (1003-1 to 1003-3) may have the same or similar shape, but may have different sizes. The electronic device (400) can determine the size of the target object (1001-1) (e.g., the size of the second region) based on a user's input to the size selection guide (1003) (e.g., an input for one of the objects (1003-1 to 1003-3)).

[0117] Referring to FIG. 10A, according to one embodiment, if there are other identified objects besides the target object (1001-1) in the first area (e.g., the entire area of ​​the display (1001)), the electronic device (400) may generate a size selection guide (1003) by referencing the sizes of the other objects. For example, the electronic device (400) may refer to the largest object, the smallest object, and the closest object (e.g., the object closest to the target object) among the other objects.

[0118] Referring to FIG. 10B, according to one embodiment, if there are no other objects identified other than the target object (1004-1) in the first area (e.g., the entire area of ​​the display (1004)), the electronic device (400) may generate a size selection guide (1006) based on a specified setting value. For example, if any one of the objects (1006-1 to 1006-3) generated according to the specified setting value (e.g., 1006-3) exceeds the first area (e.g., the entire area of ​​the display (1005)), the object (1006-3) may be deactivated. If the user adjusts his / her gaze (e.g., looks upward from the previous gaze) so that the deactivated object (1006-3) no longer exceeds the first area (e.g., the entire area of ​​the display (1005)), the object (1006-3) may be reactivated.

[0119]

[0120] FIG. 11 is a diagram illustrating a case in which there are multiple target objects according to one embodiment.

[0121] Referring to FIG. 11, according to one embodiment, a target object may include multiple target objects. An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, the electronic device (300) of FIG. 3, or the electronic device (400) of FIG. 4) may display an object control interface for controlling the multiple target objects.

[0122] According to one embodiment, the electronic device (400) can display image data captured by a camera (e.g., the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, or the camera modules (311-316) of FIG. 3) on a first area (e.g., the entire area of ​​the display (1101)) of the display (1101) (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, the display (321) of FIG. 3, or the display (410) of FIG. 4). The electronic device (400) can identify objects included in the first area (e.g., the entire area of ​​the display (1101)). The electronic device (400) can determine a plurality of target objects that the user wishes to control from among the identified objects based on a user input (e.g., a user input including a gaze input, a voice input, and / or a gesture input).

[0123] According to one embodiment, the electronic device (400) can display a plurality of target objects on a plurality of second areas (1102-1, 1102-2) of the display (1102). Each of the plurality of second areas (1102-1, 1102-2) can stream image data corresponding to one of the plurality of target objects. The image data streamed on the plurality of second areas (1102-1, 1102-2) can be controlled based on a user input to the object control interface (1102-3). Referring to FIG. 11, although a single object control interface (1102-3) for controlling the plurality of target objects is illustrated, the present invention is not limited thereto. The electronic device (400) can also display a plurality of object control interfaces for controlling respective target objects.

[0124] According to one embodiment, the locations of the plurality of second areas (1102-1 and 1102-2) may be determined as locations that do not overlap with other objects (e.g., other objects identified on the first area) on the first area (e.g., the entire area of ​​the display (1102)).

[0125]

[0126] FIG. 12 is a drawing for explaining a case in which there are multiple cameras according to one embodiment.

[0127] Referring to FIG. 12, according to one embodiment, tracking of a target object may be possible depending on the number of cameras. An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, the electronic device (300) of FIG. 3, or the electronic device (400) of FIG. 4) may track a target object if it has multiple cameras.

[0128] According to one embodiment, the electronic device (400) can display image data captured by a camera (e.g., the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, or the camera modules (311-316) of FIG. 3) on a first area (e.g., the entire area of ​​the display (1201)) of the display (1201) (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, the display (321) of FIG. 3, or the display (410) of FIG. 4). The electronic device (400) can identify objects included in the first area (e.g., the entire area of ​​the display (1201)). The electronic device (400) can determine a target object (1201-1) that the user wants to control among the identified objects based on a user input (e.g., a user input including a gaze input, a voice input, and / or a gesture input).

[0129] According to one embodiment, the electronic device (400) can display a target object (1201-1) on a second area (1202-1) of a display (1202). The electronic device (400) can display an object control interface (1202-2) synchronized with image data streaming on the second area (1202-1). The image data streaming on the second area (1202-1) can be controlled based on a user input to the object control interface (1202-2).

[0130] According to one embodiment, the electronic device (400) can recognize the movement of the user's gaze (or head). Even if the target object is not identified in the image data captured by the camera due to the movement of the user's gaze (or head), the electronic device (400) can display the target object (1201-1) on the second area (1203-1) of the display (1203). For example, the electronic device (400) can display an object control interface (1203-2) synchronized with the image data streamed on the second area (1203-1). The image data streamed on the second area (1203-1) can be controlled based on a user input to the object control interface (1203-2). The target object may be tracked by any one of a plurality of cameras. When the electronic device (400) has a plurality of cameras, image data corresponding to the target object can be continuously streamed on the second area (1203-1).

[0131] FIG. 13 is a drawing for explaining a case in which the environment in which a target object is displayed changes according to one embodiment.

[0132] Referring to FIG. 13, according to one embodiment, the environment in which a target object is displayed may change. An electronic device (400) (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, or the electronic device (300) of FIG. 3) may display a target object (1301-1) in a second area of ​​a display (1301) (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, the display (321) of FIG. 3, or the display (410) of FIG. 4). The electronic device (400) may display a user interface (e.g., 1301-2) unrelated to the target object (1301-1) on the display (1301). The electronic device (400) may receive a user input for the user interface (1301-2).

[0133] In one embodiment, the electronic device (400) may switch the environment displayed on the display (1302) based on user input to the user interface (1301-2). Distinguishing the displays (e.g., 410) with different reference numbers (e.g., 1301, 1302, 1303, and 1304) may be to distinguish the displays (e.g., 410) that display different data over time. The electronic device (400) may have been displaying an AR environment on the display (1301). For example, the electronic device (400) may display image data captured by a camera (e.g., the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, or the camera modules (311-316) of FIG. 3) in a first area (e.g., the entire area) of the display (1301), and may display a result (e.g., an enlarged result) of processing at least a portion of the image data captured by the camera (e.g., image data corresponding to the target object (1301-1)) in a second area of ​​the display (1301). The electronic device (400) may switch an environment displayed on the display (1302) from an AR environment to a VR environment based on a user input to the user interface (1301-2).

[0134] According to one embodiment, the electronic device (400) can display a VR environment on the display (1303). Even within the VR environment, the electronic device (400) can display a target object (1301-1) on the second area (1303-1) of the display (1303). As the environment displayed by the electronic device (400) through the display (1303) changes, data (e.g., data based on real-world location information) (e.g., reference point information) utilized in the AR environment may be lost. For example, the second area (1303-2) where the electronic device (400) displayed the target object (1301-1) may move to the lower left of the display (1303). The second area where the target object (1301-1) was displayed on the display (1301) may be displayed as an afterimage on the display (1303) (e.g., 1303-2).

[0135] According to one embodiment, the electronic device (400) may move a second area (e.g., 1304-1) on the display (1304) that displays a target object (1301-1) based on a user input for an afterimage (1303-2). The electronic device (400) may display the target object (1301-1) on the second area (1304-1) of the display (1304). The electronic device (400) may display an object control interface (1304-2) synchronized with image data streamed on the second area (1304-1). The image data streamed on the second area (1304-1) may be controlled based on a user input for the object control interface (1304-2).

[0136]

[0137] FIG. 14 illustrates a flowchart of a method of operating an electronic device according to one embodiment.

[0138] Referring to FIG. 14, operations 1410 to 1440 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of each operation (1410 to 1440) may be changed, and at least two operations may be performed in parallel.

[0139] In operation 1410, an electronic device according to an embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (200) of FIG. 2, an electronic device (300) of FIG. 3, or an electronic device (400) of FIG. 4) may display image data captured by a camera (e.g., a camera module (180) of FIG. 1, a camera module (250) of FIG. 2, or camera modules (311-316) of FIG. 3) on a first area of ​​a display (e.g., a display module (160) of FIG. 1, a display member (201) of FIG. 2, a display (321) of FIG. 3, or a display (410) of FIG. 4).

[0140] In operation 1420, an electronic device according to one embodiment can identify objects included in a first area.

[0141] In operation 1430, an electronic device according to one embodiment may determine a target object that the user wishes to control among objects based on a user input including at least one of a gaze input, a voice input, or a gesture input.

[0142] In operation 1440, an electronic device according to one embodiment may display a target object in a second area of ​​a display. For example, the second area may be overlaid on the first area. For example, the second area may be a place where image data corresponding to the target object is streamed. For example, the image data streamed on the second area may be controlled based on user input to an object control interface associated with displaying, recording, and / or sharing the target object.

[0143]

[0144] According to one embodiment, an operating method of an electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (200) of FIG. 2, an electronic device (300) of FIG. 3, an electronic device (400) of FIG. 4) may include an operation of displaying image data captured by a camera (e.g., a camera module (180) of FIG. 1, a camera module (250) of FIG. 2, camera modules (311-316) of FIG. 3) in a first area of ​​a display (e.g., a display module (160) of FIG. 1, a display member (201) of FIG. 2, a display (321) of FIG. 3, a display (410) of FIG. 4). The operating method may include an operation of identifying objects included in the first area. The operating method may include an operation of determining a target object that the user wishes to control among the objects based on a user input including at least one of a gaze input, a voice input, and a gesture input. The method of operation may include displaying the target object in a second area of ​​the display. The second area may be overlaid on the first area. The second area may be a area where image data corresponding to the target object is streamed. The image data streamed on the second area may be controlled based on a user input to an object control interface of the target object.

[0145] In one embodiment, the user input may include at least one of first information utilized in determining the target object; or second information associated with at least one of displaying, recording, or sharing the target object. The user input providing the first information and the user input providing the second information may have different input types.

[0146] According to one embodiment, the operation of displaying in the second area may include an operation of determining a reference point of the target object. The operation of displaying in the second area may include an operation of determining a size and a position of the second area based on the reference point and the second information. The operation of displaying in the second area may include an operation of displaying the target object in the second area, where the size and the position are determined. The reference point may correspond to the coordinates of a central pixel located at the bottom among pixels constituting the target object.

[0147] According to one embodiment, the operation of determining the size and position of the second region may include an operation of displaying a size selection guide of the target object on the first region. The operation of determining the size and position of the second region may include an operation of determining the size of the second region based on a user input for the size selection guide.

[0148] In one embodiment, the size selection guide may be generated by referencing the size of another object identified in addition to the target object in the first area, if such another object exists. The size selection guide may be generated based on a preset value if no other object identified in addition to the target object exists in the first area.

[0149] According to one embodiment, the location of the second area may be determined as a location that does not overlap with another object on the first area, based on information of another object identified in addition to the target object in the first area.

[0150] According to one embodiment, the target object may include a plurality of target objects. The second region may include a plurality of second regions into which image data corresponding to any one of the plurality of target objects is streamed. The locations of the plurality of second regions may be determined to be locations that do not overlap with other objects in the first region.

[0151] According to one embodiment, the image data streamed on the second area may be recorded within the electronic device from the time the target object is determined.

[0152] In one embodiment, when there is only one camera, the target object may not be displayed on the second area at a time when the target object is not identified in the image data captured by the camera. When there are multiple cameras, the target object may be tracked by any one of the multiple cameras, so that image data corresponding to the target object may be continuously streamed on the second area.

[0153] According to one embodiment, the operating method may include an operation of adjusting at least one of a position or a size of the second area based on a subsequent user input to the object control interface. The operating method may include an operation of updating whether to record the second area based on the subsequent input. The operating method may include an operation of performing sharing of the second area based on the subsequent input. The object control interface may be associated with displaying, recording, and sharing the target object.

[0154] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3, electronic device (400) of FIG. 4) may include a display (e.g., display module (160) of FIG. 1, display member (201) of FIG. 2, display (321) of FIG. 3, display (410) of FIG. 4). The electronic device may include one or more processors (e.g., processor (120) of FIG. 1, processor (420) of FIG. 4). The electronic device may include a memory (e.g., memory (130) of FIG. 1, memory (430) of FIG. 4) that stores instructions. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to display image data captured by a camera in a first area of ​​the display. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to identify objects included in the first area. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to determine a target object that the user wishes to control among the objects based on a user input including at least one of a gaze input, a voice input, or a gesture input. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to display the target object in a second area of ​​the display. The second area may be an overlay on the first area. The second area may be a area in which image data corresponding to the target object is streamed. The image data streamed on the second area may be controlled based on a user input to an object control interface of the target object.

[0155] In one embodiment, the user input may include at least one of first information utilized in determining the target object; or second information associated with at least one of displaying, recording, or sharing the target object. The user input providing the first information and the user input providing the second information may have different input types.

[0156] According to one embodiment, the image data streamed on the second area may be recorded within the electronic device from the time the target object is determined.

[0157] In one embodiment, when there is only one camera, the target object may not be displayed on the second area at a time when the target object is not identified in the image data captured by the camera. When there are multiple cameras, the target object may be tracked by any one of the multiple cameras, so that image data corresponding to the target object may be continuously streamed on the second area.

[0158] According to one embodiment, the instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to determine a reference point of the target object. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to determine a size and a position of the second area based on the reference point and the second information. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to display the target object in the second area, the size and the position of which have been determined. The reference point may correspond to coordinates of a central pixel located at the bottom among pixels constituting the target object.

[0159] According to one embodiment, the instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to display a size selection guide of the target object on the first area. The instructions, when individually or collectively executed by the one or more processors, may cause the electronic device to determine a size of the second area based on a user input for the size selection guide.

[0160] In one embodiment, the size selection guide may be generated by referencing the size of another object identified in addition to the target object in the first area, if such another object exists. The size selection guide may be generated based on a preset value if no other object identified in addition to the target object exists in the first area.

[0161] According to one embodiment, the location of the second area may be determined as a location that does not overlap with another object on the first area, based on information of another object identified in addition to the target object in the first area.

[0162] According to one embodiment, the target object may include a plurality of target objects. The second region may include a plurality of second regions into which image data corresponding to any one of the plurality of target objects is streamed. The locations of the plurality of second regions may be determined to be locations that do not overlap with other objects in the first region.

[0163]

[0164] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0165] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0166] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0167] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., built-in memory or external memory) readable by a machine (e.g., an electronic device). For example, a processor (e.g., a processor) of the machine (e.g., an electronic device) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0168] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0169] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0170]

[0171] - Explanation of the symbol

[0172] 101: Electronic Devices

[0173] 200: Electronic devices

[0174] 300: Electronic devices

[0175] 400: Electronic devices

Claims

1. In the operating method of an electronic device (101; 200; 300; 400), An operation of displaying image data captured by a camera (180; 250; 311; 312; 313; 314; 315; 316) on a first area of ​​a display (160; 201; 321; 410); An action for identifying objects included in the first area; An operation of determining a target object that the user wishes to control among the objects based on a user input including at least one of gaze input, voice input, or gesture input; and An action of displaying the target object in the second area of ​​the display (160; 201; 321; 410). Including, The above second area is overlaid on the above first area, The second region is where image data corresponding to the target object is streamed, The image data streamed on the second area is controlled based on user input to the object control interface of the target object. How it works.

2. In paragraph 1, The above user input is, First information used to determine the target object; or Second information associated with at least one of displaying, recording, or sharing the target object; Contains at least one of: The user input providing the first information and the user input providing the second information have different input types. How it works.

3. In either of paragraphs 1 and 2, The actions displayed in the second area above are: An action for determining a reference point of the above target object; An operation of determining the size and position of the second area based on the above reference point and the second information; and An action of displaying the target object in the second area where the size and location are determined. Including, The above reference point corresponds to the coordinates of the central pixel located at the bottom among the pixels constituting the target object. How it works.

4. In any one of paragraphs 1 to 3, The operation of determining the size and location of the above second area is: An operation of displaying a size selection guide of the target object on the first area; and An operation of determining the size of the second area based on the user's input for the size selection guide. A method of operation, comprising:

5. In any one of paragraphs 1 to 4, The above size selection guide is, If there is another object identified in addition to the target object in the first area, it is generated by referencing the size of the other object. If there is no other object identified other than the target object in the above first area, it is generated based on the preset settings. How it works.

6. In any one of paragraphs 1 to 5, The location of the above second area is, Based on information of other objects identified in addition to the target object in the first area, the location is determined to not overlap with the other objects in the first area. How it works.

7. In any one of paragraphs 1 to 6, The above target object is, Contains multiple target objects, The second area above is, comprising a plurality of second regions into which image data corresponding to any one of the plurality of target objects is streamed; The locations of the above plurality of second regions are: It is determined that the location does not overlap with other objects in the first area above, How it works.

8. In any one of paragraphs 1 to 7, The image data streamed on the second area is, From the time the target object is determined, it is recorded within the electronic device (101; 200; 300; 400). How it works.

9. In any one of paragraphs 1 to 8, In the case where there is one camera (180; 250; 311; 312; 313; 314; 315; 316), at a time when the target object is not identified in the image data captured by the camera (180; 250; 311; 312; 313; 314; 315; 316), the target object is not displayed on the second area. In case there are multiple cameras (180; 250; 311; 312; 313; 314; 315; 316), the target object is tracked by any one of the multiple cameras (180; 250; 311; 312; 313; 314; 315; 316), so that image data corresponding to the target object is continuously streamed on the second area. How it works.

10. In any one of paragraphs 1 to 9, An action of adjusting at least one of the position or size of the second region based on a subsequent user input to the object control interface; An operation of updating whether or not to record the second area based on the above subsequent input; or An operation of performing sharing for the second area based on the above subsequent input. Including at least one more of: The above object control interface is, Associated with displaying, recording, and sharing the above target object, How it works.

11. In electronic devices (101; 200; 300; 400), Display (160; 201; 321; 410) One or more processors (120; 420); and Memory for storing instructions (130; 430) Including, The above instructions, when individually or collectively executed by the one or more processors (120; 420), cause the electronic device (101; 200; 300; 400) to: Displaying image data captured by a camera (180; 250; 311; 312; 313; 314; 315; 316) in a first area of ​​a display (160; 201; 321; 410), Identify the objects included in the first area above, Based on user input including at least one of gaze input, voice input, or gesture input, determine a target object that the user wishes to control among the objects, To display the above target object in the second area of ​​the display (160; 201; 321; 410), The above second area is overlaid on the above first area, The second region is where image data corresponding to the target object is streamed, The image data streamed on the second area is controlled based on user input to the object control interface of the target object. Electronic devices (101; 200; 300; 400).

12. In paragraph 11, The above user input is, First information used to determine the target object; or Second information associated with at least one of displaying, recording, or sharing the target object; Contains at least one of: The user input providing the first information and the user input providing the second information have different input types. Electronic devices (101; 200; 300; 400).

13. In either of paragraphs 11 and 12, The image data streamed on the second area is, From the time the target object is determined, it is recorded within the electronic device (101; 200; 300; 400). Electronic devices (101; 200; 300; 400).

14. In any one of paragraphs 11 to 13, In the case where there is one camera (180; 250; 311; 312; 313; 314; 315; 316), at a time when the target object is not identified in the image data captured by the camera (180; 250; 311; 312; 313; 314; 315; 316), the target object is not displayed on the second area. In case there are multiple cameras (180; 250; 311; 312; 313; 314; 315; 316), the target object is tracked by any one of the multiple cameras (180; 250; 311; 312; 313; 314; 315; 316), so that image data corresponding to the target object is continuously streamed on the second area. Electronic devices (101; 200; 300; 400).

15. In any one of paragraphs 11 to 14, The above instructions, when individually or collectively executed by the one or more processors (120; 420), cause the electronic device (101; 200; 300; 400) to: Determine the reference point of the above target object, Based on the above reference point and the second information, the size and position of the second area are determined, To display the target object in the second area where the size and location are determined, The above reference point corresponds to the coordinates of the central pixel located at the bottom among the pixels constituting the target object. Electronic devices (101; 200; 300; 400).

Images