Electronic device capable of vision recognition and operating method thereof

Abstract

An electronic device according to an embodiment of the present disclosure may comprise: a first vision sensor configured to acquire visual data; a memory storing instructions; and at least one processor. The instructions, when executed by the at least one processor, may cause the electronic device to: acquire the visual data from the first vision sensor; convert at least one component for the visual data on the basis of requirements of a designated vision engine; and perform vision recognition using the designated vision engine on the basis of the visual data in which the at least one component has been converted.

Description

Electronic device capable of vision recognition and its operating method

[0001] The present disclosure relates to an electronic device capable of vision recognition and an operating method thereof according to one embodiment.

[0002] With the advancement of electronic technology, various types of electronic devices, such as mobile terminals, personal digital assistants (PDAs), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices, are being widely used. For example, electronic devices can provide virtual reality (VR), which allows users to have a realistic experience in a computer-generated virtual world; augmented reality (AR), which adds virtual information (or objects) to the real world; and mixed reality (MR), which combines virtual reality and augmented reality.

[0003] Wearable type electronic devices that are worn by the user, such as head-mounted devices, require compact space utilization, and to utilize the space, a technology may be required to provide light corresponding to the screen display of the display to the user's eyes through a lens.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] An electronic device according to one embodiment of the present disclosure may include a first vision sensor configured to acquire visual data, a memory storing instructions, and at least one processor. The instructions, when executed by the at least one processor, may cause the electronic device to acquire the visual data from the first vision sensor. The instructions, when executed by the at least one processor, may cause the electronic device to transform at least one component of the visual data based on a requirement of a designated vision engine. The instructions, when executed by the at least one processor, may cause the electronic device to perform vision recognition using the designated vision engine based on the visual data resulting from the transformation of the at least one component.

[0006] An operating method of an electronic device according to one embodiment of the present disclosure may include an operation of acquiring visual data from a first vision sensor. The operating method may include an operation of transforming at least one component of the visual data based on requirements of a specified vision engine. The operating method may include an operation of performing vision recognition using the specified vision engine based on the visual data resulting from the transformation of the at least one component.

[0007] A storage medium storing computer-readable instructions according to one embodiment of the present disclosure, wherein the instructions, when executed by at least one processor of an electronic device, cause the electronic device to acquire visual data from a first vision sensor configured to acquire the visual data. The instructions, when executed by at least one processor of the electronic device, cause the electronic device to transform at least one component of the visual data based on a requirement of a designated vision engine. The instructions, when executed by at least one processor of the electronic device, cause the electronic device to perform vision recognition using the designated vision engine based on the visual data resulting from the transformation of the at least one component.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment of the present disclosure.

[0009] FIG. 2 is a perspective view illustrating the internal configuration of a wearable device according to one embodiment of the present disclosure.

[0010] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to one embodiment.

[0011] FIG. 4 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0012] Figures 5a, 5b, 5c, 5d and 5e illustrate visual data according to various embodiments.

[0013] FIG. 6 illustrates adaptation of visual data according to one embodiment of the present disclosure.

[0014] FIG. 7 is a flowchart of an operating method of an electronic device according to one embodiment of the present disclosure.

[0015] Figure 8 is a flowchart regarding the initial evaluation execution operation of Figure 7.

[0016] FIG. 9 is a flowchart of a transformation operation of at least one component of the visual data of FIG. 7.

[0017] FIG. 10 is an example of converting a color profile of visual data according to one embodiment of the present disclosure.

[0018] FIG. 11 is an example of adjusting parameters related to color of visual data according to one embodiment of the present disclosure.

[0019] FIG. 12 and FIG. 13 illustrate examples of fusing comparison data with visual data according to various embodiments of the present disclosure.

[0020] FIG. 14 illustrates an example of fusing comparison data with visual data according to one embodiment of the present disclosure.

[0021] FIG. 15 is an example of data transformation for an image according to one embodiment of the present disclosure.

[0022] FIG. 16 is an example of feature integration for evaluation of visual data according to one embodiment of the present disclosure.

[0023] FIG. 17 is an example of image upscaling according to one embodiment of the present disclosure.

[0024] FIG. 18 and FIG. 19 are examples of color adjustment and hand detection according to one embodiment of the present disclosure.

[0025] FIG. 20 and FIG. 21 are examples of feature extraction and vision recognition according to one embodiment of the present disclosure.

[0026] FIG. 22a illustrates an example of hand detection through vision recognition according to one embodiment of the present disclosure.

[0027] FIG. 22b, FIG. 22c and FIG. 22d are examples of application of hand detection according to various embodiments of the present disclosure.

[0028] FIG. 23a and FIG. 23b illustrate examples of facial recognition through vision recognition according to one embodiment of the present disclosure.

[0029] FIG. 24 illustrates an example of transforming visual data to meet the requirements of a vision engine, according to one embodiment of the present disclosure.

[0030] FIG. 25 illustrates an example of correcting a distortion model of visual data according to one embodiment of the present disclosure.

[0031] FIG. 26 illustrates an example of recognizing a subject according to one embodiment of the present disclosure.

[0032] FIGS. 27A and 27B illustrate examples of provision of metaverses according to one embodiment of the present disclosure.

[0033] FIG. 28 illustrates an example of conversion to a point cloud according to one embodiment of the present disclosure.

[0034] FIG. 29 and FIG. 30 illustrate examples of transformation of visual data according to one embodiment of the present disclosure.

[0035] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to one embodiment of the present disclosure.

[0036] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0037] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0038] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0039] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0040] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0041] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0042] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0043] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0044] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0045] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0046] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0047] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0048] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0049] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0050] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0051] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0052] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0053] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0054] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0055] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0056] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0057] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0058] FIG. 2 is a perspective view illustrating the internal configuration of a wearable device (200) according to one embodiment of the present disclosure.

[0059] Referring to FIG. 2, the wearable device (200) is an electronic device in the form of glasses, which allows a user to visually perceive surrounding objects or the environment while wearing the wearable device (200). For example, the wearable device (200) may be a head-mounted device (HMD) or smart glasses that can directly provide images in front of the user's eyes. The configuration of the wearable device (200) of FIG. 2 may be all or part of the same as the configuration of the electronic device (101) of FIG. 1.

[0060] According to one embodiment, the wearable device (200) may include a housing (210) that forms the exterior of the wearable device (200). The housing (210) may provide a space in which components of the wearable device (200) may be placed. For example, the housing (210) may include a lens frame (202) and at least one wearing member (203).

[0061] According to one embodiment, the wearable device (200) may include a display member (201) capable of providing visual information to a user. For example, the display member (201) may include a module equipped with a lens or a second window member, a display, a waveguide, and / or a touch circuit. According to one embodiment, the display member (201) may be formed transparently or translucently. According to one embodiment, the display member (201) may include a window member made of a translucent material or a window member whose light transmittance can be adjusted as a coloring concentration is adjusted. According to one embodiment, the display members (201) are provided as a pair, and may be arranged to correspond to the left and right eyes of the user, respectively, when the wearable device (200) is worn on the user's body.

[0062] In one embodiment, the lens frame (202) can accommodate at least a portion of the indicator member (201). For example, the lens frame (202) can surround at least a portion of an edge of the indicator member (201). In one embodiment, the lens frame (202) can position at least one of the indicator members (201) relative to a user's eye. In one embodiment, the lens frame (202) can be a rim of a typical eyeglass structure. In one embodiment, the lens frame (202) can include at least one closed curve surrounding the indicator member (201).

[0063] In one embodiment, the wearing member (203) may extend from the lens frame (202). For example, the wearing member (203) may extend from an end of the lens frame (202) and, together with the lens frame (202), may be supported or positioned on the user's body (e.g., an ear). In one embodiment, the wearing member (203) may be rotatably coupled to the lens frame (202) via a hinge structure (229). In one embodiment, the wearing member (203) may include an inner side (231c) configured to face the user's body and an outer side (231d) opposite the inner side.

[0064] According to one embodiment, the wearable device (200) may include a hinge structure (229) configured to fold the wearing member (203) relative to the lens frame (202). The hinge structure (229) may be positioned between the lens frame (202) and the wearing member (203). When the wearable device (200) is not being worn, the user may fold the wearing member (203) so that a portion overlaps the lens frame (202) and carry or store the device.

[0065] The wearable device (200) may include components accommodated in a housing (210) (e.g., at least one circuit board (241) (e.g., a printed circuit board (PCB), a printed board assembly (PBA), a flexible PCB (FPCB), or a rigid-flexible PCB (RFPCB)), at least one battery (243), at least one speaker module (245), at least one power transmission structure (246), and a camera module (250)). The configuration of the housing (210) of FIG. 2B may be all or part of the same as the configuration of the display member (201), the lens frame (202), the wearing member (203), and the hinge structure (229) of FIG. 2A.

[0066] According to one embodiment, the wearable device (200) may acquire and / or recognize a visual image of an object or environment in a direction (e.g., -Y direction) that the user is looking at or that the wearable device (200) is facing by using a camera module (250) (e.g., the camera module (180) of FIG. 1), and may receive information about the object or environment from an external electronic device (e.g., the electronic device (102, 104) of FIG. 1 or the server (108)) through a network (e.g., the first network (198) or the second network (199) of FIG. 1). In another embodiment, the wearable device (200) may provide the received information about the object or environment to the user in an acoustic or visual form. The wearable device (200) may provide the received information about the object or environment to the user in a visual form through a display member (201) by using a display module (e.g., the display module (160) of FIG. 1). For example, the wearable device (200) can implement augmented reality by visualizing information about objects or the environment and combining it with actual images of the user's surroundings.

[0067] According to one embodiment, the display member (201) may include a first side (F1) facing a direction in which external light is incident (e.g., -Y direction) and a second side (F2) facing a direction opposite to the first side (F1) (e.g., +Y direction). When a user wears the wearable device (200), at least a portion of light or an image incident through the first side (F1) may pass through the second side (F2) of the display member (201) arranged to face the user's left eye and / or right eye and be incident on the user's left eye and / or right eye.

[0068] According to one embodiment, the lens frame (202) may include at least two frames. For example, the lens frame (202) may include a first frame (202a) and a second frame (202b). According to one embodiment, when a user wears the wearable device (200), the first frame (202a) may be a frame that faces the user's face, and the second frame (202b) may be a part of the lens frame (202) that is spaced apart from the first frame (202a) in a direction of the user's gaze (e.g., -Y direction).

[0069] According to one embodiment, the light output module (211) can provide images and / or videos to the user. For example, the light output module (211) can include a display panel (not shown) capable of outputting videos, and a lens (not shown) corresponding to the user's eyes and guiding the videos to the display member (201). For example, the user can obtain videos output from the display panel (not shown) of the light output module (211) through the lens (not shown) of the light output module (211).

[0070] According to one embodiment, the light output module (211) may include a display panel (not shown) configured to display various information. For example, the display panel (not shown) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). According to one embodiment, when the display panel (not shown) and / or the light output module (211) include one of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the wearable device (200) may include a light source that irradiates light to a display area of ​​the light output module (211) and / or the display member (201).

[0071] According to one embodiment, when the display panel (not shown) and / or the light output module (211) includes either an organic light emitting diode or a micro LED, the wearable device (200) can provide a virtual image to the user without including a separate light source. According to one embodiment, when the display panel (not shown) is implemented with an organic light emitting diode or a micro LED, a light source is unnecessary, and thus the wearable device (200) can be made lightweight.

[0072] In one embodiment, a lens (not shown) may serve to adjust the focus so that the screen output to the display panel (not shown) can be viewed by the user. For example, the lens (not shown) may be configured as a Fresnel lens, a pancake lens, or a multi-channel lens.

[0073] In one embodiment, a waveguide may serve to transmit light generated from a display panel (not shown) to a user's eye. For example, the waveguide may be made of glass, plastic, or polymer, and may include nano-patterns formed on a portion of an inner or outer surface, for example, a grating structure having a polygonal or curved shape. In one embodiment, light incident on one end of the waveguide may be propagated within the waveguide by the nano-patterns and provided to the user. In addition, a waveguide formed of a free-form prism may provide the incident light to the user through a reflective mirror. In one embodiment, the waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or a reflective element (e.g., a reflective mirror). In one embodiment, the waveguide may guide light from a display panel (not shown) emitted from a light source to the user's eye by using at least one diffractive element or reflective element.

[0074] According to one embodiment, the diffractive element may include an input optical member (not shown) / output optical member (not shown). For example, the input optical member (not shown) may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a light source (e.g., Micro LED) to transmit the light to a transparent member (e.g., a first transparent member, a second transparent member) of a screen display unit. The output grating area may serve as an outlet that diffracts (or reflects) light transmitted to the display member (201) of the waveguide to a user's eye.

[0075] In one embodiment, the reflective element may include a total internal reflection (TIR) ​​optical element (not shown) or a total internal reflection waveguide (not shown) for total internal reflection. For example, total internal reflection may refer to a method of guiding light such that light (e.g., a virtual image) entering through an input grating region is 100% reflected from one side (e.g., a specific side) of the waveguide, thereby transmitting 100% of the light to the output grating region.

[0076] In one embodiment, light emitted from a display panel (not shown) may be guided along an optical path through an input optical element into a waveguide. Light traveling within the waveguide may be guided toward a user's eye through an output optical element. The screen display may be determined based on the light emitted toward the user's eye.

[0077] In one embodiment, at least a portion of the light output module (211) may be disposed within the housing (210). For example, the light output module (211) may be disposed on the wearable member (203) or the lens frame (202) to correspond to the user's right eye and left eye, respectively. In one embodiment, the light output module (211) may be connected to the display member (201) and may provide an image to the user through the display member (201).

[0078] According to one embodiment, the circuit board (241) may include components for driving the wearable device (200). For example, the circuit board (241) may include at least one integrated circuit chip, and at least one of the processor (120), the memory (130), the power management module (188), or the communication module (190) of FIG. 1 may be provided to the integrated circuit chip. According to one embodiment, the circuit board (241) may be disposed within the wearing member (203) of the housing (210). According to one embodiment, the circuit board (241) may be electrically connected to the battery (243) through the power transmission structure (246). According to one embodiment, the circuit board (241) is connected to a flexible printed circuit board (205) and can transmit electrical signals to electronic components of the electronic device (e.g., a light output module (211), a camera module (250), and a light emitting unit) through the flexible printed circuit board (205). According to one embodiment, the circuit board (241) may be a circuit board including an interposer.

[0079] According to various embodiments, the flexible printed circuit board (205) may extend from the circuit board (241) across the hinge structure (229) into the interior of the lens frame (202) and may be disposed around at least a portion of the perimeter of the indicator member (201) within the lens frame (202).

[0080] According to one embodiment, a battery (243) (e.g., battery (189) of FIG. 1) may be electrically connected to components of a wearable device (200) (e.g., light output module (211), circuit board (241), speaker module (245), microphone module (247), and camera module (250)) and may supply power to the components of the wearable device (200).

[0081] In one embodiment, at least a portion of the battery (243) may be disposed on the wearable member (203). In one embodiment, the battery (243) may be disposed on an end (203a, 203b) of the wearable member (203). For example, the battery (243) may include a first battery (243a) disposed on a first end (203a) of the wearable member (203) and a second battery (243b) disposed on a second end (203b).

[0082] According to various embodiments, the speaker module (245) (e.g., the audio module (170) or the sound output module (155) of FIG. 1) may convert an electrical signal into sound. At least a portion of the speaker module (245) may be disposed within the wearable member (203) of the housing (210). In one embodiment, the speaker module (245) may be positioned within the wearable member (203) so as to correspond to the user's ear. For example, the speaker module (245) may be disposed between the circuit board (241) and the battery (243).

[0083] In one embodiment, the power transmission structure (246) can transmit power from the battery (243) to an electronic component (e.g., an optical output module (211)) of the wearable device (200). For example, the power transmission structure (246) is electrically connected to the battery (243) and / or the circuit board (241), and the circuit board (241) can transmit power received through the power transmission structure (246) to the optical output module (211). In one embodiment, the power transmission structure (246) can be connected to the circuit board (241) through the speaker module (245). For example, when the wearable device (200) is viewed from the side (e.g., in the Z-axis direction), the power transmission structure (246) can at least partially overlap the speaker module (245).

[0084] In one embodiment, the power transmission structure (246) may be a configuration capable of transmitting power. For example, the power transmission structure (246) may include a flexible printed circuit board or wires. For example, the wires may include a plurality of cables (not shown). In various embodiments, the shape of the power transmission structure (246) may be modified in various ways, taking into account the number and / or type of cables.

[0085] According to one embodiment, the microphone module (247) (e.g., the input module (150) and / or the audio module (170) of FIG. 1) can convert sound into an electrical signal. According to one embodiment, the microphone module (247) can be disposed on at least a portion of the lens frame (202). For example, at least one microphone module (247) can be disposed on the bottom (e.g., in the direction toward the -X axis) and / or the top (e.g., in the direction toward the X axis) of the wearable device (200). According to one embodiment, the wearable device (200) can recognize the user's voice more clearly by using voice information (e.g., sound) acquired from the at least one microphone module (247). For example, the wearable device (200) can distinguish voice information from ambient noise based on the acquired voice information and / or additional information (e.g., low-frequency vibration of the user's skin and bones). For example, the wearable device (200) can clearly recognize the user's voice and perform a function of reducing ambient noise (e.g., noise canceling).

[0086] In one embodiment, the camera module (250) can capture still images and / or moving images. The camera module (250) may include at least one of a lens, at least one image sensor, an image signal processor, or a flash. In one embodiment, the camera module (250) may be disposed within the lens frame (202) and may be disposed around the display member (201).

[0087] According to one embodiment, the camera module (250) may include a light-emitting unit that can be attached to various locations. In one embodiment, the light-emitting unit may be used as an auxiliary means to facilitate detection of a user's gaze through the first camera module (251). In one embodiment, the light-emitting unit may be attached around a hinge structure (229) connecting the lens frame (202) and the wearing member (203), or adjacent to a second camera module (253) disposed between the lens frame (202), and may be used as a means to supplement ambient brightness during shooting. In particular, the light-emitting unit may be effective when subject detection is not easy in a dark environment.

[0088] According to one embodiment, the camera module (250) may include at least one first camera module (251). According to one embodiment, the first camera module (251) may capture a trajectory of a user's eye (e.g., pupil) or gaze. For example, the first camera module (251) may capture a reflection pattern of light emitted by a light emitting unit toward the user's eye. For example, the light emitting unit may emit light in an infrared band for tracking a trajectory of gaze using the first camera module (251). For example, the light emitting unit may include an IR LED. According to one embodiment, a processor (e.g., processor (120) of FIG. 1) may adjust the position of the virtual image projected on the display member (201) so that the virtual image corresponds to a direction in which the user's pupil is looking. According to one embodiment, the first camera module (251) may include a global shutter (GS) type camera, and may track the trajectory of the user's eyes or gaze using a plurality of first camera modules (251) having the same specifications and performance.

[0089] According to various embodiments, the first camera module (251) may periodically or aperiodically transmit information related to the trajectory of the user's eyes or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1). According to another embodiment, the first camera module (251) may transmit the trajectory information to the processor when it detects that the user's gaze has changed (e.g., the eyes move more than a reference value while the head is still) based on the trajectory information.

[0090] According to various embodiments, the camera module (250) may include a second camera module (253). According to one embodiment, the second camera module (253) may capture external images. According to one embodiment, the second camera module (253) may be a camera of a global shutter type or a rolling shutter (RS) type. According to one embodiment, the second camera module (253) may capture external images through a second optical hole (223) formed in the second frame (202b). For example, the second camera module (253) may include a high-resolution color camera and may be a high-resolution (HR) or photo video (PV) camera. In addition, the second camera module (253) may provide an auto focus (AF) function and an optical image stabilizer (OIS) function. The second camera module (253) according to one embodiment of the present document may include one camera or a plurality of cameras.

[0091] According to one embodiment, the wearable device (200) may include a flash (not shown) positioned adjacent to the second camera module (253). For example, the flash (not shown) may provide light to increase the brightness (e.g., illuminance) around the wearable device (200) when the second camera module (253) acquires an external image, and may reduce difficulties in acquiring images due to dark environments, mixing of various light sources, and / or reflection of light.

[0092] In one embodiment, the camera module (250) may include at least one third camera module (255). In one embodiment, the third camera module (255) may capture a user's movement or recognize a space through the first optical hole (221) formed in the lens frame (202). For example, the third camera module (255) may detect a user's hand and capture a user's gesture (e.g., hand movement). For example, the third camera module (255) may track the movement of the user's head or recognize the surrounding space.

[0093] According to one embodiment, the third camera module (255) and / or the first optical hole (221) may be disposed at opposite side ends of the lens frame (202) (e.g., the second frame (202b)), for example, at opposite ends of the lens frame (202) (e.g., the second frame (202b)) in the X direction, respectively. According to one embodiment, the third camera module (255) may be a global shutter (GS) type camera. For example, the third camera module (255) may be a camera that supports 3DoF (degrees of freedom) or 6DoF, and may provide 360-degree space (e.g., omnidirectional), position recognition, and / or movement recognition.

[0094] According to one embodiment, the third camera module (255) may perform a movement path tracking function (simultaneous localization and mapping, SLAM) and a user movement recognition function using a plurality of global shutter (GS) type cameras of the same standard and performance as a stereo camera. According to one embodiment, the third camera module (255) may include an infrared (IR) camera (e.g., a time of flight (TOF) camera or a structured light camera). For example, the IR camera may operate as at least a part of a sensor module (e.g., the sensor module (176) of FIG. 1) for detecting a distance to a subject.

[0095] According to one embodiment, at least one of the first camera module (251) or the third camera module (255) may be replaced with a sensor module (e.g., the sensor module (176) of FIG. 1) (e.g., a Lidar sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode. For example, the photodiode may include a positive intrinsic negative (PIN) photodiode or an avalanche photodiode (APD). The photodiode may be referred to as a photo detector or a photo sensor.

[0096] According to one embodiment, at least one of the first camera module (251), the second camera module (253), or the third camera module (255) may include a plurality of camera modules (not shown). For example, the second camera module (253) may be configured with a plurality of lenses (e.g., wide-angle and telephoto lenses) and image sensors and may be arranged on one side (e.g., the side facing the -Y axis) of the wearable device (200). For example, the wearable device (200) may include a plurality of camera modules, each having a different property (e.g., angle of view) or function, and may be controlled to change the angle of view of the camera module based on a user's selection and / or trajectory information. For example, at least one of the plurality of camera modules may be a wide-angle camera, and at least another may be a telephoto camera.

[0097] According to various embodiments, a processor (e.g., processor (120) of FIG. 1) may determine movement of the wearable device (200) and / or movement of the user by using information of the wearable device (200) acquired using at least one of a gesture sensor, a gyro sensor, or an acceleration sensor of a sensor module (e.g., sensor module (176) of FIG. 1) and a user's motion (e.g., approach of the user's body to the wearable device (200)) acquired using the first camera module (251). According to one embodiment, the wearable device (200) may include, in addition to the described sensors, a magnetic (geomagnetic) sensor capable of measuring direction using a magnetic field and magnetism, and / or a Hall sensor capable of acquiring movement information (e.g., direction of movement or distance of movement) using the strength of a magnetic field. For example, the processor can determine movement of the wearable device (200) and / or movement of the user based on information obtained from a magnetic (geomagnetic) sensor and / or a Hall sensor.

[0098] According to various embodiments (not shown), the wearable device (200) can perform an input function (e.g., a touch and / or pressure sensing function) that enables interaction with a user. For example, a component configured to perform a touch and / or pressure sensing function (e.g., a touch sensor and / or a pressure sensor) may be disposed on at least a portion of the wearable member (203). The wearable device (200) can control a virtual image output through the display member (201) based on information acquired through the component. For example, the sensor related to the touch and / or pressure sensing function may be configured in various ways, such as a resistive type, a capacitive type, an electromagnetic induction (EM) type, or an optical type. According to one embodiment, the component configured to perform the touch and / or pressure sensing function may be all or partly identical to the configuration of the input module (150) of FIG. 1.

[0099] According to one embodiment, the wearable device (200) may include a reinforcing member (260) disposed in the internal space of the lens frame (202) and formed to have a rigidity higher than the rigidity of the lens frame (202).

[0100] In one embodiment, the wearable device (200) may include a lens structure (270). The lens structure (270) may refract at least a portion of light. For example, the lens structure (270) may be a prescription lens having a predetermined refractive power. In one embodiment, the lens structure (270) may be positioned behind (e.g., in the +Y direction) the second window member of the display member (201). For example, the lens structure (270) may be positioned between the display member (201) and the user's eye. For example, the lens structure (270) may face one side of the display member.

[0101] In one embodiment, the housing (210) may include a hinge cover (227) that may conceal a portion of the hinge structure (229). Another portion of the hinge structure (229) may be accommodated or concealed between the inner case (231) and the outer case (233), which will be described later.

[0102] In one embodiment, the wearable member (203) may include an inner case (231) and an outer case (233). The inner case (231) is, for example, a case configured to face the user's body or to come into direct contact with the user's body, and may be made of a material having low thermal conductivity, for example, a synthetic resin. In one embodiment, the inner case (231) may include an inner side facing the user's body (e.g., the inner side (231c) of FIG. 2A). The outer case (233) may include, for example, a material capable of at least partially transmitting heat (e.g., a metal material) and may be coupled to face the inner case (231). In one embodiment, the outer case (233) may include an outer side opposite the inner side (231c) (e.g., the outer side (231d) of FIG. 2A). In one embodiment, at least one of the circuit board (241) or the speaker module (245) may be accommodated in a space separated from the battery (243) within the wearable member (203). In the illustrated embodiment, the inner case (231) may include a first case (231a) including the circuit board (241) or the speaker module (245) and a second case (231b) accommodating the battery (243), and the outer case (233) may include a third case (233a) coupled to face the first case (231a) and a fourth case (233b) coupled to face the second case (231b). For example, a first case (231a) and a third case (233a) may be combined (hereinafter, “first case portion (231a, 233a)”) to accommodate a circuit board (241) and / or a speaker module (245), and a second case (231b) and a fourth case (233b) may be combined (hereinafter, “second case portion (231b, 233b)”) to accommodate a battery (243).

[0103] According to various embodiments, the first case portion (231a, 233a) is rotatably connected to the lens frame (202) via a hinge structure (229), and the second case portion (231b, 233b) can be connected or mounted to an end of the first case portion (231a, 233a) via a connecting member (235). In some embodiments, among the connecting members (235), a portion that comes into contact with the user's body can be made of a material having low thermal conductivity, for example, an elastic material such as silicone, polyurethane, or rubber, and a portion that does not come into contact with the user's body can be made of a material having high thermal conductivity, for example, a metal material. For example, when heat is generated in the circuit board (241) or the battery (243), the connecting member (235) can block the heat from being transferred to the portion that comes into contact with the user's body, and can disperse or release the heat through the portion that does not come into contact with the user's body. According to one embodiment, a portion of the connecting member (235) that is configured to come into contact with the user's body may be interpreted as a portion of the inner case (231), and a portion of the connecting member (235) that is not configured to come into contact with the user's body may be interpreted as a portion of the outer case (233). According to one embodiment (not shown), the first case (231a) and the second case (231b) may be configured as an integral body without a connecting member (235), and the third case (233a) and the fourth case (233b) may be configured as an integral body without a connecting member (235). According to various embodiments, in addition to the illustrated components, other components (e.g., the antenna module (197) of FIG. 1) may be further included, and information about an object or environment may be provided from an external electronic device (e.g., the electronic device (102, 104) or the server (108) of FIG. 1) through a network (e.g., the first network (198) or the second network (199) of FIG. 1) using the communication module (190).

[0104] In FIG. 2, only the wearable device (200) is illustrated and described, but it is not limited thereto, and some components of the wearable device (200) illustrated in FIG. 2 may also be included in electronic devices such as smartphones and tablet PCs.

[0105] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device (300) according to one embodiment.

[0106] Referring to FIGS. 3A and 3B, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or depth sensors (317) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on the first surface (310) of the housing.

[0107] In one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0108] In one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the camera modules (311, 312) can also be used for hand detection and tracking, and user gesture.

[0109] In one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as determining the distance to an object, such as time of flight (TOF). For example, the depth sensor (317) may measure the distance to the subject using near-infrared, ultrasound, or laser. In one embodiment, the depth sensor (317) may measure the time of flight (TOF) of a signal by emitting a signal from a transmitter and measuring the signal from a receiver.

[0110] According to one embodiment, a camera module (325, 326) for facial recognition and / or a display (321) (and / or a lens) may be disposed on the second side (320) of the housing.

[0111] In one embodiment, a face recognition camera module (325, 326) adjacent to the display (321) may be used to recognize a user's face, or may recognize and / or track both eyes of the user. In one embodiment, the face recognition camera module (325, 326) may detect or track a user's facial expression.

[0112] In one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIGS. 2A and 2B .

[0113] As described above, according to one embodiment, the wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured on a body part of the user. The wearable electronic device (101) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0114] FIG. 4 is a block diagram of an electronic device (101) according to one embodiment of the present disclosure.

[0115] Referring to FIG. 4, an electronic device (101, e.g., the electronic device (101) of FIG. 1, the wearable device (200) of FIG. 2A, or the wearable electronic device (300) of FIG. 3) according to one embodiment may include a processor (410), a memory (420), a first vision sensor (430), and a second vision sensor (450).

[0116] A processor (410, e.g., processor (120) of FIG. 1) according to one embodiment may control at least one other component (e.g., hardware or software component) of the electronic device (101). In one embodiment, the processor (410) may perform various data processing or operations, and as at least a part of the data processing or operations, may store commands or data received from other components in a memory (420), process the commands or data stored in the memory (420), and store result data in the memory (420).

[0117] A memory (420, e.g., memory (130) of FIG. 1) according to one embodiment can store instructions executable by the processor (510).

[0118] According to one embodiment, the processor (410) may include a designated vision engine that performs vision recognition. In one embodiment, the processor (410) may perform vision recognition corresponding to visual data by inputting visual data into the designated vision engine. In one embodiment, the designated vision engine may provide an artificial intelligence (AI)-based vision solution trained using a designated type of visual data.

[0119] According to one embodiment, the first vision sensor (430, e.g., the sensor module (176) or the camera module (180) of FIG. 1) and / or the second vision sensor (450, e.g., the sensor module (176) or the camera module (180) of FIG. 1) may each be a vision sensor that acquires visual data. In one embodiment, the first vision sensor (430) and the second vision sensor (450) may be the same sensor so as to acquire the same type of visual data. In one embodiment, the first vision sensor (430) and the second vision sensor (450) may be different types of sensors so as to acquire different types of visual data.

[0120] In one embodiment, the visual data may be an image or video. In one embodiment, the first vision sensor (430) and / or the second vision sensor (450) may be a camera sensor that acquires an image or video.

[0121] In one embodiment, the visual data may be surface data such as a depth map or mesh. In one embodiment, the first vision sensor (430) and / or the second vision sensor (450) may be a depth sensor using time of flight (ToF), Lidar, stereo pair, or structured light.

[0122] An electronic device (101) according to one embodiment may further include a display configured to display a screen in front of a user's eyes (e.g., a display module (160) of FIG. 1, a display member (201) of FIG. 2A, or a display (321) of FIG. 3).

[0123] In addition to the configurations shown and described, an electronic device (101) according to one embodiment may further include or exclude some of the configurations included in the electronic device (101) of FIG. 1, the wearable device (200) of FIG. 2A, or the wearable electronic device (300) of FIG. 3.

[0124] Figures 5a, 5b, 5c, 5d and 5e illustrate visual data according to various embodiments.

[0125] Referring to FIGS. 5a, 5b, 5c, 5d, and 5e, an electronic device (101) according to one embodiment can obtain various types of visual data through a first vision sensor (430) and / or a second vision sensor (450).

[0126] An electronic device (101) according to one embodiment can acquire various types of visual data, such as RGB image data, infrared image data, depth data, point cloud, or hyperspectral image data.

[0127] In one embodiment, the electronic device (101) may acquire depth data, RGB image data, and / or a point cloud as visual data corresponding to the same scene, as illustrated in FIGS. 5A, 5B, and 5C. In one embodiment, the electronic device (101) may acquire hyperspectral image data as visual data, as illustrated in FIG. 5D.

[0128] An electronic device (101) according to one embodiment can acquire visual data of a specified type in response to hardware characteristics of a first vision sensor (430) and / or a second vision sensor (450).

[0129] In one embodiment, the electronic device (101) can acquire distorted infrared image data, as illustrated in FIG. 5e. For example, there are various distortion models that describe the mathematical deviation of a camera from the pinhole model. For example, various distortion models exist, such as the Polynomial Radial, Brown-Conrady, and Kannala-Brandt models, and selecting an appropriate model can affect the quality of vision recognition.

[0130] An electronic device (101) according to one embodiment can obtain a 360-degree view image or a night vision image that cannot be input into a standard vision solution applied to a designated vision engine. According to the prior art, when the electronic device (101) obtains visual data other than a designated type, such as a 360-degree view image or a night vision image, through the first vision sensor (430) and / or the second vision sensor (450), it was difficult to use a designated vision engine that uses a standard vision solution. According to the prior art, in order for the electronic device (101) to implement a vision engine corresponding to all types of visual data or all types of vision sensors, it may incur a great deal of time and cost.

[0131] FIG. 6 illustrates adaptation of visual data according to one embodiment of the present disclosure.

[0132] Referring to FIG. 6, an electronic device (101) according to one embodiment may provide a technology for preprocessing visual data of a first vision sensor (430) and / or a second vision sensor (450) to input the data into a designated vision engine and provide a vision recognition solution.

[0133] In one embodiment, the electronic device (101) may transform at least one component of visual data acquired from the first vision sensor (430) and / or the second vision sensor (450) to adapt in response to a designated vision engine. In one embodiment, the electronic device (101) may preprocess the visual data of the first vision sensor (430) and / or the second vision sensor (450) with a focus on function / data / requirement generalization.

[0134] Accordingly, the electronic device (101) can provide a precise and efficient vision recognition solution using a designated vision engine, independently of the type of visual data.

[0135] FIG. 7 is a flowchart illustrating an operation method of an electronic device (101) according to one embodiment of the present disclosure. FIG. 8 is a flowchart illustrating an initial evaluation operation of FIG. 7. FIG. 9 is a flowchart illustrating an operation of transforming at least one component of the visual data of FIG. 7.

[0136] Referring to FIG. 7, an electronic device (101) according to one embodiment may obtain visual data from a first vision sensor (430) in operation 710.

[0137] An electronic device (101) according to one embodiment may acquire visual data using a first vision sensor (430) and / or a second vision sensor (450) for vision recognition using a designated vision engine. In one embodiment, the electronic device (101) may perform vision recognition for detecting a user's hand or recognizing a hand gesture. In one embodiment, the electronic device (101) may perform vision recognition for facial recognition, subject recognition, or map generation.

[0138] According to one embodiment, the first vision sensor (430) can obtain visual data related to detection of a user's hand or recognition of a hand gesture. According to one embodiment, the electronic device (101) can detect the user's hand based on the vision recognition result performed using the visual data obtained from the first vision sensor (430). According to one embodiment, the electronic device (101) can recognize a hand gesture based on the vision recognition result performed using the visual data obtained from the first vision sensor (430).

[0139] According to one embodiment, the first vision sensor (430) can obtain visual data related to face recognition, subject recognition, or map generation. According to one embodiment, the electronic device (101) can recognize a person's face or a subject based on the vision recognition result performed using the visual data obtained from the first vision sensor (430). According to one embodiment, the electronic device (101) can generate a map through a simultaneous localization and map-building (SLAM) function based on the vision recognition result performed using the visual data obtained from the first vision sensor (430).

[0140] An electronic device (101) according to one embodiment may perform an initial evaluation to identify characteristics of a first vision sensor (430) or visual data at operation 730.

[0141] Referring further to FIG. 8, the electronic device (101) according to one embodiment may, at least as part of the operation of performing the initial evaluation, at operation 731, identify a characteristic of the first vision sensor (430) or visual data. In one embodiment, the electronic device (101) may identify a characteristic of the first vision sensor (430). For example, the electronic device (101) may identify a type, profile, and / or characteristic of the visual data output from the first vision sensor (430). In one embodiment, the electronic device (101) may identify a characteristic of the visual data.

[0142] An electronic device (101) according to one embodiment may, in operation 733, dynamically correct at least one component of visual data. In one embodiment, the electronic device (101) may dynamically correct at least one component based on the identified first vision sensor (430) or a characteristic of the visual data.

[0143] An electronic device (101) according to one embodiment may, at operation 735, evaluate at least one component of the visual data.

[0144] An electronic device (101) according to one embodiment may repeat operations 731, 733, and 735 at least once. In one embodiment, the electronic device (101) may repeat the operations to improve the quality of the evaluation.

[0145] An electronic device (101) according to one embodiment may store, in operation 737, an attribute of an evaluated component for visual data. For example, the electronic device (101) may store an evaluation result of at least one attribute for visual data corresponding to a first vision sensor (430) in a memory (e.g., memory (520) of FIG. 5 ).

[0146] Referring again to FIG. 7, the electronic device (101) according to one embodiment may, at operation 750, transform at least one component of the visual data based on the requirements of a specified vision engine.

[0147] In one embodiment, the electronic device (101) can identify requirements for a specified vision engine. For example, the electronic device (101) can identify requirements for a specified vision engine for optimal quality of vision recognition. For example, the electronic device (101) can identify requirements for a specified vision engine based on at least one component corresponding to visual data used to train the specified vision engine.

[0148] In one embodiment, the electronic device (101) can adapt the visual data acquired from the first vision sensor (430) to respond to identified requirements. In one embodiment, the electronic device (101) can convert the visual data into different types depending on the number of adaptation methods.

[0149] In one embodiment, the electronic device (101) can transform at least one component of the visual data by separating, merging, resizing, and / or cropping the image to support various aspect ratios and resolutions.

[0150] In one embodiment, the electronic device (101) can convert visual data to correspond to another type of vision sensor. For example, the electronic device (101) can convert visual data corresponding to a color image into a depth map, or convert visual data corresponding to a depth map into a color image.

[0151] In one embodiment, the electronic device (101) can change the color profile or color-related parameters of visual data. For example, the electronic device (101) can change the grayscale visual data to RGB type. For example, the electronic device (101) can change the RGB type visual data to hyperspectral type.

[0152] In one embodiment, the electronic device (101) can adjust a noise model of visual data. For example, a noise model may be particularly important for visual data used for SLAM or depth map generation.

[0153] In one embodiment, the electronic device (101) can adjust a distortion model of visual data. Additionally, the electronic device (101) can transform at least one component of the visual data in addition to the examples described above.

[0154] Referring to FIG. 8, an electronic device (101) according to one embodiment may, as at least part of an operation of transforming at least one component of the visual data, identify a number or quality of at least one component of the visual data at operation 753.

[0155] In one embodiment, the electronic device (101) can identify the number of at least one component corresponding to the visual data and identify a quality corresponding to each component. In one embodiment, the electronic device (101) can perform the evaluation operation a plurality of times based on the number of at least one component identified.

[0156] In one embodiment, the electronic device (101) may, at operation 755, evaluate the number or quality of at least one component based on the requirements of a designated vision engine. In one embodiment, the electronic device (101) may generate a plan to adjust at least one component based on the number and / or quality of the at least one component.

[0157] An electronic device (101) according to one embodiment may, in operation 757, transform at least one component based on the evaluation result.

[0158] In one embodiment, the electronic device (101) can change the resolution of visual data using a super resolution method.

[0159] In one embodiment, the electronic device (101) may adjust parameters related to the color of the visual data. For example, parameters related to color may include gamma, hue, or saturation.

[0160] In one embodiment, the electronic device (101) can geometrically transform visual data.

[0161] In one embodiment, the electronic device (101) can fuse visual data with another image. In one embodiment, the electronic device (101) can obtain comparison data corresponding to the visual data from the second vision sensor (450) and fuse the obtained comparison data with the visual data. For example, the comparison data may differ from at least one component of the visual data.

[0162] An electronic device (101) according to one embodiment can identify, in operation 770, whether a parameter related to the quality of vision recognition is maximized.

[0163] An electronic device (101) according to one embodiment may perform vision recognition using a designated vision engine based on visual data in which at least one component has been transformed, in operation 790, when a parameter related to the quality of vision recognition is maximized (operation 770 - Yes).

[0164] According to one embodiment, the electronic device (101) may repeat the operation of transforming at least one component of the visual data of operation 750 if the parameter related to the quality of vision recognition is not maximized (operation 770-No).

[0165] FIG. 10 is an example of converting a color profile of visual data according to one embodiment of the present disclosure.

[0166] Referring to FIG. 10, visual data according to one embodiment may be acquired in a grayscale format. For example, the first vision sensor (430) may generate an image in a grayscale format.

[0167] An electronic device (101) according to one embodiment can adjust the color of visual data in various ways using a colorizer. For example, the electronic device (101) can adjust the color of visual data in response to a plurality of LEVELs (260, 310, 360, 410, 470).

[0168] In one embodiment, the electronic device (101) can extract color-related parameters by using visual data with various color adjustments. In one embodiment, the electronic device (101) can maximize the number of parameters and obtain RGB parameters according to LEVEL. In one embodiment, the electronic device (101) can update and calibrate the RGB parameters of the visual data in response to parameter values ​​according to LEVEL.

[0169] An electronic device (101) according to one embodiment can perform the above colorization operation multiple times, thereby converting visual data of grayscale type into RGB type.

[0170] FIG. 11 is an example of adjusting parameters related to color of visual data according to one embodiment of the present disclosure.

[0171] Referring to FIG. 11, an electronic device (101) according to one embodiment can obtain an image frame from a first vision sensor (430). For example, the first vision sensor (430) can generate an RGB type image frame.

[0172] An electronic device (101) according to one embodiment can adjust parameters related to the color of visual data based on a LEVEL corresponding to a specified vision engine. In one embodiment, the electronic device (101) can generate an image frame with adjusted parameters related to the color of an input image frame.

[0173] In one embodiment, the electronic device (101) can adjust the tone of the image frame. In one embodiment, the electronic device (101) can adjust the curve of the image frame. In one embodiment, the electronic device (101) can adjust the hue, saturation, and / or brightness of the image frame.

[0174] An electronic device (101) according to one embodiment may set a LEVEL for adjusting parameters related to the color of visual data. In one embodiment, the electronic device (101) may set the LEVEL based on a designated vision engine. For example, the LEVEL may be set based on the purpose of the vision engine, such as whether the designated vision engine is for hand detection, SLAM functionality, and / or depth perception.

[0175] In one embodiment, the electronic device (101) can set the LEVEL according to the characteristics of the visual data (e.g., Shi-Tomasi, Fast9, Harris).

[0176] FIG. 12 and FIG. 13 illustrate examples of fusing comparison data with visual data according to various embodiments of the present disclosure.

[0177] Referring to FIGS. 12 and 13, an electronic device (101) according to one embodiment may further include a second vision sensor (450) separately from the first vision sensor (430). In one embodiment, the second vision sensor (450) may have at least one component that is different from the visual data obtained from the first vision sensor (430).

[0178] In one embodiment, the first vision sensor (430) and the second vision sensor (450) can each acquire images of corresponding scenes. For example, the first vision sensor (430) and the second vision sensor (450) can each acquire differently arranged images of the same scene.

[0179] As illustrated in FIG. 12, the first vision sensor (430) and the second vision sensor (450) according to one embodiment may each acquire different types of images. For example, the first vision sensor (430) may be a black-and-white camera configured to acquire black-and-white images, and the second vision sensor (450) may be a color camera configured to acquire color images.

[0180] As illustrated in FIG. 13, the types of images acquired by the first vision sensor (430) and the second vision sensor (450) according to one embodiment may be the same. For example, the first vision sensor (430) and the second vision sensor (450) may both be HDR (high dynamic range) cameras that acquire high contrast images. In one embodiment, the images acquired by the first vision sensor (430) and the second vision sensor (450) may differ in at least one component. For example, the visual data of the first vision sensor (430) and the comparison data of the second vision sensor (450) may have different exposure settings.

[0181] An electronic device (101) according to one embodiment can fuse visual data acquired through a first vision sensor (430) with comparison data acquired through a second vision sensor (450).

[0182] As illustrated in FIG. 12, an electronic device (101) according to one embodiment can convert a high-quality black-and-white image acquired from a first vision sensor (430) into a high-quality color image by fusing it with a color image acquired from a second vision sensor (450).

[0183] As illustrated in FIG. 12, an electronic device (101) according to one embodiment can increase a dynamic range and reduce noise by fusing a first HDR image acquired from a first vision sensor (430) with a second HDR image acquired from a second vision sensor (450).

[0184] FIG. 14 illustrates an example of fusing comparison data with visual data according to one embodiment of the present disclosure.

[0185] Referring to FIG. 14, an electronic device (101) according to one embodiment can obtain an RGB image frame from a first vision sensor (430). An electronic device (101) according to one embodiment can obtain a grayscale type stereo image frame from a second vision sensor (450).

[0186] An electronic device (101) according to one embodiment may preprocess an RGB image frame acquired from a first vision sensor (430) and a grayscale type stereo image frame acquired from a second vision sensor (450) to register the images, respectively. In one embodiment, the electronic device (101) may register each image frame based on extrinsic matrices of an RGB-grayscale camera.

[0187] An electronic device (101) according to one embodiment can fuse an RGB image frame acquired from a first vision sensor (430) with a grayscale type stereo image frame acquired from a second vision sensor (450).

[0188] An electronic device (101) according to one embodiment can post-process a fused image that fuses an RGB image frame and a grayscale type stereo image frame.

[0189] An electronic device (101) according to one embodiment can convert an RGB image frame acquired from a first vision sensor (430) into a colorized stereo image by fusing it with a grayscale type stereo image frame acquired from a second vision sensor (450).

[0190] FIG. 15 is an example of data transformation for an image according to one embodiment of the present disclosure.

[0191] Referring to FIG. 15, an electronic device (101) according to one embodiment may generate depth data by converting at least one component of raw depth data. For example, the raw depth data may be obtained from various sources (e.g., various vision sensors).

[0192] In one embodiment, the designated vision engine of the electronic device (101) may be a vision engine for understanding an integrated spatial scene. In one embodiment, the designated vision engine of the electronic device (101) may be a vision engine for 3D edge detection, morphology extraction, and spatial segmentation.

[0193] An electronic device (101) according to one embodiment can obtain visual data from a first vision sensor (430) or a second vision sensor (450). In one embodiment, a grayscale image including depth data can be obtained. In one embodiment, the electronic device (101) can obtain a color image (e.g., RGBD). In one embodiment, the electronic device (101) can register each image obtained from the first vision sensor (430) or the second vision sensor (450).

[0194] An electronic device (101) according to one embodiment can convert a registered image including depth data into a depth image whose depth can be visually understood through a learned variational autoencoder (VAE).

[0195] An electronic device (101) according to one embodiment can transform at least one component based on the requirements of a designated vision engine corresponding to 3D edge detection, shape extraction, or space segmentation through a transformed depth image.

[0196] An electronic device (101) according to one embodiment may extract features from depth data adapted to transform at least one component, and / or evaluate features.

[0197] An electronic device (101) according to one embodiment may perform additional feature extraction from a reconstructed spatial scene. In one embodiment, the electronic device (101) may perform post-processing based on a request from another vision engine.

[0198] FIG. 16 is an example of feature integration for evaluation of visual data according to one embodiment of the present disclosure.

[0199] Referring to FIG. 16, an electronic device (101) according to one embodiment can extract features of visual data acquired from a first vision sensor (430) and / or a second vision sensor (450) and integrate the extracted features.

[0200] In one embodiment, the electronic device (101) can acquire an image of an infrared camera and a grayscale depth image, respectively, and extract features from each of the acquired images. In one embodiment, the electronic device (101) can integrate the features extracted from each of the images.

[0201] As an example of feature integration, colorization, depth transformation, camera model transfer, separation and unification, and superpixels can be performed.

[0202] In one embodiment, the electronic device (101) can compare the features of the integrated image corresponding to each image through a comparator. In one embodiment, the electronic device (101) can compare the features of the integrated image at a semantic level with respect to the application program interface (API) of a specified vision engine.

[0203] In one embodiment, the electronic device (101) may extract features of an image using a pre-trained network, and generate a semantic latent vector and a residual latent vector using a semantic encoder and a residual encoder, respectively. In one embodiment, the electronic device (101) may generate transformed image features using the semantic latent vector and the residual latent vector through a decoder.

[0204] In one embodiment, the electronic device (101) can extract semantic data from an image and compare the extracted semantic data. For example, a semantic encoder can capture data related to all attributes included in the image features. For example, a residual encoder can capture data unrelated to attributes.

[0205] FIG. 17 is an example of image upscaling according to one embodiment of the present disclosure.

[0206] Referring to FIG. 17, an electronic device (101) according to one embodiment can transform at least one component to upscale visual data based on a parameter related to a size corresponding to the visual data.

[0207] In one embodiment, the electronic device (101) can acquire images of various sizes from the first vision sensor (430) and / or the second vision sensor (450). In one embodiment, a designated vision engine of the electronic device (101) can request a designated image size, and the requirements of the designated vision engine can include parameters related to the size of the image.

[0208] In one embodiment, the electronic device (101) may resize the acquired image by interpolating or resizing it. In this case, a problem may arise in which features related to the quality of the image are degraded.

[0209] In one embodiment, the electronic device (101) can upscale the acquired image so as to maintain the characteristics related to the image quality. For example, the electronic device (101) can adjust the size of the image using ESRGAN, which is an artificial intelligence-based upscaler. As illustrated in FIG. 17, when upscaling an image of size 56X56 to size 640X640, it can be confirmed that the quality of the image is maintained compared to the case of resizing.

[0210] FIG. 18 and FIG. 19 are examples of color adjustment and hand detection according to one embodiment of the present disclosure.

[0211] Referring to FIGS. 18 and 19, an electronic device (101) according to one embodiment may perform hand detection using a designated vision engine. For example, the designated vision engine may be trained on color images and may not operate on black and white images.

[0212] In one embodiment, the electronic device (101) may acquire a grayscale image from the first vision sensor (430) and / or the second vision sensor (450). In one embodiment, the electronic device (101) may convert the acquired grayscale image into a color image through colorization conversion. In one embodiment, the electronic device (101) may automatically perform the colorization conversion based on the requirements of a specified vision engine.

[0213] In one embodiment, the electronic device (101) can perform hand detection using a designated vision engine using a converted color image. In one embodiment, the electronic device (101) was unable to detect a hand from a grayscale image due to a black and white color profile, but can successfully detect a hand from a color image converted through colorization conversion.

[0214] FIG. 20 and FIG. 21 are examples of feature extraction and vision recognition according to one embodiment of the present disclosure.

[0215] Referring to FIGS. 20 and 21, an electronic device (101) according to one embodiment may extract at least one feature from visual data. In one embodiment, the electronic device (101) may perform vision recognition based on the at least one extracted feature.

[0216] In one embodiment, the electronic device (101) may perform vision recognition using features extracted from a convolutional / pooling layer or a variational autoencoder (VAE), instead of directly utilizing visual data or images. For example, such algorithms may be used to collect pre-trained representations of arbitrary images.

[0217] In one embodiment, the electronic device (101) can extract one or more features from an image using various training models. For example, AlexNet, ResNet, VGG, and Unet are provided as examples of training models, but other training models may also be used.

[0218] In one embodiment, the electronic device (101) may process features acquired from images using each training model through a vision engine API. In one embodiment, the electronic device (101) may perform vision recognition based on features of images acquired from one or more vision engine APIs through the vision engine core.

[0219] An electronic device (101) according to one embodiment can perform vision recognition through preprocessing of a grayscale image to correspond to a designated vision engine, as illustrated in FIG. 21. An electronic device (101) according to one embodiment can extract a depth map and a color image from visual data through a designated vision engine, and perform vision recognition based on the depth map and the color image.

[0220] In one embodiment, the electronic device (101) can acquire a grayscale image from the first vision sensor (430) or the second vision sensor (450). In one embodiment, the electronic device (101) can colorize the acquired grayscale image through a colorizer. In one embodiment, the electronic device (101) can generate a depth map for estimating the depth of the colorized image. In one embodiment, the electronic device (101) can adjust the color of the colorized image.

[0221] In one embodiment, the electronic device (101) can extract features of the generated depth map and color-adjusted image based on each vision engine API. In one embodiment, the electronic device (101) can perform vision recognition based on the features of the image extracted through the core of the designated vision engine.

[0222] FIG. 22a illustrates an example of hand detection through vision recognition according to one embodiment of the present disclosure. FIGS. 22b, 22c, and 22d illustrate application examples of hand detection according to various embodiments of the present disclosure.

[0223] Referring to FIGS. 22a, 22b, 22c, and 22d, an electronic device (101) according to one embodiment may provide content related to augmented reality (AR), virtual reality (VR), or extended reality (XR), and receive interaction input based on a user's gesture therefor.

[0224] In one embodiment, as illustrated in FIGS. 22b, 22c, and 22d, the electronic device (101) may display a screen designated for the tracked hand, receive input for game operation as a gesture, or provide educational entertainment (edutainment) according to a gesture.

[0225] In one embodiment, the designated vision engine of the electronic device (101) may be trained using a deep learning network and may be trained using color images. For example, a designated vision engine trained using color images may be trained using color-based feature points or hand features, which are calculated based on color from the color image and learned based on light intensity, making them more robust to various influences than features from grayscale images.

[0226] An electronic device (101) according to one embodiment may include a first vision sensor (430) or a second vision sensor (450) that acquires a grayscale image rather than an RGB camera, taking into account the complexity of data processing and limited power consumption.

[0227] In one embodiment, the electronic device (101) may obtain a grayscale image from the first vision sensor (430) or the second vision sensor (450), convert the obtained grayscale image into a color image, and perform hand detection or gesture recognition through a designated vision engine.

[0228] FIG. 23a and FIG. 23b illustrate examples of facial recognition through vision recognition according to one embodiment of the present disclosure.

[0229] Referring to FIGS. 23a and 23b, an electronic device (101) according to one embodiment can recognize a human face from an acquired image and detect an expression from the recognized face.

[0230] In one embodiment, the electronic device (101) can recognize a human face through a designated vision engine and recognize facial expressions from the recognized face. In one embodiment, the designated vision engine may be pre-trained to recognize facial expressions from facial images, and may be pre-trained with color images. Face detection for facial expression recognition may be more effective in color images than in grayscale images, because skin color distribution and Laws texture energy in a five-color space are used to represent facial features.

[0231] In one embodiment, the electronic device (101) may acquire a grayscale image from the first vision sensor (430) or the second vision sensor (450), convert the acquired grayscale image into a color image, and perform facial recognition or expression detection through a designated vision engine.

[0232] FIG. 24 illustrates an example of transforming visual data to meet the requirements of a vision engine, according to one embodiment of the present disclosure.

[0233] Referring to FIG. 24, an electronic device (101) according to one embodiment can convert acquired visual data to correspond to the requirements of a vision engine.

[0234] An electronic device (101) according to one embodiment can obtain an orthographic projection image from a first vision sensor (430) or a second vision sensor (450).

[0235] In one embodiment, the designated vision engine may require input of a fisheye projection image. In one embodiment, the designated vision engine may require input of a color image.

[0236] An electronic device (101) according to one embodiment can convert an orthogonal projection image into a colorized fish-eye projection image in response to the requirements of a specified vision engine. In one embodiment, the electronic device (101) can utilize various image conversion methods.

[0237] FIG. 25 illustrates an example of correcting a distortion model of visual data according to one embodiment of the present disclosure.

[0238] Referring to FIG. 25, an electronic device (101) according to one embodiment can correct a distortion model of acquired visual data to meet the requirements of a vision engine. In one embodiment, a designated vision engine can perform instance segmentation from an input image.

[0239] In one embodiment, the electronic device (101) may correct a distortion model of an image acquired from the first vision sensor (430) or the second vision sensor (450) in response to the requirements of a designated vision engine. Accordingly, the quality of instance segmentation in the designated vision engine may be improved.

[0240] FIG. 26 illustrates an example of recognizing a subject according to one embodiment of the present disclosure.

[0241] Referring to FIG. 26, an electronic device (101) according to one embodiment can obtain a grayscale image from a first vision sensor (430) or a second vision sensor (450). In one embodiment, the electronic device (101) can convert the obtained grayscale image into a color image.

[0242] An electronic device (101) according to one embodiment can recognize a subject through vision recognition using a converted color image. In one embodiment, the electronic device (101) can detect a specified object through vision recognition.

[0243] An electronic device (101) according to one embodiment may provide content related to augmented reality (AR), virtual reality (VR), or extended reality (XR) that displays the results of vision recognition.

[0244] FIGS. 27A and 27B illustrate examples of provision of metaverses according to one embodiment of the present disclosure.

[0245] Referring to FIGS. 27A and 27B , an electronic device (101) according to one embodiment can acquire a grayscale image and provide a metaverse based on the grayscale image. However, since the electronic device (101) is based on grayscale, it may fail to perform 3D sensing, thereby generating a screen with broken rendering.

[0246] An electronic device (101) according to one embodiment can collect different characteristics regarding resolution, field of view, and optionally information about stereo lenses, depending on the different types of cameras.

[0247] In one embodiment, one or more components of visual data acquired from a camera can be transformed. According to one embodiment, the electronic device (101) can output an accurate rendering similar to an image of a real environment using a metaverse vision engine.

[0248] FIG. 28 illustrates an example of conversion to a point cloud according to one embodiment of the present disclosure.

[0249] Referring to FIG. 28, an electronic device (101) according to one embodiment can convert a surface mesh or grayscale image into a point cloud using a specified algorithm.

[0250] In one embodiment, the specified algorithm can extract a concise 3D surface model of a scene from actual sensor data. In one embodiment, the specified algorithm can use as input a surface mesh describing an image registered with the scene. In one embodiment, the specified algorithm can output a surface mesh with fewer vertices and faces that map the shape of the scene.

[0251] In one embodiment, the electronic device (101) can generate an accurate and realistic description of a scene even in the presence of sensor noise, sparse data, and incomplete scene description, according to a specified algorithm.

[0252] In one embodiment, the generated point cloud can be utilized in a vision engine that provides instance segmentation.

[0253] FIG. 29 and FIG. 30 illustrate examples of transformation of visual data according to one embodiment of the present disclosure.

[0254] Referring to FIGS. 29 and 30, in the case of automatic image enhancement according to the comparative example, the image can be transformed (e.g., coloring and contrast of a human face, coloring of some objects (e.g., buildings outside a window), or making a distorted face more beautiful) to make the image bright enough for the photographer's eyes.

[0255] In contrast, the present disclosure may target as many features as possible, as many textured scenes as possible, as many non-uniform / inhomogeneous as possible, projection models, camera-specific distortions, and color characteristics as possible for the vision engine of the electronic device (101).

[0256] According to an electronic device (101) according to one embodiment, when a scene with a strong color or a specific color is displayed, as illustrated in FIG. 29, the camera may react accordingly. The case regarding color is merely an example, and more complex cases may also be included.

[0257] In one embodiment, the electronic device (101) may include a depth camera, a ToF camera, or a stereo module for acquiring depth data. In one embodiment, the electronic device (101) may acquire depth data through matching from a mono camera.

[0258] In one embodiment, an electronic device (101) that does not include a depth camera can transform an acquired video stream into characteristics required by a vision engine corresponding to hand tracking, as illustrated in FIG. 30. For example, a vision engine corresponding to hand tracking can operate by colorizing depth using a pinhole projection camera model. According to one embodiment, the electronic device (101) can dynamically provide real-time stream characteristic transformation.

[0259] For the camera system according to the comparative example, only the camera settings such as exposure, gain, white balance, and FoV can be optimized to best suit the machine vision task.

[0260] In contrast, an electronic device (101) according to one embodiment of the present disclosure can estimate and adjust parameters in real time during all working hours based on an image conversion application, rather than in a predefined manner.

[0261] Accordingly, the electronic device (101) can track a response to a specific influence. For example, the camera system of the present disclosure can track a response corresponding to a TV screen with an arbitrary subject (e.g., a hand), an object with an arbitrary color change, or other arbitrary influences on a parameter (e.g., a hand 3D pose, a focal length, a screen distortion, a screen brightness).

[0262] The law of the random parameter probability distribution for a system with this structure can be unique and identical for each implementation. Therefore, whether the method of the present disclosure has been implemented can be easily determined by comparing the probability distributions of known output parameters with those of the obtained output parameters (e.g., matrix values ​​or 3D hand poses).

[0263] For example, a stationary random process (noise) with a Gaussian distribution can follow the input of the algorithm, and this noise can be added to the focal length values. Data can be collected based on the output of the correction matrix, a differential probability distribution can be constructed, and a fitting criterion (e.g., a chi-square test) can be used to check whether the determined distribution law converges with the expected distribution law.

[0264] The technical tasks to be achieved in this document are not limited to the technical tasks mentioned above, and other technical tasks not mentioned will be clearly understood by those with ordinary skill in the technical field to which this document pertains from the description below.

[0265] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0266] An electronic device (101; 200; 300) according to one embodiment of the present disclosure may include a first vision sensor (430; 176; 180) configured to acquire visual data, a memory (420; 130) storing instructions, and at least one processor (410, 120). The instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to acquire the visual data from the first vision sensor (430; 176; 180). The instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to transform at least one component of the visual data based on requirements of a designated vision engine. The above instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to perform vision recognition using the designated vision engine based on visual data that has transformed the at least one component.

[0267] In an electronic device (101; 200; 300) according to one embodiment, the first vision sensor (430; 176; 180) may be configured to obtain visual data related to detection of a user's hand or recognition of a hand gesture.

[0268] In an electronic device (101; 200; 300) according to one embodiment, the first vision sensor (430; 176; 180) may be configured to acquire visual data related to facial recognition, facial expression recognition, or map generation.

[0269] In an electronic device (101; 200; 300) according to one embodiment, the instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to perform an initial evaluation identifying a characteristic of the first vision sensor (430; 176; 180) or the visual data, and to identify or transform the at least one component based on a result of the initial evaluation, at least as part of an operation of transforming the at least one component.

[0270] In an electronic device (101; 200; 300) according to one embodiment, the instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to, as at least part of an operation of transforming the at least one component, identify a number or quality of the at least one component of the visual data, evaluate the number or quality of the at least one component based on requirements of the designated vision engine, and transform the at least one component based on a result of the evaluation.

[0271] In an electronic device (101; 200; 300) according to one embodiment, the instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to repeat the transformation of the at least one component based on a parameter related to the quality of the vision recognition, at least as part of an operation of transforming the at least one component.

[0272] In an electronic device (101; 200; 300) according to one embodiment, at least one component of the visual data may include a parameter related to a color of the visual data. The instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to adjust the parameter related to the color, at least as part of an operation of transforming the at least one component.

[0273] An electronic device (101; 200; 300) according to one embodiment may further include a second vision sensor (450; 176; 180). The instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to, as at least part of an operation of transforming the at least one component, obtain comparison data corresponding to the visual data from the second vision sensor (450; 176; 180), wherein at least some of the at least one component is different, and to fuse the obtained comparison data with the visual data.

[0274] In an electronic device (101; 200; 300) according to one embodiment, at least one component of the visual data may include a parameter related to a size corresponding to the visual data. The instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to upscale the visual data based on the parameter related to the size, at least as a part of an operation of transforming the at least one component.

[0275] In an electronic device (101; 200; 300) according to one embodiment, the instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to extract at least one feature from the visual data, respectively, and to perform the vision recognition based on the at least one feature extracted through the designated vision engine, as at least part of an operation of performing vision recognition using the designated vision engine.

[0276] In an electronic device (101; 200; 300) according to one embodiment, the instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to extract a depth map and a color image from the visual data, respectively, and to perform the vision recognition based on the depth map and the color image through the designated vision engine, as at least a part of an operation of performing vision recognition using the designated vision engine.

[0277] According to one embodiment of the present disclosure, an operating method of an electronic device (101; 200; 300) may include an operation (710) of acquiring visual data from a first vision sensor (430; 176; 180). According to one embodiment, an operating method of an electronic device (101; 200; 300) may include an operation (750) of converting at least one component of the visual data based on a requirement of a specified vision engine. According to one embodiment, an operating method of an electronic device (101; 200; 300) may include an operation (790) of performing vision recognition using the specified vision engine based on the visual data obtained by converting the at least one component.

[0278] In an operating method of an electronic device (101; 200; 300) according to one embodiment, the visual data may be data related to user hand detection, hand gesture recognition, facial recognition, facial expression recognition, or map generation.

[0279] The method of operating an electronic device (101; 200; 300) according to one embodiment may further include an operation (730) of performing an initial evaluation to identify a characteristic of the first vision sensor (430; 176; 180) or the visual data. In the method of operating an electronic device (101; 200; 300) according to one embodiment, the operation (750) of converting the at least one component may identify or convert the at least one component based on a result of the initial evaluation.

[0280] In an operating method of an electronic device (101; 200; 300) according to one embodiment, the operation (750) of transforming the at least one component may include an operation (753) of identifying the number or quality of the at least one component for the visual data, an operation (755) of evaluating the number or quality of the at least one component based on requirements of the designated vision engine, and an operation (757) of transforming the at least one component based on the evaluation result.

[0281] In an operating method of an electronic device (101; 200; 300) according to one embodiment, the operation (715) of converting the at least one component may repeat the conversion for the at least one component based on a parameter related to the quality of the vision recognition.

[0282] In an operating method of an electronic device (101; 200; 300) according to one embodiment, at least one component of the visual data may include a parameter related to a color of the visual data. In an operating method of an electronic device (101; 200; 300) according to one embodiment, an operation (750) of converting the at least one component may include an operation of adjusting a parameter related to the color.

[0283] In an operating method of an electronic device (101; 200; 300) according to one embodiment, the operation (750) of converting the at least one component may obtain, from a second vision sensor (450; 176; 180), comparison data corresponding to the visual data and in which at least some of the at least one component is different, and fuse the obtained comparison data with the visual data.

[0284] In an operating method of an electronic device (101; 200; 300) according to one embodiment, at least one component of the visual data may include a parameter related to a size corresponding to the visual data. In an operating method of an electronic device (101; 200; 300) according to one embodiment, an operation (750) of converting the at least one component may upscale the visual data based on the parameter related to the size.

[0285] In an operating method of an electronic device (101; 200; 300) according to one embodiment, the operation (750) of performing vision recognition using the designated vision engine may extract at least one feature from the visual data, and perform the vision recognition based on the at least one feature extracted through the designated vision engine.

[0286] In a storage medium storing computer-readable instructions according to one embodiment of the present disclosure, the instructions, when executed by at least one processor (410; 120) of an electronic device (101; 200; 300), may cause the electronic device (101; 200; 300) to obtain the visual data from the first vision sensor (430; 176; 180). The instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to transform at least one component of the visual data based on a requirement of a designated vision engine. The above instructions, when executed by the at least one processor (410, 120), may cause the electronic device (101; 200; 300) to perform vision recognition using the designated vision engine based on visual data that has transformed the at least one component.

[0287] Electronic devices according to the embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0288] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0289] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0290] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0291] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0292] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101; 200; 300), A first vision sensor (430; 176; 180) configured to acquire visual data; Memory for storing instructions (420; 130); and Containing at least one processor (410, 120), The above instructions, when executed by the at least one processor (410, 120), cause the electronic device (101; 200; 300) to: Obtaining the visual data from the first vision sensor (430; 176; 180), Based on the requirements of a given vision engine, transforming at least one component of said visual data, Causing vision recognition to be performed using the specified vision engine based on the visual data that has transformed at least one component. Electronic devices (101; 200; 300).

2. In paragraph 1, The above first vision sensor (430; 176; 180) causes the user to obtain the visual data related to the detection of the user's hand or the recognition of the hand gesture. Electronic devices (101; 200; 300).

3. In paragraph 1, The above first vision sensor (430; 176; 180) Set to acquire said visual data related to facial recognition, facial expression recognition or map generation, Electronic devices (101; 200; 300).

4. In any one of paragraphs 1 to 3, The above instructions cause the electronic device (101; 200; 300) to: Performing an initial evaluation to identify the characteristics of the first vision sensor (430; 176; 180) or the visual data, At least as part of the operation of transforming said at least one component, causing said at least one component to be identified or transformed based on the results of said initial evaluation, Electronic devices (101; 200; 300).

5. In any one of paragraphs 1 to 4, The above instructions cause the electronic device (101; 200; 300) to: At least as part of the operation of transforming at least one component, Identifying the number or quality of at least one component of said visual data, Based on the requirements of the above-mentioned vision engine, the number or quality of at least one component is evaluated, Based on the above evaluation results, causing at least one component to be converted, Electronic devices (101; 200; 300).

6. In any one of paragraphs 1 to 5, The above instructions cause the electronic device (101; 200; 300) to: At least as part of the operation of transforming at least one component, causing the transformation of at least one component to be repeated based on a parameter related to the quality of the vision recognition; Electronic devices (101; 200; 300).

7. In any one of paragraphs 1 to 6, At least one component of said visual data includes a parameter related to a color of said visual data, The above instructions cause the electronic device (101; 200; 300) to: causing at least as part of the operation of transforming at least one component to adjust a parameter related to said color; Electronic devices (101; 200; 300).

8. In any one of paragraphs 1 to 7, Further comprising a second vision sensor (450; 176; 180), The above instructions cause the electronic device (101; 200; 300) to: At least as part of the operation of transforming at least one component, From the second vision sensor (450; 176; 180), comparison data corresponding to the visual data and at least some of the at least one component are obtained, Causing the acquired comparison data to be fused with the visual data; Electronic devices (101; 200; 300).

9. In any one of paragraphs 1 to 8, At least one component of said visual data includes a parameter related to a size corresponding to said visual data, The above instructions cause the electronic device (101; 200; 300) to: At least as part of the operation of transforming at least one component, Based on the parameters related to the above size, causing the visual data to be upscaled, Electronic devices (101; 200; 300).

10. In any one of paragraphs 1 to 9, The above instructions cause the electronic device (101; 200; 300) to: At least as part of the operation of performing vision recognition using the above-mentioned designated vision engine, extracting at least one feature from the visual data, and causing the vision recognition to be performed based on the at least one feature extracted through the above-mentioned designated vision engine. Electronic devices (101; 200; 300).

11. In Article 10, The above instructions cause the electronic device (101; 200; 300) to: At least as a part of the operation of performing vision recognition using the above-mentioned designated vision engine, extracting a depth map and a color image from the visual data, and causing the vision recognition to be performed based on the depth map and the color image through the above-mentioned designated vision engine. Electronic devices (101; 200; 300).

12. In the operating method of an electronic device (101; 200; 300), An operation (710) of acquiring visual data from a first vision sensor (430; 176; 180); An operation (750) of transforming at least one component of said visual data based on the requirements of a given vision engine; and An operation (790) for performing vision recognition using the specified vision engine based on visual data that has converted at least one component. How it works.

13. In paragraph 12, The above visual data is data related to user hand detection, hand gesture recognition, face recognition, facial expression recognition or map generation. How it works.

14. In clauses 12 to 13, Further comprising an operation (730) of performing an initial evaluation to identify characteristics of the first vision sensor (430; 176; 180) or the visual data; The operation (750) of transforming at least one component identifies or transforms at least one component based on the result of the initial evaluation. How it works.

15. In a storage medium storing computer-readable instructions, the instructions, when executed by at least one processor (410; 120) of an electronic device (101; 200; 300), cause the electronic device (101; 200; 300) to: From a first vision sensor (430; 176; 180) set to acquire visual data, the visual data is acquired, Based on the requirements of a given vision engine, transforming at least one component of said visual data, Causing vision recognition to be performed using the specified vision engine based on the visual data that has transformed at least one component. Storage medium.

Images