ELECTRONIC DEVICES AND VISUAL INFORMATION PROCESSING DEVICES

Abstract

The invention relates to an electronic device and a device for processing visual information using an image sensor. In visual mode, pixel data can be acquired by selecting only the visual pixel that is pre-set for visual mode from among the pixels in the image sensor, and the pixel data of the visual pixel can be processed in the form of visual information.

Description

Device and method for processing vision information using an image sensor

[0001] The following embodiments relate to a technology for processing vision information using an image sensor.

[0002] Augmented reality (AR) refers to a computer graphics technique that synthesizes virtual objects or information into a real-world environment, making them appear as if they were in the original environment. AR is a display technology that superimposes virtual objects onto the real world viewed by the user. This technology can be applied to products such as head-mounted displays (HMDs) or AR glasses to provide diverse user experiences. Furthermore, eye tracking (ET) technology can be used to track the user's gaze based on light reflected from their eyes, enabling the user to interact with the AR experience provided through AR glasses.

[0003] AR glasses are composed of many cameras.

[0004] Some cameras included in AR glasses may be color cameras with the ability to obtain high-quality images using image sensors that may be referred to as HR (high resolution) or PV (photo video).

[0005] Some other cameras included in AR glasses may be GS (global shutter) cameras for eye tracking (ET) purposes, which are used to detect and track the pupils.

[0006] Another camera included in the AR glasses may be a global shutter (GS) camera that acquires vision information for head tracking, hand detection and tracking, gestures, and / or spatial recognition.

[0007] Because of the many cameras included in AR glasses, there are limitations to changing the design of AR glasses and they consume a lot of power.

[0008] As a related prior art, there is Korean Patent Publication No. 2013-0046174 (Title: Vision Recognition Device and Method). This publication discloses that some pixels among all pixels are designated as vision pixels.

[0009] According to one embodiment of the present disclosure, an electronic device includes an image sensor; and a processor, wherein the processor operates in a first mode for obtaining first vision information using pixels of a preset color among pixels included in the image sensor, and when an input corresponding to a mode change is received through the electronic device during the first mode operation, the processor operates in a second mode for obtaining image data using pixels of a plurality of colors included in the image sensor, and obtaining second vision information based at least in part on luminance information of the image data.

[0010] According to one embodiment of the present disclosure, a device for processing vision information includes: an image sensor; a first power circuit for supplying a first power to a vision pixel, which is a pixel preset for a vision mode among pixels included in the image sensor; a second power circuit for supplying a second power to pixels remaining in the image sensor except for the vision pixel; and a processor, wherein the processor controls, in the vision mode, to output pixel data from the vision pixel of the image sensor, receives the output pixel data and processes it into the vision information, and, in the vision mode, supplies the first power only to the vision pixel among the image sensor through the first power circuit.

[0011] In a computer-readable recording medium according to an embodiment of the present disclosure, in a vision mode, the method may include an operation of selecting only vision pixels, which are pixels preset for the vision mode, from among pixels included in the image sensor, and receiving pixel data, and an operation of processing pixel data of the vision pixels into vision information.

[0012] Figure 1 is a drawing illustrating an example of obtaining vision information from an image sensor.

[0013] FIG. 2 is a diagram illustrating an example of obtaining pixel data of a green pixel as vision information according to one embodiment.

[0014] FIG. 3 is a diagram schematically illustrating a configuration of an image processing device according to one embodiment.

[0015] FIG. 4 is a diagram illustrating a circuit configuration of an image sensor in an image processing device according to one embodiment.

[0016] FIG. 5 is a diagram illustrating an example of selecting a vision pixel in an image processing device according to one embodiment.

[0017] FIG. 6 is a diagram illustrating a configuration of a power circuit that supplies power to an image sensor in an image processing device according to one embodiment.

[0018] FIG. 7 is a diagram illustrating an example of an RGBW type image sensor that can be included in an image processing device according to one embodiment.

[0019] FIG. 8 is a flowchart illustrating an operation of an image processing device according to one embodiment when the device operates in vision mode.

[0020] FIG. 9 is a flowchart illustrating an operation of an image processing device according to one embodiment when the device operates in capture mode.

[0021] FIG. 10 is a flowchart illustrating an operation of an image processing device according to one embodiment when performing vision mode and video mode simultaneously.

[0022] FIG. 11 is a flowchart illustrating an operation of an image processing device according to one embodiment when switching from a first mode to a second mode.

[0023] FIG. 12 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present invention to perform a first vision mode and a second vision mode in a first mode.

[0024] FIG. 13 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present invention to simultaneously perform a shooting mode and a vision mode in a second mode.

[0025] FIG. 14 is a diagram illustrating examples of pixel data acquired when an image processing device according to one embodiment operates in a second mode, when operating in a high-resolution vision mode and when operating in a low-resolution vision mode.

[0026] FIG. 15 is a diagram illustrating an example in which a vision pixel is implemented with two photodiodes in an image sensor of an image processing device according to one embodiment.

[0027] FIG. 16 is a block diagram of an electronic device within a network environment according to one embodiment.

[0028] FIG. 17 is a diagram illustrating the structure of an electronic device implemented in the form of wearable augmented reality glasses according to one embodiment.

[0029] FIG. 18 is a drawing showing the front of a wearable electronic device according to one embodiment.

[0030] FIG. 19 is a drawing showing the back of a wearable electronic device according to one embodiment.

[0031] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be modified in various ways, and the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, or alternatives to the embodiments are included within the scope of the patent application.

[0032] The terms used in the examples are for the purpose of description only and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, it should be understood that terms such as "comprise" or "have" specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0033] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiments pertain. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0034] In addition, when describing with reference to the attached drawings, identical components will be assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted. When describing embodiments, if a detailed description of a related known technology is judged to unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

[0035] Additionally, terms such as first, second, A, B, (a), or (b) may be used to describe components of the embodiments. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms. When a component is described as being "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be "connected," "coupled," or "connected" between each component.

[0036] Components included in one embodiment and components with common functions will be described using the same names in other embodiments. Unless otherwise stated, the descriptions given in one embodiment may also apply to other embodiments, and detailed descriptions will be omitted to the extent of overlap.

[0037] Hereinafter, a device and method for processing vision information using an image sensor according to an embodiment of the present disclosure will be described in detail with reference to the attached FIGS. 1 to 19.

[0038] For head tracking, hand detection and tracking, gesture and / or spatial recognition, vision information can typically be acquired using pixel data acquired through a global shutter camera.

[0039] However, vision information can be obtained using pixel data acquired through an image sensor such as an RGB (red green blue) sensor or an RGBW (red green blue white) sensor corresponding to a high-resolution RS (rolling shutter) camera.

[0040] Vision information may include information about the position and size of an object, the classification of an object, the shape and boundaries of an object, the motion of an object, properties of an image (e.g., properties such as color, contrast, or texture), and / or quality of an image (e.g., resolution, noise, or distortion of an image).

[0041] An example of obtaining vision information using an image sensor is described below through Figure 1.

[0042] Figure 1 is a drawing illustrating an example of obtaining vision information from an image sensor.

[0043] Referring to FIG. 1, the image sensor (100) is an RGB sensor composed of red pixels, green pixels, and blue pixels.

[0044] The vision information acquisition method can acquire pixel data (112) of a red pixel, pixel data (114) of a green pixel, and pixel data (116) of a blue pixel output from an image sensor (100).

[0045] And, the vision information acquisition method can generate demosaiced pixel data (130) composed of demosaiced red pixel data (122), demosaiced green pixel data (124), and demosaiced blue pixel data (126) by demosaicing pixel data (112) of red pixels, pixel data (114) of green pixels, and pixel data (116) of blue pixels. At this time, demosaicing is a digital image processing algorithm used to reconstruct a full color image from incomplete color samples output from an image sensor in which a color filter array (CFA) is overlapped.

[0046] And, the vision information acquisition method can convert demosaiced pixel data (130) in RGB form into a YCrCb image (140) in YCrCb form, and acquire luminance information (142) from the YCrCb image (140) as vision information.

[0047] In order to obtain vision information through the general process of the image sensor (100) as shown in Fig. 1, a lot of processing of the processor is required, and therefore, a lot of power consumption occurs.

[0048] FIG. 2 is a diagram illustrating an example of obtaining pixel data of a green pixel as vision information according to one embodiment.

[0049] Referring to FIG. 2, a method for obtaining vision information can obtain pixel data (114) of a green pixel output from an image sensor (100) as vision information (210).

[0050] The reason why the pixel data (114) of the green pixel can be treated as vision information (210) is that, as a result of the verification, the pixel data (114) of the green pixel and the spectral characteristics of the green pixel are very similar to the spectral characteristics of the luminance information, so that even if the pixel data (114) of the green pixel is treated as vision information (210), it can be used as vision information for head tracking, hand detection and tracking, gesture, and / or space recognition.

[0051] In the present disclosure, by treating pixel data (114) of a green pixel as vision information (210), when acquiring vision information through an image sensor (100), the necessary demosaicing operation and the operation of converting to a YCrCb image (140) can be omitted, thereby reducing power and time consumed by computing power.

[0052] FIG. 3 is a diagram schematically illustrating a configuration of an image processing device according to one embodiment.

[0053] Referring to FIG. 3, the image processing device (300) of the present disclosure may include an image sensor (310) and a processor (320).

[0054] The image sensor (310) may be an RGB sensor that divides the image into red, green, and blue.

[0055] The processor (320) can determine whether the current mode is a vision mode. At this time, the current mode may include a capture mode, a video mode, and a vision mode. The capture mode is a mode for capturing images, the video mode is a mode for recording videos, and the vision mode is a mode for acquiring vision information for head tracking, hand detection and tracking, gestures, and / or spatial recognition.

[0056] If the current mode is the vision mode, the processor (320) can select only the vision pixels, which are pixels preset for the vision mode among the pixels included in the image sensor (310), receive pixel data, and process the pixel data of the vision pixels into vision information.

[0057] A vision pixel is a pixel set for vision mode among the pixels included in the image sensor. If the image sensor is an RGB sensor, it may be all green pixels included in the image sensor or a part of the green pixels included in the image sensor.

[0058] Meanwhile, in FIG. 3, the image sensor (310) and the processor (320) are configured as separate entities, but the image sensor (310) and the processor (320) may also be implemented as a single entity.

[0059]

[0060] FIG. 4 is a diagram illustrating a circuit configuration of an image sensor in an image processing device according to one embodiment.

[0061] Referring to FIG. 4, the image sensor (310) can be configured as an image sensor circuit (410), and can acquire pixel data by selecting a specific pixel through a switch. For example, the processor (320) can receive only pixel data of green pixels corresponding to vision pixels by controlling only the first column and the third column corresponding to green pixels to be output when the first row of the image sensor (310) is selected, and controlling only the second column and the fourth column to be output when the second row of the image sensor (310) is selected.

[0062] FIG. 5 is a diagram illustrating an example of selecting a vision pixel in an image processing device according to one embodiment.

[0063] Referring to Figure 5, the resolution required for vision information is much lower than the resolution of the image output from the image sensor.

[0064] Accordingly, the processor (320) may operate by selecting all green pixels as vision pixels, as in the example of 510, but may also select only some of the green pixels as vision pixels, as in the example of 520.

[0065] Returning to the description of FIG. 3, if the current mode is a capture mode, the processor (320) may receive pixel data from all pixels of the image sensor, perform demosaicing for each color of the filter included in the image sensor (310) to generate a demosaiced image, convert the demosaiced image into a YCbCr image in the YCbCr format, and process the YCbCr image as capture information. At this time, if the image sensor (310) is an RGB sensor, an RGB image, which is a demosaiced image composed of a red image, a green image, and a blue image, may be generated through demosaicing.

[0066] When the current mode performs vision mode and video mode simultaneously, the processor (320) can demosaic all pixel data received from the image sensor (310) to generate a demosaiced image, convert the demosaiced image into a YCbCr format to generate a YCbCr image, process the YCbCr image as video information in the video mode, and process luminance information of the YCbCr image as vision information in the vision mode.

[0067] Meanwhile, when the processor (320) processes a YCbCr image as video information in video mode, it can convert the YCbCr image to a size required in the video mode and process it as video information.

[0068] When the processor (320) processes the luminance information of the YCbCr image into vision information in the vision mode, it can convert the luminance information of the YCbCr image into a size required by the vision mode and process it into vision information.

[0069] Meanwhile, the image sensor (310) can be powered through different power circuits for the vision pixels and the remaining pixels, as shown in FIG. 6 below.

[0070] FIG. 6 is a diagram illustrating a configuration of a power circuit that supplies power to an image sensor in an image processing device according to one embodiment.

[0071] Referring to FIG. 6, the image sensor (310) can be supplied with power through a first power circuit (610) and a second power circuit (620). The first power circuit (610) is a circuit that supplies power to the vision pixel.

[0072] The second power circuit (620) is a circuit that supplies power to the pixels of the image sensor (310) except for the vision pixels. At this time, the power supplied to the first power circuit (610) and the second power circuit (620) may be the same power, but may also be supplied through different power sources. That is, the first power circuit (610) may supply the first power source, and the second power circuit (620) may supply the second power source.

[0073] When the processor (320) operates only in vision mode, it can reduce the power consumption of the image processing device (300) by supplying power only to the vision pixels among the image sensors (310) through the first power circuit (610) and not supplying power through the second power circuit (620).

[0074] The processor (320) can supply power to all pixels of the image sensor (310) through the first power circuit (610) and the second power circuit (620) when the image sensor (310) operates in capture mode or video mode.

[0075] Meanwhile, the image sensor (310) of the image processing device (300) may be an RGBW image sensor instead of an RGB image sensor.

[0076] FIG. 7 is a diagram illustrating an example of an RGBW type image sensor that can be included in an image processing device according to one embodiment.

[0077] Referring to Fig. 7, an RGBW type image sensor may be configured with red, green, blue, and white pixels in various patterns (710, 720, 730, 740). If the image sensor is an RGBW sensor, the vision pixel may be a white pixel or a portion of the white pixels included in the image sensor.

[0078] The white pixel of an RGBW image sensor is a pixel that does not contain a color filter and can be the ideal vision information closest to the luminance information.

[0079]

[0080] Hereinafter, the method according to the present disclosure configured as above will be described with reference to the drawings below.

[0081] FIG. 8 is a flowchart illustrating an operation of an image processing device according to one embodiment when the device operates in vision mode.

[0082] Referring to FIG. 8, in operation 810, an image processing device (e.g., the image processing device (300) of FIG. 3) according to an embodiment may determine whether a current mode is a vision mode. The image processing device may include a capture mode, a video mode, and a vision mode. The capture mode is a mode for capturing images, the video mode is a mode for recording videos, and the vision mode is a mode for acquiring vision information for head tracking, hand detection and tracking, gestures, and / or spatial recognition.

[0083] If the current mode is not the vision mode as a result of the confirmation of operation 810, for example, if it is the capture mode or the video mode, the image processing device may perform an operation corresponding to the capture mode or an operation corresponding to the video mode. If the current mode is the vision mode as a result of the confirmation of operation 810, in operation 820, the image processing device according to an embodiment may supply power only to the vision pixels among the image sensors through the first power circuit (e.g., the first power circuit (610) of FIG. 6) that supplies power to the vision pixels of the image sensor. Operation 820 may be omitted. That is, the image processing device may supply power to all pixels of the image sensor without distinguishing between vision pixels. Here, the vision pixels are pixels preset for the vision mode among the pixels included in the image sensor. If the image sensor is an RGB sensor having red, green, and blue filters, the vision pixels may be all green pixels included in the image sensor or some green pixels included in the image sensor. Additionally, if the image sensor is an RGBW sensor with red, green, blue, and white filters, the vision pixels may be all white pixels contained in the image sensor or some of the white pixels contained in the image sensor.

[0084] In operation 830, an image processing device according to one embodiment can receive pixel data by selecting only vision pixels from an image sensor.

[0085] In operation 840, an image processing device according to an embodiment can process pixel data of a vision pixel into vision information.

[0086] FIG. 9 is a flowchart illustrating an operation of an image processing device according to one embodiment when the device operates in capture mode.

[0087] Referring to FIG. 9, in operation 910, an image processing device according to an embodiment (e.g., the image processing device (300) of FIG. 3) can check whether the current mode is a capture mode.

[0088] If the current mode is not the capture mode as a result of the confirmation of operation 910, for example, if it is the vision mode or the video mode, the image processing device can perform an operation corresponding to the vision mode (e.g., the operation of FIG. 8) or an operation corresponding to the video mode. If the current mode is the capture mode as a result of the confirmation of operation 910, in operation 920, the image processing device can supply power to all pixels of the image sensor through a first power circuit that supplies power to the vision pixels of the image sensor and a second power circuit that supplies power to the remaining pixels of the image sensor that are not the vision pixels.

[0089] In operation 930, the image processing device according to one embodiment can receive pixel data from all pixels of the image sensor.

[0090] In operation 940, an image processing device according to an embodiment can generate a demosaiced image by performing demosaicing according to the color of a filter included in an image sensor. If the image sensor is an RGB sensor, an RGB image, which is a demosaiced image composed of a red image, a green image, and a blue image, can be generated through demosaicing. In addition, if the image sensor is an RGBW sensor, an RGBW image, which is a demosaiced image composed of a red image, a green image, a blue image, and a white image, can be generated through demosaicing.

[0091] In operation 950, the image processing device according to one embodiment can convert a demosaiced image into a YCbCr image of the YCbCr format.

[0092] In operation 960, an image processing device according to an embodiment can store a YCbCr image.

[0093] In operation 970, an image processing device according to an embodiment can process a YCbCr image as capture information.

[0094] FIG. 10 is a flowchart illustrating an operation of an image processing device according to one embodiment when performing vision mode and video mode simultaneously.

[0095] Referring to FIG. 10, in operation 1010, an image processing device according to an embodiment (e.g., the image processing device (300) of FIG. 3) can check whether the current mode performs the vision mode and the video mode simultaneously.

[0096] If the current mode does not perform vision mode and video mode simultaneously as a result of the verification of operation 1010, the image processing device according to an embodiment may perform an operation corresponding to the current mode. If the current mode does perform vision mode and video mode simultaneously as a result of the verification of operation 1010, in operation 1020, the image processing device according to an embodiment may supply power to all pixels of the image sensor through the first power circuit and the second power circuit.

[0097] In operation 1030, the image processing device according to one embodiment can receive pixel data from all pixels of the image sensor.

[0098] In operation 1040, an image processing device according to an embodiment can perform demosaicing for each color of a filter included in an image sensor to generate a demosaiced image.

[0099] In operation 1050, the image processing device according to one embodiment can convert a demosaiced image into a YCbCr image of the YCbCr format.

[0100] In operation 1060, an image processing device according to one embodiment can convert a YCbCr image to a preset size required in a video mode.

[0101] In one embodiment of the invention, an image processing device can store a converted YCbCr image.

[0102] In operation 1064, a converted YCbCr image according to one embodiment can be processed as video information.

[0103] At this time, the 1060 operation can be omitted, that is, the image processing device can store the YCbCr image in the 1062 operation without converting the size of the YCbCr image, and process the YCbCr image as video information in the 1064 operation.

[0104] An image processing device according to one embodiment can convert a YCbCr image to a preset size required in a vision mode in a 1070 operation, separately from a 1060 operation.

[0105] In operation 1072, the image processing device according to one embodiment can extract luminance information from a YCbCr image of a preset size. In operation 1074, the image processing device according to one embodiment can process the converted luminance information as vision information.

[0106] At this time, operation 1070 can be omitted, that is, the image processing device can extract luminance information from the YCbCr image that has not been resized in operation 1072, and process the extracted luminance in operation 1074 as vision information. Meanwhile, in the description of Fig. 10, operation 1072 is performed after operation 1070, but it can also be configured to perform operation 1070 after operation 1072.

[0107] According to various embodiments, if the current mode is determined to perform vision mode and video mode simultaneously, operations 820 and below of FIG. 8 related to vision processing may be performed in combination with operations related to video information processing in the video mode of FIG. 10. In this scenario, operations related to video information processing and operations related to the vision mode may be performed simultaneously and in parallel. The operations related to the video information processing may be part of the operations described in FIG. 10 (1020, 1030, 1040, 1050, 1060, 1062, 1064). The operations related to the vision mode may be part of the operations described in FIG. 8 (820, 830, 840). That is, the vision mode and the video mode may be processed through separate paths. For example, the image processing device may transmit only pixel data to be used for vision from the pixel data of the image sensor to the path that processes the vision mode through the MIPI (mobile industry processor interface) interface.

[0108] FIG. 11 is a flowchart illustrating an operation of an image processing device according to one embodiment when switching from a first mode to a second mode.

[0109] Referring to FIG. 11, in operation 1110, an image processing device according to an embodiment (e.g., the image processing device (300) of FIG. 3) may determine whether the current mode is a first mode for processing vision information. In operation 1110, if the current mode is the first mode for processing vision information, in operation 1120, the image processing device according to an embodiment (e.g., the image processing device (300) of FIG. 3) may obtain first vision information by using pixels of a preset color (e.g., green or white) among pixels included in an image sensor (e.g., the image sensor (310) of FIG. 3). A specific description of operation 1120 will be described later with reference to FIG. 12.

[0110] In operation 1130, an image processing device according to an embodiment may determine whether an input corresponding to a mode change is received during a first mode operation. At this time, the input corresponding to the mode change may be an input that takes into account at least one of a change amount of an object recognized in the first mode, a type of an object recognized, or a user's image capturing input. More specifically, the input corresponding to the mode change may be a case where the change amount of an object recognized in the first mode exceeds a preset reference value, a case where the type of an object recognized is included in a preset object, or a case where a program related to capturing is executed to start capturing an image of the user.

[0111] If no input corresponding to mode switching is received as a result of the confirmation of operation 1130, the image processing device according to one embodiment returns to operation 1120.

[0112] If an input corresponding to a mode change is received as a result of the confirmation of operation 1130, the image processing device according to an embodiment may change the current mode from the first mode to the second mode in operation 1140. At this time, the second mode may be a mode in which a vision mode for collecting vision information and a shooting mode for taking videos or photos are performed simultaneously. In other words, the second mode may include a vision mode and a shooting mode.

[0113] In operation 1150, an image processing device according to an embodiment may acquire image data using pixels of multiple colors included in an image sensor, and acquire second vision information based at least in part on luminance information of the image data. A detailed description of operation 1150 will be described later with reference to FIG. 13.

[0114]

[0115] FIG. 12 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present invention to perform a first vision mode and a second vision mode in a first mode.

[0116] Referring to FIG. 12, in operation 1210, an image processing device according to an embodiment (e.g., the image processing device (300) of FIG. 3) can check whether the first condition is met.

[0117] At this time, the first condition is a condition for determining whether to use the first vision mode. The first vision mode can operate only with vision pixels having low resolution. The first vision mode can operate when the electronic device does not require much vision information. For example, when a user is wearing the electronic device and sitting still while watching a video, the electronic device can recognize that the user does not require much vision information by using information coming from the acceleration sensor and the image sensor, and can operate in the first vision mode. In addition, the first vision mode can operate when the electronic device is in a sleep mode or power saving mode to reduce battery consumption. Therefore, the first condition can be at least one of the following: when a light shield (a light shielding member that can be separately attached to the front of an electronic device such as an augmented reality (AR) device) is mounted, when the user is not wearing the electronic device, or when the amount of light coming into the image sensor in a bright environment is greater than a preset reference value, allowing operation at a high frame rate.

[0118] If the verification result of operation 1210 satisfies the first condition, in operation 1220, the image processing device according to one embodiment can set the current mode to the first vision mode.

[0119] In operation 1230, an image processing device according to an embodiment may receive pixel data of a first group of pixels from an image sensor. At this time, the first group of pixels may have one color (green or white) of the image sensor and may represent a group of pixels corresponding to a first vision mode.

[0120] In one embodiment of the 1240 operation, the image processing device can process vision information using pixel data of the first group of pixels.

[0121] If the verification result of operation 1210 does not meet the first condition, or if vision information is processed in operation 1240, the image processing device according to an embodiment can check whether the second condition is met in operation 1250. In this case, the second condition may be a condition for determining whether high-resolution vision information is required.

[0122] For example, the second condition may be when the acceleration sensor detects acceleration exceeding a preset threshold value, or when the vision information obtained in the first vision mode is used to detect movement of the user exceeding a preset threshold value, since more vision information about the surrounding environment is required when the user is sitting still and then moving.

[0123] For example, a second condition might be that the type of information displayed on the electronic device is a pre-defined, pre-classified type. This is because, due to the nature of augmented reality devices, where the user's field of view and the content provided share a common field of view, providing a large amount of information (e.g., bright information, high-frame rate information, colorful information, or video) would obscure the user's field of view, requiring more peripheral vision information.

[0124] For example, the second condition may be when an acceleration sensor or a proximity sensor is used to detect that a user is wearing an image processing device, when a microphone in an electronic device including the image processing device or an external device (e.g., a Bluetooth peripheral (buds, watch), or a smartphone) functionally connected to the electronic device detects a specific sound or a sound above a threshold (e.g., a scream, a honking sound, or a user's designated voice), when a message, a phone call, or a system notification is received, or when a heart rate sensor attached to an external device functionally connected to the electronic device is used to detect a user's heart rate and the user's heart rate rises or falls above a certain threshold.

[0125] For example, the second condition may be when there is no significant change in vision information in the first vision mode, and then a change greater than a certain amount is detected (e.g., detection of a specific object, change in brightness, movement of an object greater than a certain amount, recognition of a user's hand, or recognition of another user's face).

[0126] If the second condition is met as a result of the verification of operation 1250, the image processing device according to an embodiment can set the current mode to the second vision mode in operation 1260.

[0127] In operation 1270, an image processing device according to an embodiment may receive pixel data of a second group of pixels from an image sensor. At this time, the second group of pixels may have one color (green or white) of the image sensor and may represent a group of pixels corresponding to a second vision mode.

[0128] In one embodiment of the 1280 operation, the image processing device can process vision information using pixel data of the second group pixels.

[0129] If the result of the verification of operation 1250 does not meet the second condition, or if the vision information was processed in operation 1280, the image processing device according to one embodiment may proceed to operation 1130 of FIG. 11.

[0130] In FIG. 12, the first mode includes a first vision mode for low resolution and a second vision mode for high resolution, and the number of pixels in the first group of pixels is smaller than the number of pixels in the second group of pixels, and the first group of pixels can be included in the second group of pixels. In addition, the first vision mode and the second vision mode can have different frames per second, applied gain, sensitivity, or applied noise reduction levels. In addition, the first group of pixels in the first vision mode can be the vision pixels indicated in the example 520 in FIG. 5, and the second group of pixels in the second vision mode can be the vision pixels indicated in the example 510 in FIG. 5.

[0131]

[0132] FIG. 13 is a flowchart illustrating an operation of an image processing device according to one embodiment of the present invention to simultaneously perform a shooting mode and a vision mode in a second mode.

[0133] Referring to FIG. 13, in operation 1310, an image processing device according to an embodiment (e.g., the image processing device (300) of FIG. 3) can determine whether the shooting mode included in the second mode is a video mode for shooting a video or a capture mode for shooting a photo.

[0134] If the current shooting mode is a video mode as a result of the confirmation of operation 1310, in operation 1320, the image processing device according to one embodiment may receive pixel data output from the image sensor, and output image data (e.g., an image of video resolution) of a preset size required by the video mode using the received pixel data. More specifically, the image processing device may perform demosaicing according to the color of the filter included in the image sensor to generate a demosaiced image, and may convert the generated image data using the demosaiced image into a preset size required by the video mode.

[0135] In operation 1322, the image processing device according to one embodiment can convert the image data output in operation 1320 into a YUV (or YCbCr) image in YUV (or YCbCr) format.

[0136] In operation 1324, an image processing device according to an embodiment may provide a YUV (or YCbCr) image as an image required in a video mode, and process vision information using luminance information of the YUV (or YCbCr) image. At this time, the image processing device may check a resolution required for vision information processing, change the size of the YUV (or YCbCr) image, and process vision information using luminance information of the changed YUV (or YCbCr) image.

[0137] If the current shooting mode is a capture mode rather than a video mode as a result of the confirmation of operation 1310, in operation 1330, the image processing device according to one embodiment may receive pixel data output from the image sensor and output image data (e.g., an image of photographic resolution) of a preset size required by the capture mode using the received pixel data. More specifically, the image processing device may perform demosaicing according to the color of the filter included in the image sensor to generate a demosaiced image, and may convert the generated image data using the demosaiced image into a preset size required by the capture mode.

[0138] In operation 1332, the image processing device according to one embodiment can convert the image data output in operation 1330 into a YUV (or YCbCr) image in YUV (or YCbCr) format.

[0139] In operation 1334, an image processing device according to an embodiment may provide a YUV (or YCbCr) image as an image required in a capture mode, and process vision information using luminance information of the YUV (or YCbCr) image. At this time, the image processing device may check a resolution required for vision information processing, change the size of the YUV (or YCbCr) image, and process vision information using luminance information of the changed YUV (or YCbCr) image.

[0140] Meanwhile, in operation 1324 or operation 1334, when the image processing device processes vision information, it may display at least a portion of the image data on a display (a configuration of an electronic device including the image processing device).

[0141] In the case of the vision mode used in Fig. 13, it can be operated by dividing into a third vision mode that processes low-resolution vision information and a fourth vision mode that processes high-resolution vision information, as in Fig. 12. At this time, the conditions for operating in the third vision mode can refer to the first condition of Fig. 12, and the conditions for operating in the fourth vision mode can refer to the second condition of Fig. 12. In addition, the third vision mode and the fourth vision mode can differ in the number of frames per second, applied gain, sensitivity, or the degree of applied noise reduction.

[0142]

[0143] FIG. 14 is a diagram illustrating examples of pixel data acquired when an image processing device according to one embodiment operates in a second mode, when operating in a high-resolution vision mode and when operating in a low-resolution vision mode.

[0144] Referring to FIG. 14, an image processing device according to an embodiment (e.g., the image processing device (300) of FIG. 3) can receive pixel data for each color of a filter included in the image sensor from an image sensor, as in the example of 1410.

[0145] The image processing device can convert pixel data of 1410 into image data in YUV format, as in the example of 1420.

[0146] And, in the case of the third vision mode that processes low-resolution vision information, the image processing device can process vision information by using some luminance information from the image data of 1420, as in the example of 1430.

[0147] Additionally, in the case of the fourth vision mode that processes high-resolution vision information, the image processing device can process vision information using all luminance information from image data of 1420, such as in the example of 1440.

[0148]

[0149] FIG. 15 is a diagram illustrating an example in which a vision pixel is implemented with two photodiodes in an image sensor of an image processing device according to one embodiment.

[0150] Referring to FIG. 15, an electronic device including an image processing device (e.g., the image processing device (300) of FIG. 3) according to one embodiment may implement phase difference pixels in an image sensor for more accurate implementation of 6 DOF (degrees of freedom).

[0151] Phase difference pixels can utilize 2PD technology, which implements each pixel with two photodiodes, or can utilize on-plane phase difference technology, which utilizes one of the two pixels to cover part of the left side and the other pixel to cover part of the right side as a single phase difference group.

[0152] When implementing phase-detection pixels together with the pixels of an image sensor, since each pixel contains two or more photodiodes, there may be a loss of effective pixel space depending on the arrangement. Therefore, phase-detection pixels can be implemented in at least a portion of the G pixels of a Bayer pattern or the W pixels of an RGBW pattern, which have relatively high sensitivity.

[0153]

[0154] As in the example of FIG. 15, phase difference pixels can be implemented by utilizing 2PD technology in at least some of the pixels utilized in the first vision mode and the second vision mode of the present disclosure.

[0155] Phase difference pixels can also obtain phase difference information using fewer pixels in the first vision mode than in the second vision mode. The phase difference information is processed separately from the image processing information and can be utilized together in the vision mode. When switching between each mode, the phase difference information can be referenced to determine the movement of an object or the distance from an electronic device, and this can be used to switch the vision mode.

[0156] Additionally, the phase difference pixels may be located only in pixels utilized in the first vision mode, as in the example of 1520, or in pixels utilized in the second vision mode, as in the example of 1510. In the examples of 1510 and 1520, the pixels depicted as containing two boxes among the green pixels indicated by G are phase difference pixels.

[0157] For example, since the first vision mode is a low-power mode, additional power consumption is required to calculate phase difference information, and thus, phase difference pixels may not be located in pixels applied to the first vision mode.

[0158] At this time, phase difference pixels may be located in pixels utilized only in the second vision mode. When switching from the first vision mode to the second vision mode, phase difference information in at least a portion of the pixels utilized in the second vision mode can be utilized to obtain more information than in the first vision mode.

[0159]

[0160] Meanwhile, the image processing device (300) of FIG. 3 may be configured in the form of an electronic device (1601) in a network environment as in FIG. 16 below, or may be configured in the form of wearable augmented reality glasses (1700) or a wearable electronic device (1800) as in FIGS. 17 to 19.

[0161] FIG. 16 is a block diagram of an electronic device (1601) within a network environment (1600) according to one embodiment.

[0162] Referring to FIG. 16, in a network environment (1600), an electronic device (1601) (e.g., the image processing device (300) of FIG. 3) may communicate with an electronic device (1602) via a first network (1698) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1604) or a server (1608) via a second network (1699) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1601) may communicate with the electronic device (1604) via the server (1608). According to one embodiment, the electronic device (1601) may include a processor (1620), a memory (1630), an input module (1650), an audio output module (1655), a display module (1660), an audio module (1670), a sensor module (1676), an interface (1677), a connection terminal (1678), a haptic module (1679), a power management module (1688), a battery (1689), a communication module (1690), a subscriber identification module (1696), or an antenna module (1697). In some embodiments, the electronic device (1601) may omit at least one of these components (e.g., the connection terminal (1678)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (1676), the camera module (1680), or the antenna module (1697)) may be integrated into one component (e.g., the display module (1660)). At this time, the camera module (1680) can perform the role of the image sensor (310) of FIG. 3.

[0163] The processor (1620) may control at least one other component (e.g., a hardware or software component) of the electronic device (1601) connected to the processor (1620) by executing, for example, software (e.g., a program (1640)), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1620) may store commands or data received from other components (e.g., a sensor module (1676) or a communication module (1690)) in a volatile memory (1632), process the commands or data stored in the volatile memory (1632), and store result data in a non-volatile memory (1634). According to one embodiment, the processor (1620) may include a main processor (1621) (e.g., a central processing unit or an application processor) or a secondary processor (1623) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1621). For example, when the electronic device (1601) includes the main processor (1621) and the secondary processor (1623), the secondary processor (1623) may be configured to use less power than the main processor (1621) or to be specialized for a given function. The secondary processor (1623) may be implemented separately from the main processor (1621) or as a part thereof.

[0164] The auxiliary processor (1623) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1660), the sensor module (1676), or the communication module (1690)) of the electronic device (1601), for example, on behalf of the main processor (1621) while the main processor (1621) is in an inactive (e.g., sleep) state, or together with the main processor (1621) while the main processor (1621) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1623) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1680) or a communication module (1690)). In one embodiment, the auxiliary processor (1623) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1601) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1608)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0165] Meanwhile, the processor (1620) can perform the operation of the processor (320) of FIG. 3.

[0166] The memory (1630) can store various data used by at least one component (e.g., the processor (1620) or the sensor module (1676)) of the electronic device (1601). The data can include, for example, software (e.g., the program (1640)) and input data or output data for commands related thereto. The memory (1630) can include volatile memory (1632) or non-volatile memory (1634).

[0167] The program (1640) may be stored as software in memory (1630) and may include, for example, an operating system (1642), middleware (1644), or an application (1646).

[0168] The input module (1650) can receive commands or data to be used in a component of the electronic device (1601) (e.g., a processor (1620)) from an external source (e.g., a user) of the electronic device (1601). The input module (1650) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0169] The audio output module (1655) can output audio signals to the outside of the electronic device (1601). The audio output module (1655) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0170] The display module (1660) can visually provide information to an external party (e.g., a user) of the electronic device (1601). The display module (1660) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (1660) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0171] The audio module (1670) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1670) can acquire sound through the input module (1650), output sound through the sound output module (1655), or an external electronic device (e.g., electronic device (1602)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1601).

[0172] The sensor module (1676) can detect the operating status (e.g., power or temperature) of the electronic device (1601) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1676) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, a Hall sensor, or an illuminance sensor.

[0173] Additionally, the sensor module (1676) may further include a camera module capable of capturing still images and moving images. In this case, the camera module may include one or more lenses, image sensors, image signal processors, or flashes.

[0174] Meanwhile, the camera module can correspond to the image sensor (310) of FIG. 3.

[0175] The interface (1677) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1601) with an external electronic device (e.g., the electronic device (1602)). In one embodiment, the interface (1677) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0176] For example, the electronic device (1601) can transmit a video signal to an external electronic device through a connection terminal (1678). The electronic device (1601) can transmit a video signal for the external electronic device to output a video to a display module (1660) of the external electronic device.

[0177] The connection terminal (1678) may be for outputting a video signal or for outputting an audio signal. For example, the connection terminal (1678) may be for outputting a video signal and an audio signal simultaneously. For example, the electronic device (1601) may output a video signal and an audio signal to an external electronic device through an interface such as HDMI, DP, or Thunderbolt from the connection terminal (1678) that outputs a video signal and an audio signal simultaneously.

[0178] The connection terminal (1678) may include a connector through which the electronic device (1601) may be physically connected to an external electronic device (e.g., the electronic device (1602)). According to one embodiment, the connection terminal (1678) may include, for example, an HDMI connector, a DP connector, a Thunderbolt connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0179] The haptic module (1679) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1679) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0180] The power management module (1688) can manage power supplied to the electronic device (1601). According to one embodiment, the power management module (1688) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0181] A battery (1689) may power at least one component of the electronic device (1601). In one embodiment, the battery (1689) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0182] The communication module (1690) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1601) and an external electronic device (e.g., electronic device (1602), electronic device (1604), or server (1608)), and the performance of communication through the established communication channel. The communication module (1690) may operate independently from the processor (1620) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1690) may include a wireless communication module (1692) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1694) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1604) via a first network (1698) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1699) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1692) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1696) to identify or authenticate the electronic device (1601) within a communication network such as the first network (1698) or the second network (1699).

[0183] The wireless communication module (1692) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1692) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1692) can support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1692) can support various requirements specified in the electronic device (1601), an external electronic device (e.g., the electronic device (1604)), or a network system (e.g., the second network (1699)). According to one embodiment, the wireless communication module (1692) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0184] The antenna module (1697) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1697) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1697) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1698) or the second network (1699), may be selected from the plurality of antennas by, for example, the communication module (1690). A signal or power may be transmitted or received between the communication module (1690) and the external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1697).

[0185] According to various embodiments, the antenna module (1697) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0186] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0187] According to one embodiment, commands or data may be transmitted or received between the electronic device (1601) and an external electronic device (1604) via a server (1608) connected to a second network (1699). Each of the external electronic devices (1602 or 1604) may be the same or a different type of device as the electronic device (1601). According to one embodiment, all or part of the operations executed in the electronic device (1601) may be executed in one or more of the external electronic devices (1602, 1604, or 1608). For example, when the electronic device (1601) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1601) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1601). The electronic device (1601) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1601) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1604) may include an Internet of Things (IoT) device. The server (1608) may be an intelligent server utilizing machine learning and / or a neural network.In one embodiment, an external electronic device (1604) or server (1608) may be included within the second network (1699). The electronic device (1601) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0188] FIG. 17 is a diagram illustrating the structure of an electronic device implemented in the form of wearable augmented reality glasses according to one embodiment.

[0189] Referring to FIG. 17, an electronic device (1700) may be worn on a user's face and may provide the user with images related to an augmented reality service and / or a virtual reality service.

[0190] In one embodiment, the electronic device (1700) includes a first display (1705), a second display (1710), a first screen display unit (1715a), a second screen display unit (1715b), an input optical member (1720), a first transparent member (1725a), a second transparent member (1725b), a lighting unit (1730a, 1730b), a first PCB (1735a), a second PCB (1735b), a first hinge (1740a), a second hinge (1740b), a first camera (1745a, 1745b, 1745c, 1745d), a plurality of microphones (e.g., a first microphone (1750a), a second microphone (1750b), a third microphone (1750c)), a plurality of speakers (e.g., a first speaker (1755a), a second It may include a speaker (1755b)), a battery (1760), a second camera (1775a, 1775b), a third camera (1765), and a visor (1770a, 1770b).

[0191] In one embodiment, the displays (e.g., the first display (1705) and the second display (1710)) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown, if the displays are comprised of one of the liquid crystal display, the digital mirror display, or the silicon liquid crystal display, the electronic device (1700) may include a light source that irradiates light onto a screen output area of ​​the displays. In another embodiment, if the displays are capable of generating light on their own, for example, if they are comprised of one of the organic light emitting diodes or the micro LEDs, the electronic device (1700) may provide a good quality virtual image to the user even without including a separate light source. In one embodiment, if the display is implemented with an organic light-emitting diode or micro LED, a light source is unnecessary, so the electronic device (1700) can be made lighter. In the following, a display capable of generating light on its own is referred to as a self-luminous display, and the description is based on the assumption of a self-luminous display.

[0192] Displays according to various embodiments of the present disclosure (e.g., the first display (1705) and the second display (1710)) may be composed of at least one micro LED (micro light emitting diode). For example, the micro LED can express red (R, red), green (G, green), and blue (B, blue) by self-luminescence, and has a small size (e.g., 100 μm or less), so that one chip can implement one pixel (e.g., one of R, G, and B). Accordingly, when the display is composed of the micro LED, it can provide a high resolution without a backlight unit (BLU).

[0193] Not limited thereto, one pixel may include R, G, and B, and one chip may be implemented with multiple pixels including R, G, and B.

[0194] In one embodiment, the display (e.g., the first display (1705) and the second display (1710)) may be configured with a display area composed of pixels for displaying a virtual image and light-receiving pixels (e.g., photo sensor pixels) arranged between the pixels for receiving light reflected from the eye, converting it into electrical energy, and outputting it.

[0195] In one embodiment, the electronic device (1700) can detect a user's gaze direction (e.g., eye movement) through light-receiving pixels. For example, the electronic device (1700) can detect and track a gaze direction for the user's right eye and a gaze direction for the user's left eye through one or more light-receiving pixels constituting a first display (1705) and one or more light-receiving pixels constituting a second display (1710). The electronic device (1700) can determine the position of the center of the virtual image based on the gaze directions of the user's right and left eyes (e.g., the direction in which the pupils of the user's right and left eyes are gazing) detected through one or more light-receiving pixels.

[0196] In one embodiment, light emitted from displays (e.g., a first display (1705) and a second display (1710)) may pass through a lens (not shown) and a waveguide to reach a first screen display unit (1715a) formed on a first transparent member (1725a) positioned to face the user's right eye and a second screen display unit (1715b) formed on a second transparent member (1725b) positioned to face the user's left eye. For example, light emitted from displays (e.g., a first display (1705) and a second display (1710)) may pass through a waveguide to be reflected by a grating area formed on an input optical member (1720) and screen display units (1715a, 1715b) and transmitted to the user's eyes. The first transparent member (1725a) and / or the second transparent member (1725b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent.

[0197] In one embodiment, a lens (not shown) may be positioned in front of a display (e.g., a first display (1705) and a second display (1710)). The lens (not shown) may include a concave lens and / or a convex lens. For example, the lens (not shown) may include a projection lens or a collimation lens.

[0198] In one embodiment, the screen display (1715a, 1715b) or the transparent member (e.g., the first transparent member (1725a), the second transparent member (1725b)) may include a lens including a waveguide, a reflective lens.

[0199] In one embodiment, the waveguide may be made of glass, plastic, or polymer, and may include nano-patterns formed on one surface of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one end of the waveguide may be propagated within the display waveguide by the nano-patterns and provided to the user. In one embodiment, a waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The waveguide may include at least one diffractive element, for example, a diffractive optical element (DOE), a holographic optical element (HOE), or a reflective element, for example, a reflective mirror. In one embodiment, the waveguide may guide light emitted from the display (1705, 1710) to the user's eyes by using at least one diffractive element or reflective element included in the waveguide.

[0200] According to various embodiments, the diffractive element may include an input optical member (1720) / output optical member (not shown). For example, the input optical member (1720) may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a display (e.g., a first display (1705) and a second display (1710)) (e.g., a micro LED) to transmit the light to a transparent member (e.g., a first transparent member (1750a) and a second transparent member (1750b)) of a screen display unit (1715a, 1715b). The output grating region can act as an outlet to diffract (or reflect) light transmitted to the transparent member of the waveguide (e.g., the first transparent member (1750a), the second transparent member (1750b)) toward the user's eyes.

[0201] According to various embodiments, the reflective element may include a total internal reflection (TIR) ​​optical element or a total internal reflection waveguide. For example, total internal reflection may refer to a method of guiding light such that light (e.g., a virtual image) entering through an input grating region is 100% reflected from one surface (e.g., a specific surface) of the waveguide, thereby transmitting 100% of the light to the output grating region.

[0202] In one embodiment, light emitted from a display (1705, 1710) may be guided along an optical path through an input optical element (1720) into a waveguide. Light traveling within the waveguide may be guided toward a user's eyes through an output optical element. The screen display (1715a, 1715b) may be determined based on the light emitted toward the user's eyes.

[0203] In one embodiment, the first camera (1745a, 1745b, 1745c, 1745d) may include a camera used for 6 degrees of freedom (6DoF), 6DoF head tracking, hand detection and tracking, gesture and / or spatial recognition. For example, the first camera (1745a, 1745b, 1745c, 1745d) may include a global shutter (GS) camera to detect and track head and hand movements.

[0204] For example, the first camera (1745a, 1745b, 1745c, 1745d) may be a stereo camera for head tracking and spatial recognition, and cameras of the same specifications and performance may be used. The first camera (1745a, 1745b, 1745c, 1745d) may be a GS camera with excellent performance (e.g., image drag) for detecting and tracking fine movements such as rapid hand movements and fingers.

[0205] According to various embodiments, the first camera (1745a, 1745b, 1745c, 1745d) may be a rolling shutter (RS) camera. The first camera (1745a, 1745b, 1745c, 1745d) may perform SLAM functions through spatial recognition and depth shooting for 6 DoF. The first camera (1745a, 1745b, 1745c, 1745d) may perform a user gesture recognition function.

[0206] In one embodiment, the second cameras (1775a, 1775b) may be used to detect and track pupils. The second cameras (1775a, 1775b) may be referred to as ET (eye tracking) cameras. The second cameras (1775a, 1775b) may track the user's gaze direction. The electronic device (1700) may consider the user's gaze direction and position the center of the virtual image projected on the screen display unit (1715a, 1715b) according to the direction in which the user's pupils are looking.

[0207] The second camera (1775a, 1775b) for tracking the gaze direction may be a GS camera that can detect the pupil and track rapid eye movements. The second cameras (1775a, 1775b) may be installed for the left and right eyes, respectively, and the second cameras (1775a, 1775b) for the left and right eyes may be cameras with identical performance and specifications.

[0208] In one embodiment, the third camera (1765) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. The third camera (1765) may include a color camera equipped with functions for obtaining high-quality images, such as an auto focus (AF) function and an optical image stabilizer (OIS). However, the third camera (1765) may not be limited thereto, and may include a GS (global shutter) camera or an RS (rolling shutter) camera.

[0209] In one embodiment, at least one sensor (e.g., a gyroscope sensor, an acceleration sensor, a magnetometer sensor, a touch sensor, an ambient light sensor, and / or a gesture sensor), and a first camera (1745a, 1745b, 1745c, 1745d) can perform at least one of head tracking for 6DoF, pose estimation & prediction, gesture and / or spatial recognition, and slam function with depth shooting.

[0210] In another embodiment, the first camera (1745a, 1745b, 1745c, 1745d) may be used separately as a camera for head tracking and a camera for hand tracking.

[0211] In one embodiment, the lighting units (1730a, 1730b) may have different uses depending on the attachment location. For example, the lighting units (1730a, 1730b) may be attached together with the first camera (1745a, 1745b, 1745c, 1745d) mounted around a hinge connecting a frame and a temple (e.g., a first hinge (1740a), a second hinge (1740b)) or around a bridge connecting the frames. When shooting with a GS camera, the lighting units (1730a, 1730b) may be used as a means of supplementing the ambient brightness. For example, the lighting units (1730a, 1730b) may be used when it is difficult to detect a subject to be shot due to a dark environment or mixing or reflecting light from multiple light sources.

[0212] Meanwhile, the first camera (1745a, 1745b, 1745c, 1745d), the second camera (1775a, 1775b), and the third camera (1765) may be replaced in whole or in part with the image sensor (310) of FIG. 3.

[0213] Additionally, among the first camera (1745a, 1745b, 1745c, 1745d), the second camera (1775a, 1775b), and the third camera (1765), some may be replaced with the image sensor (310) of FIG. 3, and some may be excluded from the electronic device (1700). That is, the number of cameras included in the electronic device (1700) may be reduced by using the vision information processing method using the image sensor (310) of the present disclosure.

[0214] In one embodiment, a PCB (e.g., a first PCB (1735a), a second PCB (1735b)) may include a processor (not shown), a memory (not shown), and a communication module (not shown) that control components of the electronic device (1700).

[0215] Meanwhile, the PCB can process the operation of the processor (320) of FIG. 3.

[0216] A communication module (not shown) may support the establishment of a direct (e.g., wired) or wireless communication channel between the electronic device (1700) and an external electronic device, and the performance of communication through the established communication channel. The PCB may transmit electrical signals to components constituting the electronic device (1700).

[0217] A communication module (not shown) may include one or more communication processors that operate independently from the processor and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (not shown) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules (not shown) may communicate with an external electronic device via a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0218] The wireless communication module can support 5G networks and next-generation communication technologies beyond 4G networks, such as new radio access technology (NR). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module can support high-frequency bands (e.g., mmWave bands) to achieve high data rates, for example. The wireless communication module may support various technologies to secure performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna.

[0219] The electronic device (1700) may further include an antenna module (not shown). The antenna module may transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a first PCB (1735a), a second PCB (1735b)). In one embodiment, the antenna module may include a plurality of antennas (e.g., an array antenna).

[0220] In one embodiment, multiple microphones (e.g., a first microphone (1750a), a second microphone (1750b), and a third microphone (1750c)) can process external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function being performed (or the application being executed) by the electronic device (1700).

[0221] In one embodiment, multiple speakers (e.g., first speaker (1755a), second speaker (1755b)) can output audio data received from a communication module or stored in a memory.

[0222] In one embodiment, one or more batteries (1760) may be included and may supply power to components that make up the electronic device (1700).

[0223] In one embodiment, the visor (1770a, 1770b) can adjust the amount of external light transmitted into the user's eyes based on the transmittance. The visor (1770a, 1770b) can be positioned in front or behind the screen display unit (1715a, 1715b). The front of the screen display unit (1715a, 1715b) can mean the direction opposite to the user wearing the electronic device (1700), and the back can mean the direction toward the user wearing the electronic device (1700). The visor (1770a, 1770b) can protect the screen display unit (1715a, 1715b) and adjust the amount of external light transmitted.

[0224] For example, the visor (1770a, 1770b) may include an electrochromic element that changes color according to the applied power to control the transmittance. Electrochromism is a phenomenon in which the color changes due to an oxidation-reduction reaction caused by the applied power. The visor (1770a, 1770b) can control the transmittance of external light by utilizing the color change of the electrochromic element.

[0225] For example, the visor (1770a, 1770b) may include a control module and an electrochromic element. The control module may control the electrochromic element to adjust the transmittance of the electrochromic element.

[0226]

[0227] FIG. 18 and FIG. 19 are drawings showing the front and back of a wearable electronic device (1800) according to one embodiment.

[0228] When a user wears a wearable electronic device (1800), the appearance seen by the user's eyes may be as shown in FIG. 19.

[0229] Referring to FIG. 18, according to various embodiments, the electronic device (1601) of FIG. 16 may include a wearable electronic device (1800) that provides a service that provides an extended reality (XR) experience to a user. For example, XR or XR service may be defined as a service that collectively refers to virtual reality (VR), augmented reality (AR), and / or mixed reality (MR).

[0230] According to one embodiment, the wearable electronic device (1800) may have a form factor for being worn on a user's head. The wearable electronic device (1800) may refer to a head-mounted device or a head-mounted display worn on the user's head, but may also be configured in the form of at least one of glasses, goggles, a helmet, or a hat. The wearable electronic device (1800) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through glasses when worn, or a VST (video see-through) type configured to allow light emitted from a display to reach the user's eyes when worn, but block external light so that external light does not reach the user's eyes.

[0231] According to one embodiment, a wearable electronic device (1800) may be worn on a user's head and may provide the user with an image related to an extended reality (XR) service. For example, the wearable electronic device (1800) may provide XR content (hereinafter referred to as an XR content image) that outputs at least one virtual object to be superimposed on a display area or an area determined to be the user's field of view (FoV). According to one embodiment, XR content may mean an image or image that appears to have at least one virtual object superimposed on an image related to a real space acquired through a camera (e.g., a camera for filming). According to one embodiment, the wearable electronic device (1800) may provide XR content based on a function being performed by the wearable electronic device (1800) and / or a function being performed by one or more external electronic devices (e.g., electronic devices (1602, 1604, or 1608) of FIG. 16).

[0232] According to one embodiment, the wearable electronic device (1800) is at least partially controlled by an external electronic device (e.g., electronic devices (1602 or 1604) of FIG. 16), and may perform at least one function under the control of the external electronic device, but may also perform at least one function independently.

[0233] Referring to FIGS. 18 and 19 , a wearable electronic device (1800) may include a housing (1810) in which at least some of the components of FIG. 16 are arranged. The housing (1810) may be configured to be wearable on a user's head. For example, the housing (1810) may include a strap and / or a wearing member for being secured on a body part of the user. For example, the user may wear the wearable electronic device (1800) on the head so that the first direction (①) of the wearable electronic device (1800) is facing the user.

[0234] Referring to FIG. 19, a fourth function camera (e.g., a face recognition camera) (1825, 1826, 1827) and / or a display assembly (1900) may be disposed in a first direction (①) of the housing (1810) facing the user's face. Referring to FIG. 18, a first function camera (e.g., a recognition camera) (1815), a second function camera (e.g., a shooting camera) (1811, 1812), a depth sensor (1817), and / or a touch sensor (1813) may be disposed in a second direction (②) of the housing (1810) opposite to the first direction (①). Although not shown in the drawing, the housing (1810) may include memory (e.g., memory (1630) of FIG. 16) and a processor (e.g., processor (1620) of FIG. 16) within the housing, and may further include other components shown in FIG. 16.

[0235] In one embodiment, the display assembly (1900) may be positioned in the first direction (①) of the wearable electronic device (1800). For example, the display assembly (1900) may be positioned toward the user's face. The display assembly (1900) may include a display panel (e.g., the display module (1660) of FIG. 16 and / or the display panel (1910) of FIG. 3A) and a lens assembly (e.g., the lens assembly (1920) of FIG. 3A).

[0236] According to one embodiment, the display assembly (1900) may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0237] In one embodiment, when the display assembly (1900) is formed of one of a liquid crystal display (LCD), a digital mirror display (DMD), or a silicon liquid crystal display (SiLCD), the wearable electronic device (1800) may include a light source that irradiates light (e.g., visible light) to a screen output area of ​​the display assembly (1900). In another embodiment, when the display assembly (1900) can generate light (e.g., visible light) on its own, for example, when the wearable electronic device (1800) is formed of one of an organic light-emitting diode (OLED) or a micro LED, the wearable electronic device (1800) may provide a user with good quality XR content images even without including a separate light source. For example, if the display assembly (1900) is implemented with an organic light-emitting diode (OLED) or a micro LED, a light source is unnecessary, and thus the wearable electronic device (1800) may be lightweight.

[0238] According to one embodiment, the display assembly (1900) may include a first display assembly (1900a) and / or a second display assembly (1900b). According to one embodiment, the first display assembly (1900a) may be positioned to face the user's left eye in the fourth direction (④), and the second display assembly (1900b) may be positioned to face the user's right eye in the third direction (③).

[0239] According to one embodiment, the display assembly (1900) may include a lens assembly (e.g., lens assembly (1920) of FIG. 3A) including a transparent waveguide. The lens assembly (1920) may serve to adjust a focus so that a screen (e.g., XR content image) output from a display panel (e.g., display panel (1910) of FIG. 3A) can be viewed by a user. For example, light (e.g., visible light) emitted from the display panel (1910) may pass through the lens assembly (1920) and be transmitted to the user through a waveguide formed within the lens assembly (1920). The lens assembly (1920) may include at least one of a Fresnel lens, a pancake lens, a convex lens, or a multi-channel lens.

[0240] In one embodiment, the first function cameras (e.g., recognition cameras) (1815) can acquire images when the wearable electronic device (1800) is worn by the user. The first function cameras (1815) can be used for the purpose of detecting user movements or recognizing user gestures. For example, the first function cameras (1815) can be used for at least one of hand detection, hand tracking, recognition of user gestures (e.g., hand movements), and / or space recognition. For example, the first function cameras (1815) mainly use GS (global shutter) cameras, which have superior performance compared to RS (rolling shutter) cameras, to detect and track fine movements of hand movements and fingers, and can be configured as a stereo camera including two or more GS cameras for head tracking and space recognition. The first function cameras (1815) can be used for 3DoF, 6DoF head tracking, location (spatial, environmental) recognition, and / or movement recognition. The first function camera (1815) can perform simultaneous localization and mapping (SLAM) functions to recognize information (e.g., location and / or direction) related to the surrounding space through spatial recognition for 6DoF and depth shooting. In one embodiment, the second function cameras (1811, 1812) can also be used for hand detection and tracking, and user gestures.

[0241] In one embodiment, the second function camera (e.g., a photographing camera) (1811, 1812) can obtain an image related to the surrounding environment of the wearable electronic device (1800). The second function camera (1811, 1812) can be used to photograph the outside and generate an image or video corresponding to the outside and transmit it to a processor (e.g., a processor (1620) of FIG. 16). The processor (1620) can display the image provided from the second function camera (1811, 1812) on the display assembly (1900). The second function camera (1811, 1812) may also be referred to as an HR (high resolution) or PV (photo video) camera and may include a high-resolution camera. For example, the second function camera (1811, 1812) may include, but is not limited to, a color camera equipped with functions for obtaining high-quality images, such as an auto focus (AF) function and an optical image stabilizer (OIS). The second function camera (1811, 1812) may also include a GS camera or an RS camera.

[0242] In one embodiment, a third function camera (e.g., a gaze tracking camera) (e.g., a camera (1940) of FIG. 3B) may be positioned in the display assembly (1900) (or inside the housing (1810)) so that the camera lens faces the user's eyes when the user wears the wearable electronic device (1800). The third function camera (1940) may be used for detecting and tracking pupils (ET) and / or recognizing the user's iris. The processor (1620) may track the movement of the user's left and right eyes in the images received from the third function camera (1940) to determine the gaze direction. The processor (1620) may track the position of the pupils in the images so that the center of the XR content image displayed in the screen display area is positioned according to the direction in which the pupils are gazing. As an example, a GS camera may be used as the third function camera (1940) to detect the pupil and track eye movement. The third function camera (1940) may be installed for the left and right eyes, respectively, and cameras with the same performance and specifications may be used.

[0243] In one embodiment, the fourth functional camera (e.g., a face recognition camera) (1825, 1826, 1827) may be used to detect and track (FT) the user's facial expression when the user wears the wearable electronic device (1800). For example, the fourth functional camera (1825, 1826, 1827) may be used to recognize the user's face, or to recognize and / or track the user's two eyes.

[0244] According to one embodiment, a depth sensor (or depth camera) (1817) can be used to check the distance to an object (e.g., an object), such as time of flight (TOF). Time of flight (TOF) is a technology that measures the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser). After a transmitter transmits a signal, a receiver measures the signal, and the distance to the object can be measured based on the flight time of the signal. For example, the depth sensor (1817) can be configured to transmit a signal and receive a signal reflected from a subject. Instead of or in addition to the depth sensor (1817), a first camera (1845a, 1845b, 1845c, 1845d) can check the distance to an object.

[0245] According to one embodiment, the touch sensor (1813) may be arranged in the second direction (②) of the housing (1810). The touch sensor (1813) may be implemented as a single type or a left / right separated type depending on the shape of the housing (1810), but is not limited thereto. For example, when the touch sensor (1813) is implemented as a left / right separated type as illustrated in FIG. 18, when the user wears the wearable electronic device (1800), the first touch sensor (1813a) may be arranged at the user's left eye position, such as in the fourth direction (④), and the second touch sensor (1813b) may be arranged at the user's right eye position, such as in the third direction (③).

[0246] In one embodiment, the touch sensor (1813) can recognize a touch input in at least one of, for example, a capacitive, a pressure-sensitive, an infrared, or an ultrasonic manner. For example, the capacitive touch sensor (1813) can recognize a physical touch (or contact) input or a hovering input (or proximity) of an external object. According to some embodiments, the wearable electronic device (1800) may also utilize a proximity sensor (not shown) to enable proximity recognition of an external object.

[0247] According to one embodiment, the touch sensor (1813) has a two-dimensional surface and can transmit touch data (e.g., touch coordinates) of an external object (e.g., a user's finger) that comes into contact with the touch sensor (1813) to a processor (e.g., the processor (1620) of FIG. 16). The touch sensor (1813) can detect a hovering input for an external object (e.g., a user's finger) that approaches within a first distance from the touch sensor (1813), or detect a touch input that touches the touch sensor (1813).

[0248] According to one embodiment, the touch sensor (1813) may provide two-dimensional information about the point of contact as “touch data” to the processor (1620) when an external object touches the touch sensor (1813). The touch data may be described as a “touch mode.” The touch sensor (1813) may provide hovering data about the time or location of hovering around the touch sensor (1813) to the processor (1620) when an external object is located within a first distance from the touch sensor (or in proximity, hovering above the touch sensor). The hovering data may be described as a “hovering mode / proximity mode.”

[0249] According to one embodiment, the wearable electronic device (1800) can obtain hovering data using at least one of a touch sensor (1813), a proximity sensor (not shown), or / and a depth sensor (1817) to generate information about a distance, location, or time point between the touch sensor (1813) and an external object.

[0250] According to one embodiment, the interior of the housing (1810) may include components of FIG. 16, for example, a processor (e.g., processor (1620) of FIG. 16) and memory (e.g., memory (1630) of FIG. 16).

[0251] In one embodiment, the memory (1630) may store various instructions that may be performed by the processor (1620). The instructions may include control commands such as arithmetic and logical operations, data movement, or input / output that may be recognized by the processor (1620). The memory (1630) may temporarily or permanently store various data, including volatile memory (e.g., volatile memory (1632) of FIG. 16) and non-volatile memory (e.g., non-volatile memory (1634) of FIG. 16).

[0252] In one embodiment, the processor (1620) may be operatively, functionally, and / or electrically connected to each component of the wearable electronic device (1800) and may be configured to perform operations or data processing related to control and / or communication of each component. Operations performed by the processor (1620) may be stored in the memory (1630) and, when executed, may be executed by instructions that cause the processor (1620) to operate.

[0253] Hereinafter, the computational and data processing functions that the processor (1620) can implement on the wearable electronic device (1800) are not limited, but a series of operations related to the XR content service function will be described. The operations of the processor (1620) described below can be performed by executing instructions stored in the memory (1630).

[0254] According to one embodiment, the processor (1620) may generate a virtual object based on virtual information based on image information. The processor (1620) may output a virtual object related to an XR service together with background space information through the display assembly (1900). For example, the processor (1620) may capture an image related to a real space corresponding to the field of view of a user wearing the wearable electronic device (1800) through the second function camera (1811, 1812) to obtain image information or generate a virtual space for a virtual environment. For example, the processor (1620) may control the display assembly (1900) to display XR content (hereinafter referred to as an XR content screen) in which at least one virtual object is output to be overlapped in an area determined to be a display area or a field of view (FoV) of the user.

[0255]

[0256] According to one embodiment, an electronic device includes an image sensor (e.g., an image sensor (310) of FIG. 3; a camera module (1680) of FIG. 16); and a processor (e.g., a processor (320) of FIG. 3; a processor (1620) of FIG. 16), wherein the processor operates in a first mode for obtaining first vision information by using pixels of a preset color among pixels included in the image sensor, and when an input corresponding to a mode change is received through the electronic device during the first mode operation, the electronic device may operate in a second mode for obtaining image data by using pixels of a plurality of colors included in the image sensor, and obtaining second vision information based at least in part on luminance information of the image data.

[0257] According to one embodiment, the input corresponding to the mode change may be an input that considers at least one of the amount of change in the object recognized in the first mode, the type of the object recognized, or the user's image capture input.

[0258] According to one embodiment, the second mode, when the mode is switched from the first mode, acquires the second vision information based on at least a part of the first vision information acquired in the first mode, and the first vision information may include at least one of a location of a recognized object, a type of a recognized object, or brightness of a surrounding environment.

[0259] According to one embodiment, the second mode may include a low-resolution mode including acquiring video image data for video shooting and acquiring the second vision information based on luminance information of the video image data, and a high-resolution mode including acquiring photo image data having a higher resolution than the video image data for image shooting and acquiring third vision information having a higher resolution than the second vision information based on luminance information of the photo image data.

[0260] According to one embodiment, the first vision information and the second vision information may be characterized by having the same resolution.

[0261] According to one embodiment, the electronic device further includes a display, and the processor, when operating in the second mode, can control at least a portion of the image data acquired using the plurality of colored pixels to be displayed on the display.

[0262] According to one embodiment, the first mode includes a first vision mode and a second vision mode, and the number of pixels of the first group of pixels of the one color of the image sensor used in the first vision mode is smaller than the number of pixels of the second group of pixels of the one color of the image sensor used in the second vision mode, and the first group of pixels can be included in the second group of pixels.

[0263] According to one embodiment, the first sub-vision information used in the first vision mode and the second sub-vision information used in the second vision mode may be characterized by different numbers of frames per second, different degrees of applied gain, different degrees of applied noise reduction.

[0264] According to one embodiment, the second sub-vision information can be obtained based on brightness, location of a detected object, or type of a detected object included in the first sub-vision information.

[0265] According to one embodiment, the power circuits supplied to the first mode and the second mode may be characterized in that they are separated.

[0266] In one embodiment, the first vision mode may operate in a low power state or when the user is not wearing the electronic device.

[0267] According to one embodiment, a device for processing vision information includes an image sensor (e.g., an image sensor (310) of FIG. 3; a camera module (1680) of FIG. 16); a first power circuit (a first power circuit (610) of FIG. 6) for supplying a first power to a vision pixel, which is a pixel preset for a vision mode among pixels included in the image sensor; a second power circuit (a second power circuit (620) of FIG. 6) for supplying a second power to the remaining pixels of the image sensor excluding the vision pixel; and a processor, wherein the processor (e.g., the processor (320) of FIG. 3; the processor (1620) of FIG. 16) controls, in the vision mode, to output pixel data from the vision pixel of the image sensor, receives the output pixel data, and processes the output pixel data into the vision information, and in the vision mode, supplies the first power only to the vision pixel among the image sensors through the first power circuit.

[0268] In one embodiment, the vision pixel may be a green pixel if the image sensor is an RGB sensor, or may be a portion of the green pixels if the image sensor is an RGB sensor.

[0269] In one embodiment, the vision pixel may be a white pixel if the image sensor is an RGBW sensor, or may be a portion of a white pixel if the image sensor is an RGBW sensor.

[0270] According to one embodiment, the processor may generate a demosaiced image by demosaicing pixel data of all pixels received from the image sensor when the current mode includes a shooting mode, convert the demosaiced image into a YCbCr format to generate a YCbCr image, process the YCbCr image as image information of the shooting mode, and process luminance information of the YCbCr image as vision information of the vision mode.

[0271] According to one embodiment, the resolution of the vision information may be characterized as being smaller than the resolution of the YCbCr image.

[0272] In one embodiment, the processor may supply power to all pixels of the image sensor by supplying the first power to the first power circuit and the second power to the second power circuit.

[0273] According to one embodiment, the shooting mode may include a low-resolution mode including acquiring video image data for video shooting and acquiring first vision information based on luminance information of the video image data, and a high-resolution mode including acquiring photo image data having a higher resolution than the video image data for image shooting and acquiring second vision information having a higher resolution than the first vision information based on luminance information of the photo image data.

[0274] According to one embodiment, when the processor receives a user input to change the current mode to a shooting mode, the processor can perform the shooting mode in response to the user input.

[0275] According to one embodiment, the processor can detect an object from the vision information and perform a shooting mode based on the movement of the object or the type of the object.

[0276]

[0277] The method according to the embodiment may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, or data structures, either singly or in combination. The program instructions recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, or flash memories. Examples of the program instructions include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0278] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems, and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0279] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the above. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described systems, structures, devices, or circuits are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0280] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices. image sensor; and Contains a processor, The above processor, It operates in a first mode for obtaining first vision information by using pixels of one preset color among the pixels included in the image sensor, When an input corresponding to a mode change is received through the electronic device during the first mode operation, the image sensor operates in a second mode for obtaining image data using pixels of multiple colors included in the image sensor and obtaining second vision information based at least in part on the brightness information of the image data. Electronic devices.

2. In paragraph 1, The input corresponding to the above mode switching is, An input that considers at least one of the amount of change in the object recognized in the first mode, the type of the object recognized, or the user's image capture input. Electronic devices.

3. In any one of paragraphs 1 and 2, The second mode is, When the mode is switched in the first mode, the second vision information is acquired based on at least a part of the first vision information acquired in the first mode, The above first vision information is, Contains at least one of the location of the recognized object, the type of the recognized object, or the brightness of the surrounding environment. Electronic devices.

4. In any one of paragraphs 1 to 3, The second mode is, A low-resolution mode including acquiring video image data for video shooting and acquiring second vision information based on luminance information of the video image data; Including a high-resolution mode including obtaining photograph image data having a higher resolution than the video image data for image capture, and obtaining third vision information having a higher resolution than the second vision information based on brightness information of the photograph image data. Electronic devices.

5. In any one of paragraphs 1 to 4, The above first vision information and the above second vision information are, characterized by the same resolution Electronic devices.

6. In any one of paragraphs 1 to 5, The above electronic device, Including more displays, The above processor, When operating in the second mode, at least a portion of the image data acquired using the pixels of the plurality of colors is controlled to be displayed on the display. electronic devices, 7. In any one of paragraphs 1 to 6, The above first mode is, Includes first vision mode and second vision mode, The number of pixels of the first group of pixels of the one color of the image sensor used in the first vision mode is smaller than the number of pixels of the second group of pixels of the one color of the image sensor used in the second vision mode, The above first group pixels are included in the above second group pixels. Electronic devices.

8. In any one of paragraphs 1 to 7, The first sub-vision information used in the first vision mode and the second sub-vision information used in the second vision mode are, characterized by different frames per second, gain applied, sensitivity, or degree of noise reduction applied. Electronic devices.

9. In any one of paragraphs 1 to 8, The above second sub-vision information is, Obtained based on the brightness, location of the detected object, or type of the detected object included in the first sub-vision information. Electronic devices.

10. In any one of paragraphs 1 to 9, The power circuits supplied to the first and second modes are characterized in that they are separated. Electronic devices.

11. In any one of paragraphs 1 to 10, The above first vision mode is, Operating in low power conditions or when the user is not wearing the electronic device. Electronic devices.

12. In a device for processing vision information, image sensor; A first power circuit for supplying a first power to a vision pixel, which is a pixel preset for a vision mode among the pixels included in the image sensor; A second power circuit for supplying a second power to the remaining pixels of the image sensor except for the vision pixel; and Processor Including, The above processor, In the above vision mode, control is given to output pixel data from the vision pixel of the image sensor, receive the output pixel data and process it into the vision information, In the above vision mode, the first power is supplied only to the vision pixel among the image sensors through the first power circuit. device.

13. In paragraph 12, The above processor, When the current mode includes shooting mode, Demosaicing is performed using pixel data of all pixels received from the image sensor to generate a demosaiced image, Converting the above demosaiced image into YCbCr format to generate a YCbCr image, Processing the above YCbCr image as image information of the above shooting mode, Processing the luminance information of the above YCbCr image into vision information of the above vision mode. device.

14. In any one of paragraphs 12 to 13, The above processor, Supplying power to all pixels of the image sensor by supplying the first power to the first power circuit and the second power to the second power circuit. device.

15. In any one of paragraphs 12 to 14, The above shooting mode is, A low-resolution mode including acquiring video image data for video shooting and acquiring first vision information based on luminance information of the video image data; Including a high-resolution mode including obtaining photograph image data having a higher resolution than the video image data for image capturing, and obtaining second vision information having a higher resolution than the first vision information based on brightness information of the photograph image data. device.

Images