Electronic device for performing vision perception from image acquired using meta lens, and operating method thereof

Abstract

An electronic device for performing vision perception from an image acquired using a meta lens, and an operating method thereof are provided. The electronic device according to one embodiment of the present disclosure comprises: a meta lens having a pattern formed on the surface thereof and including a plurality of pillars or pins with different shapes, heights and widths; an image sensor which receives phase-modulated light reflected from an object and transmitted through the meta lens, and which converts the received light into an electrical signal so as to acquire a coded image; and at least one processor which inputs the coded image into an artificial intelligence model, and which uses the artificial intelligence model to acquire a label that indicates a perception result of an object through inference, wherein the artificial intelligence model can be a neural network model trained to acquire a simulation image by inputting an RGB image into a model reflecting optical characteristics of the meta lens, and output, as the perception result of the simulation image, a label indicating a ground truth of the RGB image that was input.

Description

Electronic device for performing vision recognition from an image acquired using a metalens and its operating method

[0001] The present disclosure relates to an electronic device that performs vision recognition from an image acquired using a meta-lens, and an operating method thereof. Specifically, the present disclosure discloses an electronic device that acquires a coded image through a camera including a meta-lens, and performs vision recognition, such as object detection, classification, face detection, or segmentation, from the coded image using an artificial intelligence model.

[0002] Recently, cameras are increasingly being installed in home appliances and IoT (Internet of Things) devices, such as TVs, refrigerators, and robot vacuums, and the images captured by these cameras are being used for monitoring purposes. However, advancements in vision-based AI technology have raised the risk of personal information leakage due to image data hijacking or hacking when cameras are used for monitoring in places where personal privacy should not be exposed, such as the home or indoors. Due to this risk of privacy leakage, some users are reluctant to incorporate cameras into home appliances or IoT devices.

[0003] Conventional methods for preventing personal information leakage include using event cameras, which capture images based on changes in light intensity rather than capturing RGB images by receiving light reflected from an object. Event cameras offer advantages in privacy and preventing personal information leakage because they do not capture RGB images that can be visually identified by humans. However, this method suffers from the limitation that it can only capture images when there is a change in light intensity.

[0004] One aspect of the present disclosure provides an electronic device that acquires a vision recognition result from an image acquired through a metalens. According to an embodiment of the present disclosure, the electronic device may include a metalens having a pattern formed on a surface of a plurality of pillars or pins having different shapes, heights, and widths, and having optical properties that modulate the phase of light reflected from an object through the pattern on the surface. According to an embodiment of the present disclosure, the electronic device may include an image sensor that receives phase-modulated light reflected from an object and transmitted through the metalens, and converts the received light into an electrical signal to acquire a coded image. According to an embodiment of the present disclosure, the electronic device may include a memory that stores one or more instructions; and at least one processor. The at least one processor may execute the one or more instructions, so that the electronic device inputs the coded image into an artificial intelligence (AI) model and acquires a label representing the recognition result of the object through inference using the AI ​​model. In one embodiment of the present disclosure, the artificial intelligence model may be a neural network model trained to input an RGB image into a model reflecting the optical characteristics of a meta lens to obtain a simulation image, and output a label indicating the ground truth of the RGB image as a recognition result of the obtained simulation image. The artificial intelligence model may be trained to minimize the loss value by applying a similarity value that numerically represents the degree of similarity between the RGB image and the simulation image as a loss value.

[0005] Another aspect of the present disclosure provides a method for an electronic device to recognize an object from an image acquired through a metalens. The method may include the steps of receiving phase-modulated light reflected from an object and transmitted through the metalens, and converting the received light into an electrical signal to acquire a coded image. The method may include the steps of inputting the coded image into an artificial intelligence (AI) model, and acquiring a label representing the object recognition result through inference using the AI ​​model.

[0006] Another aspect of the present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may include instructions readable by an electronic device, such that the electronic device performs the following operations: receiving phase-modulated light reflected from an object and transmitted through a metalens, converting the received light into an electrical signal to obtain a coded image, and inputting the coded image into an artificial intelligence (AI) model and obtaining a label representing the object recognition result through inference using the AI ​​model.

[0007] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0008] FIG. 1 is a conceptual diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to acquire a coded image using a metalens and output a vision recognition result using an artificial intelligence model.

[0009] FIG. 2 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to obtain a vision recognition result from an image acquired using a meta lens.

[0010] FIG. 3 is a perspective view illustrating the shape of a surface pattern of a metalens according to one embodiment of the present disclosure.

[0011] FIG. 4 is a block diagram illustrating components of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 5 is a block diagram illustrating components of an electronic device and a server according to one embodiment of the present disclosure.

[0013] FIG. 6 is a diagram for explaining a method for training an artificial intelligence model according to one embodiment of the present disclosure.

[0014] FIG. 7 is a diagram illustrating a method for training an artificial intelligence model according to one embodiment of the present disclosure.

[0015] FIG. 8 is a diagram illustrating a method for training an artificial intelligence model according to one embodiment of the present disclosure.

[0016] FIG. 9 is a diagram for explaining a method for training an artificial intelligence model according to one embodiment of the present disclosure.

[0017] FIG. 10 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to obtain a vision recognition result using an artificial intelligence model changed according to a vision task.

[0018] FIG. 11 is a diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to change the configuration of an artificial intelligence model according to a vision task.

[0019] FIG. 12 is a diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to change the configuration of an artificial intelligence model according to a vision task.

[0020] FIG. 13 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to change the configuration of an artificial intelligence model when a replacement or change of a metalens is recognized.

[0021] FIG. 14 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to change the configuration of an artificial intelligence model when a replacement or change of a metalens is recognized.

[0022] FIG. 15 is a diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to obtain a vision recognition result using a camera system including a metalens, as well as a depth sensor and an audio sensor.

[0023] FIG. 16 is a diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to obtain a vision recognition result using a camera system including a metalens, as well as a depth sensor and an audio sensor.

[0024] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0025] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0026] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0027] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0028] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0029] In the present disclosure, a 'metalens' is a lens having an optical property of modulating the phase of light reflected from an object, and including a metasurface composed of a pattern of nano-sized pillars or fins. In one embodiment of the present disclosure, the metasurface may be formed of a plurality of pillars or fins having different shapes, heights, and widths.

[0030] In the present disclosure, a "coded image" is an image obtained using phase-modulated light transmitted through a metalens. In one embodiment of the present disclosure, an image sensor can obtain a coded image by receiving phase-modulated light transmitted through a metalens and converting the received light into an electrical signal.

[0031] In the present disclosure, functions related to 'artificial intelligence' are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0032] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0033] In the present disclosure, the 'artificial intelligence model' may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operations through operations between the operation results of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network model may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0034] In the present disclosure, 'vision recognition' means image signal processing that inputs an RGB image or an encoded image into an artificial intelligence model, and recognizes an object (detection), recognizes a human face (face detection), classifies an object into a specific category (classification), or segments an object from the input image through inference using the artificial intelligence model.

[0035] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0036] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0037] FIG. 1 is a conceptual diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to obtain an encoded image (20) using a meta lens (110) and output a vision recognition result (30) using an artificial intelligence model (200).

[0038] FIG. 2 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to obtain a vision recognition result from an image acquired using a meta lens.

[0039] Hereinafter, with reference to FIGS. 1 and 2 together, a method for the electronic device of the present disclosure to output a vision recognition result (30) from an encoded image (20) acquired through a meta lens (110) will be described in detail.

[0040] Referring to step S210 of FIG. 2, the electronic device receives phase-modulated light transmitted through a metalens (110, see FIG. 1), and converts the received light into an electrical signal to obtain a coded image (20, see FIG. 1).

[0041] Referring to FIG. 1 together, the electronic device may include a metalens (110), an image sensor (120), and an artificial intelligence model (200).

[0042] A meta-lens (110) is a lens including a meta-surface composed of nano-sized pillars or fins. The meta-surface is a three-dimensional surface having a pattern composed of a plurality of pillars or a plurality of fins having different heights and widths, and the spacing between the plurality of pillars or the plurality of fins may be different. The meta-lens (110) may have an optical characteristic of encoding light by changing the amount of light transmitted through the meta-surface and reflected from the object (10). In one embodiment of the present disclosure, the meta-lens (110) may induce a phase delay of light and modulate the phase of light by changing the refractive index of light according to a pattern composed of a plurality of pillars or fins included in the meta-surface. In one embodiment of the present disclosure, the height, width, and spacing of the plurality of functions or fins included in the meta-surface may be determined by parameter values ​​of a mathematically modeled model, such as a point spread function (PSF), according to the learning results of the artificial intelligence model (200). The metasurface is described in detail in Fig. 3.

[0043] The image sensor (120) is configured to receive light whose phase is modulated by passing through the metalens (110), and to convert the received light into an electrical signal to obtain an encoded image (20). In one embodiment of the present disclosure, the image sensor (120) may be formed of a complementary metal-oxide-semiconductor (CMOS), but is not limited thereto. Light reflected by the object (10) has its refractive index changed by the metalens (110) to be phase-modulated, and the phase-modulated light is received by a specific pixel of the image sensor (120). The image sensor (120) can obtain an encoded image (20) by converting the received light into an electrical signal.

[0044] A coded image (20) is an image obtained by having the light reflected from an object (10) phase-modulated by a meta lens (110) and receiving the modulated light by an image sensor (120). The coded image (20) may be a non-readable image in which the shape of the object (10) cannot be confirmed by the human eye, unlike an RGB image, due to a focus being modulated or distorted.

[0045] Referring to step S220 of FIG. 2, the electronic device inputs an encoded image (20) into an artificial intelligence model (200, see FIG. 1), and performs inferencing using the artificial intelligence model (200) to obtain a label indicating a vision recognition result (30, see FIG. 1) of the encoded image (20). Referring also to FIG. 1, the 'artificial intelligence model (200)' may be a neural network model trained in a supervised learning manner to input an RGB image into a model reflecting the optical characteristics of a meta lens (110) to obtain a simulated image, and output a label indicating the ground truth of the input RGB image as a recognition result of the obtained simulated image. In one embodiment of the present disclosure, the artificial intelligence model (200) may be implemented as a convolutional neural network model (CNN). However, it is not limited thereto, and the artificial intelligence model (200) may be implemented as, for example, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.

[0046] The 'label' obtained as a result of inference by the electronic device using the artificial intelligence model (200) may be classification information of an object included in an encoded image input to the artificial intelligence model. For example, if an object photographed through a meta lens (110) and an image sensor (120) is a cat, the artificial intelligence model (200) may output a probability value that the object can be classified as a label indicating 'bird'. In the embodiment illustrated in FIG. 1, the electronic device may output a label corresponding to 'bird', which is a vision recognition result (30) of an encoded image (20) input to the artificial intelligence model (200), through inference using the learned artificial intelligence model (200).

[0047] However, it is not limited thereto, and the electronic device may output an object recognized from an encoded image (20), an inference result using an artificial intelligence model (200), or obtain a segmentation result of an object.

[0048] In the present disclosure, the artificial intelligence model (200) illustrated and described in FIG. 2 may be replaced with the artificial intelligence models (200a, 200b, 200c) illustrated in FIGS. 7 to 9. A specific method for learning the artificial intelligence models (200, 200a, 200b, 200c) will be described in detail in FIGS. 6 to 9.

[0049] Recently, cameras are increasingly being installed in home appliances such as TVs, refrigerators, and robot vacuums, as well as Internet of Things (IoT) devices, and the images captured by these cameras are increasingly being used for monitoring individuals. Advances in vision-based AI technology have raised the risk of personal information leakage due to image data hijacking or hacking when cameras are used for monitoring in places where personal privacy should not be exposed, such as the home or indoors. Conventional event cameras, which have been used to prevent personal information leakage, capture images based on changes in light intensity, which may not capture RGB images that are visually discernible to humans. However, event cameras have the limitation that they can only capture images when there is a change in light intensity. Furthermore, recent advances in AI technology have made it possible to restore visually discernible images from images captured by event cameras. Therefore, solutions that protect personal privacy and enhance security are needed.

[0050] The purpose of the present disclosure is to provide an electronic device and an operating method thereof that obtains an encoded image (20) that cannot be visually identified by a person using a meta lens (110) and obtains a vision recognition result (30) from the encoded image (20) in order to protect an individual's privacy and enhance security when a camera is used for monitoring, etc.

[0051] The electronic device according to the embodiments illustrated in FIGS. 1 and 2 obtains an encoded image (20) that cannot be visually identified and confirmed by a person by modulating light reflected from an object (10) through a meta-lens (110), thereby providing a technical effect that can prevent the risk of personal information leakage in advance and enhance security even when used indoors such as at home. In addition, the electronic device according to one embodiment of the present disclosure obtains a vision recognition result (30) from the encoded image (20) through inference using a learned artificial intelligence model (200), thereby improving the recognition rate compared to existing methods (e.g., event cameras, etc.). In addition, since the meta-lens (110) does not require a plurality of lenses to receive light, unlike a conventional camera lens, the electronic device according to one embodiment of the present disclosure also provides the effect of making the lens thin and realizing a small form factor by using the meta-lens (110).

[0052] In the embodiments illustrated in FIGS. 1 and 2, the electronic device is illustrated and described as using a metalens (110) to obtain a coded image (20), but the present disclosure is not limited thereto. Throughout the present disclosure, the metalens (110) may be replaced with a phase mask. In one embodiment of the present disclosure, the electronic device may induce a phase delay of light by changing the refractive index of light reflected by an object using a phase mask, and may receive the light in which the phase delay has occurred to obtain a coded image. In the present disclosure, the 'phase mask' is an element that has a coded aperture including a plurality of apertures having different shapes and sizes, and changes the refractive index of light reflected from an object by differently controlling the amount of light transmitted according to the aperture amount by the plurality of apertures included in the coded aperture, thereby causing a focus distortion phenomenon. In one embodiment of the present disclosure, focus distortion of an image due to a phase mask can be mathematically modeled by a point spread function (PSF), thereby enabling deconvolution to obtain an image in a normal focus state.

[0053] An electronic device according to one embodiment of the present disclosure can obtain a vision recognition result (30) by inputting an encoded image obtained through a phase mask into an artificial intelligence model (200).

[0054] FIG. 3 is a perspective view illustrating the shape of a surface pattern of a metalens (110) according to one embodiment of the present disclosure.

[0055] Referring to FIG. 3, the meta-lens (110) may include a meta-surface composed of a plurality of pillars (112) having nano-sized dimensions. The plurality of pillars (112) protrude upward on the substrate (114), and the plurality of pillars (112) may form a three-dimensional surface pattern. The plurality of pillars (112) may be formed to have different heights (h) and widths. In the embodiment illustrated in FIG. 3, the plurality of pillars (112) are illustrated in a cylindrical shape, but the shape of the plurality of pillars (112) is not limited to a cylindrical shape. In one embodiment of the present disclosure, the shape of the plurality of pillars (112) may be a cylindrical shape, a cone shape, a square pillar shape, or a combination thereof. The plurality of pillars (112) formed on the meta-surface may also be replaced with a rough or fin structure.

[0056] The height (h), cross-sectional diameter (d), and volume of each of the plurality of columns (112) included in the meta-lens (110) and the spacing between the plurality of columns (112) can be determined based on model parameter values ​​according to the learning results of the artificial intelligence models (200, 200a, 200b, 200c, see FIGS. 6 to 9). In one embodiment of the present disclosure, the artificial intelligence models (200, 200a, 200b, 200c) include a differentiable mathematical model that mathematically models a point spread function (PSF) representing the optical characteristics of the meta-lens (110) and sensor noise of the image sensor (120, see FIG. 1), and the model parameters can be updated and optimized through supervised learning using pairs of input images and ground truth values. The height (h), cross-sectional diameter (d), and volume of the plurality of columns (112) of the metasurface and the spacing between the plurality of columns (112) can be determined based on a phase function that simulates phase modulation by the optical characteristics of the meta-lens (110) among the model parameters optimized through learning of the artificial intelligence model (200, 200a, 200b, 200c). In one embodiment of the present disclosure, the phase function can be an equation that defines coefficients of a polynomial that mathematically represents the degree of phase modulation according to the distance from the center of the meta-lens (110).

[0057] FIG. 4 is a block diagram illustrating components of an electronic device (100) according to one embodiment of the present disclosure.

[0058] Referring to FIG. 4, the electronic device (100) may be implemented as a smart phone, tablet PC, laptop computer, digital camera, e-book terminal, digital broadcasting terminal, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), navigation, or MP3 player, which includes a camera system. In one embodiment of the present disclosure, the electronic device (100) may be a home appliance, such as a smart TV, air conditioner, robot vacuum cleaner, or clothes manager, which includes a camera. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the electronic device (100) may also be implemented as a wearable device, such as a smart watch, glasses-type augmented reality device (e.g., AR (Augmented Reality) glasses), or head-mounted device (HMD), which includes a camera.

[0059] Referring to FIG. 4, the electronic device (100) may include a meta lens (110), an image sensor (120), a processor (130), and a memory (140). The image sensor (120), the processor (130), and the memory (140) may be electrically and / or physically connected to each other, respectively. In FIG. 4, only essential components for explaining the operation of the electronic device (100) are illustrated, and the components included in the electronic device (100) are not limited to those illustrated in FIG. 4. In one embodiment of the present disclosure, the electronic device (100) may further include a depth sensor or an audio sensor. When the electronic device (100) is implemented as a mobile device, the electronic device (100) may further include a battery that supplies power to the image sensor (120) and the processor (130).

[0060] In one embodiment of the present disclosure, the electronic device (100) may further include a communication interface (150, see FIG. 5) configured to perform data communication with a server or an external device. An embodiment in which the electronic device (100) includes the communication interface (150) will be described in detail with reference to FIG. 5.

[0061] A meta-lens (110) is a lens having optical characteristics that change the amount of light transmitted from an object and modulate or distort the phase of light by refracting or diffracting the light. The meta-lens (110) may include a meta-surface composed of a plurality of pillars or fins having nano-sized dimensions. The meta-surface is a three-dimensional surface having a pattern composed of a plurality of pillars or a plurality of fins having different heights and widths, and the spacing between the plurality of pillars or the plurality of fins may be different. The meta-lens (110) may have optical characteristics that encode light reflected from an object (10) through the meta-surface. In one embodiment of the present disclosure, the meta-lens (110) may induce a phase delay of light and modulate the phase of light by changing the refractive index of light according to a pattern composed of a plurality of pillars or fins included in the meta-surface. Since the structure of the meta-lens (110) including the meta-surface is the same as that described in FIG. 3, a redundant description will be omitted.

[0062] The image sensor (120) is configured to receive light whose phase is modulated by passing through the metalens (110), and to convert the received light into an electrical signal to obtain a coded image. In one embodiment of the present disclosure, the image sensor (120) may be formed of a complementary metal-oxide-semiconductor (CMOS), but is not limited thereto. Light reflected by an object has its refractive index changed by the metalens (110) to be phase-modulated, and the phase-modulated light is received by a specific pixel of the image sensor (120). The image sensor (120) can obtain a coded image by converting the received light into an electrical signal. The image sensor (120) can provide image data of the obtained coded image to the processor (130).

[0063] The processor (130) can execute one or more instructions of a program stored in the memory (140). The processor (130) may be configured as hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (130) is illustrated as a single element in FIG. 4, it is not limited thereto. In one embodiment of the present disclosure, the processor (130) may be configured with one or more elements. In one embodiment of the present disclosure, the processor (130) may be configured with one or more elements. One or more processors included in the processor (130) may be circuitry such as a system on chip (SoC), an integrated circuit (IC), etc.

[0064] In one embodiment of the present disclosure, the processor (130) may be implemented as a general-purpose processor such as one or more CPUs (Central Processing Units), APs (Application Processors), DSPs (Digital Signal Processors), a graphics-only processor such as a GPU (Graphics Processing Unit), VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU (Neural Processing Unit). The processor (130) may be controlled to process input data according to predefined operation rules or artificial intelligence models (200) stored in the memory (140). Alternatively, when the processor (130) is an artificial intelligence-only processor, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0065] The processor (130) may include various processing circuits and / or multiple processors. For example, the term "processor" as used in this disclosure, including the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to perform various functions described in this disclosure individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0066] The memory (140) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0067] The memory (140) may store instructions related to operations for obtaining vision recognition results from encoded images acquired by the electronic device (100) through the meta lens (110) and the image sensor (120). In one embodiment of the present disclosure, the memory (140) may store at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor (130). The instructions, algorithms, data structures, and program codes stored in the memory (140) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0068] Below, the functions or operations performed by the processor (130) by executing commands or program codes included in modules stored in the memory (140) will be described.

[0069] The processor (130) can obtain an encoded image through a meta lens (110) and an image sensor (120). The processor (130) inputs the encoded image into an artificial intelligence model (200) and can obtain a label indicating the recognition result of the encoded image through inference using the artificial intelligence model (200).

[0070] In the embodiment illustrated in FIG. 4, the artificial intelligence model (200) may be stored on-device in the electronic device (100). In one embodiment of the present disclosure, instructions, algorithms, data structures, or program codes constituting the artificial intelligence model (200) may be stored in the memory (140).

[0071] The artificial intelligence model (200) may be a neural network model trained in a supervised learning manner to input an RGB image into a model that reflects the optical characteristics of the meta lens (110) to obtain a simulated image, and output a label indicating the ground truth of the input RGB image as a recognition result of the obtained simulated image. In one embodiment of the present disclosure, the artificial intelligence model (200) may be implemented as a convolutional neural network (CNN). However, the present invention is not limited thereto, and the artificial intelligence model (200) may also be implemented as, for example, a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.

[0072] In one embodiment of the present disclosure, the artificial intelligence model (200) may include a first artificial intelligence model trained to output a simulation image from an RGB image by convolving a point spread function that numerically models the optical characteristics of the meta-lens (110) with an RGB image. The first artificial intelligence model may be a deep neural network model trained to minimize the loss value by calculating the similarity between the RGB image and the simulation image using a differentiable mathematical model or neural network algorithm that numerically measures the similarity between a plurality of images and updating the weights through backpropagation that applies the calculated similarity as a loss value. The similarity between the RGB image applied as an input image to the training of the first artificial intelligence model and the simulation image output during the training process may be calculated, for example, through any one of visual information fidelity (VIF), Learned Perceptual Image Patch Similarity (LPIPS), or a combination thereof.

[0073] In one embodiment of the present disclosure, the artificial intelligence model (200) may include a second artificial intelligence model trained to output a recognition result according to the characteristics of a vision task indicating the purpose or use of vision recognition when an encoded image is input. The second artificial intelligence model may be a deep neural network model trained to apply a difference value between a label indicating a correct answer value paired with an RGB image input to the artificial intelligence model (200) and a label indicating a recognition result according to the vision task as a loss value, and to minimize the loss value. A specific method for training the artificial intelligence model (200) including the first artificial intelligence model and the second artificial intelligence model will be described in detail with reference to FIG. 6.

[0074] In one embodiment of the present disclosure, the artificial intelligence model (200) may further include a third artificial intelligence model trained to reconstruct an RGB image, which is an original image before being distorted by the optical characteristics of the meta-lens (110), when a simulation image is input. The third artificial intelligence model may be a deep neural network model trained to calculate a difference value between the original RGB image input to the artificial intelligence model and the reconstructed image, and to apply the inverse of the calculated difference value as a loss value to minimize the loss value. When the artificial intelligence model (200) includes the third artificial intelligence model, the artificial intelligence model (200) illustrated in FIG. 4 may be replaced with the artificial intelligence model (200a) illustrated in FIG. 7. A specific method of training the artificial intelligence model (200a, see FIG. 7) including the third artificial intelligence model will be described in detail in FIG. 7.

[0075] In one embodiment of the present disclosure, the artificial intelligence model (200) may further include a fourth artificial intelligence model trained to output personal identification information recognized from an encoded image when an encoded image is input. The fourth artificial intelligence model may be a deep neural network model trained to calculate a difference value between a ground truth paired with an RGB image input to the artificial intelligence model (200) and personal identification information output as a recognition result, and to apply the inverse of the calculated difference value as a loss value to minimize the loss value. In the present disclosure, 'personal identification information' refers to personal information that can be recognized or identified from an encoded image, and may include, for example, at least one of gender, age, race, skin color, hair color, or eye color. When the artificial intelligence model (200) includes the fourth artificial intelligence model, the artificial intelligence model (200) illustrated in FIG. 4 may be replaced with the artificial intelligence model (200b) illustrated in FIG. 8. A specific method for learning an artificial intelligence model (200b, see Fig. 8) including the fourth artificial intelligence model will be described in detail in Fig. 8.

[0076] In one embodiment of the present disclosure, the artificial intelligence model (200) may include all of the first to fourth artificial intelligence models. In this case, the artificial intelligence model (200) may be trained to minimize all loss values ​​generated during the training process of the first to fourth artificial intelligence models included. When the artificial intelligence model (200) includes all of the first to fourth artificial intelligence models, the artificial intelligence model (200) illustrated in FIG. 4 may be replaced with the artificial intelligence model (200c) illustrated in FIG. 9. A specific method for training the artificial intelligence model (200c) including all of the first to fourth artificial intelligence models will be described in detail in FIG. 9.

[0077] The processor (130) can obtain a label representing the vision recognition result of the encoded image through inference by inputting the encoded image to the artificial intelligence model (200). In the present disclosure, the 'label' may be classification information of an object included in the input encoded image. For example, if an object captured through the meta lens (110) and the image sensor (120) is a cat, the artificial intelligence model (200) can output a probability value that the object can be classified with a label indicating 'cat'.

[0078] However, it is not limited thereto, and the processor (130) may detect an object, detect a face, or segment an object from an encoded image as a result of vision recognition using the artificial intelligence model (200). For example, if the vision recognition using the artificial intelligence model (200) is 'face recognition', the processor (130) may input an RGB image into the artificial intelligence model (200), perform inference using the artificial intelligence model (200), and output the location coordinate values ​​of an area recognized as a human face as a result of vision recognition.

[0079] In one embodiment of the present disclosure, the artificial intelligence model (200) may include a backbone network trained to extract a feature map from an input encoded image and a head network trained to output a label indicating a recognition result of the encoded image from the feature map. The processor (130) may change at least one of the backbone network and the head network based on a vision task to be recognized using the artificial intelligence model (200). In one embodiment of the present disclosure, the artificial intelligence model (200) may include a plurality of backbone networks and a plurality of head networks. The processor (130) may select a backbone network optimized according to a vision task indicating a purpose or use of vision recognition from among the plurality of backbone networks, and change the backbone network of the artificial intelligence model (200) to the selected backbone network. In addition, the processor (130) may select a head network optimized according to a vision task from among the plurality of head networks, and change the head network of the artificial intelligence model (200) to the selected head network.

[0080] In one embodiment of the present disclosure, when the head network changes according to a vision task, the backbone network can be applied equally regardless of the vision task. In this case, the structure of the backbone network can also be shared with the changed head network. A specific embodiment in which the processor (130) changes at least one of the backbone network and the head network based on the vision task will be described in detail in FIGS. 10 to 12 .

[0081] In one embodiment of the present disclosure, the artificial intelligence model (200) may further include a metalens profiler model that recognizes replacement or change of the metalens (110) by the user. The processor (130) may recognize replacement or change of the metalens (110) by the user through the metalens profiler model. When replacement or change of the metalens (110) is recognized, the processor (130) may identify a vision task corresponding to the replaced or changed metalens (110) and change the backbone network and head network of the artificial intelligence model (200) to a predetermined backbone network and head network as models optimized for the identified vision task. A specific embodiment in which the processor (130) changes the backbone network and head network of the artificial intelligence model (200) according to replacement or change of the metalens (110) by the user will be described in detail with reference to FIGS. 13 and 14.

[0082] In one embodiment of the present disclosure, the electronic device (100) may further include a depth sensor (170, see FIGS. 15 and 16) or an audio sensor (180, see FIGS. 15 and 16). The processor (130) may obtain depth value information of an object from the depth sensor (170) and may obtain an acoustic signal emitted by the object from the audio sensor (180). The processor (130) may input the depth value and the acoustic signal into an artificial intelligence model (200) and obtain a vision recognition result of an encoded image through inference using the artificial intelligence model (200). A specific embodiment in which the processor (130) obtains a vision recognition result of an encoded image using the depth value and the acoustic signal will be described in detail with reference to FIGS. 15 and 16.

[0083] FIG. 5 is a block diagram illustrating components of an electronic device (100) and a server (300) according to one embodiment of the present disclosure.

[0084] Referring to FIG. 5, the electronic device (100) may further include a communication interface (150) that performs data communication with the server (300). The metalens (110), the image sensor (120), and the processor (130) included in the electronic device (100) are the same as the metalens (110), the image sensor (120), and the processor (130) illustrated in FIG. 4, and therefore, redundant descriptions are omitted.

[0085] The communication interface (150) can transmit and receive data with the server (300) through a wired or wireless communication network, and process the data. The communication interface (150) can perform data communication with the server (300) using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication. However, the present invention is not limited thereto, and when the electronic device (100) is implemented as a mobile device, the communication interface (150) can transmit and receive data to and from the server (300) through a network that follows a mobile communication standard such as a communication method using CDMA, WCDMA, 3G, 4G (LTE), 5G, and / or millimeter wave (mmWave).

[0086] In one embodiment of the present disclosure, the communication interface (150) can transmit an encoded image to the server (300) under the control of the processor (130) and receive a label indicating a recognition result of the encoded image through the artificial intelligence model (200) from the server (300). The communication interface (150) can provide information on the label received from the server (300) to the processor (130).

[0087] The server (300) may include a communication interface (310) for communicating with the electronic device (100), a memory (330) for storing at least one instruction or program code, and a processor (320) configured to execute the instructions or program code stored in the memory (330).

[0088] A trained artificial intelligence model (200) may be stored in the memory (330) of the server (300). The artificial intelligence model (200) stored in the server (300) is identical to the artificial intelligence model (200) illustrated and described in FIGS. 1 and 4 , and thus, a duplicate description thereof will be omitted. In one embodiment of the present disclosure, any one of the artificial intelligence models (200, 200a, 200b, 200c) illustrated in FIGS. 6 to 9 may be stored in the memory (330) of the server (300). The server (300) may obtain image data of an encoded image from the electronic device (100) via the communication interface (310). The processor (320) of the server (300) inputs the acquired encoded image into an artificial intelligence model (200, 200a, 200b, 200c) and performs inference using the artificial intelligence model (200, 200a, 200b, 200c), thereby obtaining a label indicating the recognition result of the encoded image. The processor (320) can control the communication interface (310) to transmit the data of the label to the electronic device (100).

[0089] In general, the electronic device (100) may be limited in the storage capacity of the memory (140, see FIG. 3), the computational processing speed of the processor (130, see FIG. 3), the collection capability of the learning data set, etc., compared to the server (300). Therefore, the electronic device (100) according to one embodiment of the present disclosure transmits input data used for vision recognition, for example, image data of an encoded image acquired using a meta lens (110), to the server (300) through a communication network, and the server (300) may perform inference through an artificial intelligence model (200, 200a, 200b, 200c) and transmit the vision recognition result of the encoded image, for example, label data, to the electronic device (100). Through this, the electronic device (100) can receive and use data representing the vision recognition result from the server (300) without a processor having a large memory and fast computing capability, thereby shortening the processing time required for vision recognition of an encoded image and improving the accuracy of vision recognition.

[0090] FIG. 6 is a diagram for explaining a method of training an artificial intelligence model (200) according to one embodiment of the present disclosure.

[0091] The artificial intelligence model (200) illustrated in FIG. 6 may be included in an on-device manner within the electronic device (100), but is not limited thereto, and in one embodiment of the present disclosure, the artificial intelligence model (200) may be stored in a server or an external device.

[0092] The artificial intelligence model (200) may be a neural network model trained in a supervised learning manner to input an RGB image (600) and output a label (602) corresponding to a ground truth value paired with the RGB image (600). In one embodiment of the present disclosure, the artificial intelligence model (200) may be an end-to-end neural network model trained in a manner of minimizing a loss value (630), which is a difference value between a label (620) predicted as a recognition result from the RGB image (600) and a label (602) corresponding to the ground truth value. In the embodiment illustrated in FIG. 6, the artificial intelligence model (200) is a model trained to recognize coordinate values ​​at which a face is recognized from an input RGB image (600), and the label (602) corresponding to the ground truth value may be, for example, face recognition coordinates. However, it is not limited thereto, and the label (602) corresponding to the correct answer value may be a different value depending on the purpose or use of the vision recognition of the artificial intelligence model (200). For example, the label (602) may be a value indicating classification information of an object (e.g., a label value indicating 'person') or a location coordinate value for segmentation of an object.

[0093] Referring to FIG. 6, the artificial intelligence model (200) may include a first artificial intelligence model (210) and a second artificial intelligence model (220).

[0094] The first artificial intelligence model (210) may be a neural network model trained to output a simulation image (610) from an input RGB image (600) by reflecting the optical characteristics of the meta-lens (110, see FIG. 1). The first artificial intelligence model (210) may include a differentiable mathematical model that mathematically models the optical characteristics of the meta-lens (110). The first artificial intelligence model (210) may input the RGB image (600) into the differentiable mathematical model to output the simulation image (610). In one embodiment of the present disclosure, the first artificial intelligence model (210) may mathematically model the image data of the RGB image (600) by reflecting the phase modulation by the meta-surface of the meta-lens (110), and convolve a point spread function (PSF) that mathematically models the focus distortion or modulation per pixel of the RGB image (600), thereby obtaining the simulation image (610).

[0095] In the present disclosure, the 'simulated image (610)' is an image generated by a differentiable mathematical model learned to reflect the optical characteristics of the meta-lens (110), and may be a simulated image similar to a coded image obtained by an image sensor (120, see FIG. 1) by transmitting the meta-lens (110). In one embodiment of the present disclosure, the first artificial intelligence model (210) may obtain the simulated image (610) by adding simulated sensor noise of the image sensor (120, see FIG. 1) to the convolution result of the RGB image (600) and the point spread function. The first artificial intelligence model (210) may output the obtained simulated image (610) to the second artificial intelligence model (220).

[0096] The second artificial intelligence model (220) may be a neural network model trained to output a predicted label (620) based on the vision recognition result from the input simulation image (610). In one embodiment of the present disclosure, the second artificial intelligence model (220) may be implemented as a convolutional neural network (CNN). However, the present disclosure is not limited thereto, and the second artificial intelligence model (220) may also be implemented as, for example, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.

[0097] In one embodiment of the present disclosure, the learned second artificial intelligence model (220) may be a neural network model learned to output a recognition result according to the characteristics of a vision task indicating the purpose or use of vision recognition when an encoded image is input. The processor (130, see FIG. 4) of the electronic device (100) inputs a distorted encoded image through a meta-lens (110, see FIGS. 1, 3, and 4) into the learned second artificial intelligence model (220), and performs inference through the second artificial intelligence model (220) to obtain a label indicating a recognition result according to the characteristics of the vision task. In the embodiment illustrated in FIG. 6, the processor (130) inputs an encoded image into the second artificial intelligence model (220), and recognizes a face from the encoded image through inference using the second artificial intelligence model (220), and outputs a label indicating the location coordinate values ​​where the face is recognized. However, it is not limited thereto, and the second artificial intelligence model (220) may be implemented as a deep neural network model that recognizes objects from an input image, outputs classification information that classifies the category of an object, or segments the object. In this case, the processor (130) may input an encoded image to the second artificial intelligence model (220) and output a label indicating the object recognition result, classification category information, or segmentation result.

[0098] The artificial intelligence model (200) may be a neural network model trained to update and optimize multiple weight values ​​of neural network layers included in the first artificial intelligence model (210) and the second artificial intelligence model (220) through backpropagation that applies a loss value (630) which is a difference between a label (602), which is a correct answer value paired with an input RGB image (600), and a label (620) predicted from the RGB image (600), to the first artificial intelligence model (210) and the second artificial intelligence model (220). In one embodiment of the present disclosure, the artificial intelligence model (200) may be an end-to-end neural network model trained to output a label (602), which is a correct answer value, from an input RGB image (600). In one embodiment of the present disclosure, during the learning process of the artificial intelligence model (200), the gradient by the partial differentiation of the error function representing the loss value (630) is applied in the order of the first artificial intelligence model (210) to the second artificial intelligence model (220) and backpropagated, thereby updating and optimizing the multiple weight values ​​of the multiple neural network layers included in the first artificial intelligence model (210) and the second artificial intelligence model (220). As the multiple weight values ​​of the multiple neural network layers included in the first artificial intelligence model (210) and the second artificial intelligence model (220) are updated and optimized, the loss value (630) of the artificial intelligence model (200) can be learned to be minimized.

[0099] In the learning process of the artificial intelligence model (200), the first artificial intelligence model (210) may be trained to minimize information indicating the similarity between the input RGB image (600) and the output simulation image (610). In one embodiment of the present disclosure, the first artificial intelligence model (210) may be trained to minimize the loss value by calculating the similarity (640) between the RGB image (600) and the simulation image (610) using a differentiable mathematical model or neural network algorithm that numerically measures the similarity between a plurality of images and applying the calculated similarity (640) as a loss value. In one embodiment of the present disclosure, the 'similarity (640)' may be a value calculated by numerically measuring the structural similarity such as the shape of the contour edge, the intensity, and the color between the RGB image (600) and the simulation image (610). The first artificial intelligence model (210) can calculate the similarity (640) between the RGB image (600) and the simulation image (610) using, for example, at least one of visual information fidelity (VIF), Learned Perceptual Image Patch Similarity (LPIPS), or a combination thereof. However, the present invention is not limited thereto, and the first artificial intelligence model (210) can calculate the similarity (640) between the RGB image (600) and the simulation image (610) using any type of metric that can numerically measure the similarity between images, such as cosine similarity.

[0100] The first artificial intelligence model (210) can be trained to minimize the loss value by updating the weights through back propagation that applies the calculated similarity (640) as the loss value. In the case of VIF, a larger value means a higher similarity. For example, when calculating the similarity (640) between an RGB image (600) and a simulation image (610) through VIF, the first artificial intelligence model (210) can be trained to minimize the loss value by applying the similarity (640) as the loss value. In the case of LPIPS, a larger value means a lower similarity. For example, when calculating the similarity (640) between an RGB image (600) and a simulation image (610) through LPIPS, the first artificial intelligence model (210) can be trained to apply the inverse of the similarity (640) as the loss value so that the value calculated by LPIPS becomes the maximum.

[0101] If image data is hijacked or hacked during the learning or inference process of the artificial intelligence model (200), there is a risk of personal information leakage. In particular, the greater the correlation between the RGB image (600) and the simulation image (610), for example, the cross entropy, the higher the risk that the simulation image (610) output by the first artificial intelligence model (210) may be restored to a human-readable image if it is leaked.

[0102] The first artificial intelligence model (210) according to the embodiment illustrated in FIG. 6 is trained to calculate the similarity (640) between the RGB image (600) and the simulation image (610) through metrics such as VIF or LPIPS during the learning process, and to apply the similarity (640) as a loss value to minimize the loss value. Therefore, even if the simulation image (610) is leaked due to hijacking or hacking, the leaked simulation image (610) may not be similar to the RGB image (600). In addition, since the simulation image (610) is a distorted image reflecting the optical characteristics of the meta-lens, it may be a non-readable image that cannot be identified by a human. That is, the electronic device (100) according to one embodiment of the present disclosure provides a technical effect of protecting an individual's privacy and enhancing security, as even if an encoded image is leaked during the process of inputting an image acquired through the meta-lens into the artificial intelligence model (200) and performing inference, an image that is not similar to the input image is leaked.

[0103] FIG. 7 is a diagram for explaining a method of training an artificial intelligence model (200a) according to one embodiment of the present disclosure.

[0104] Referring to FIG. 7, the artificial intelligence model (200a) may be a neural network model trained in a supervised learning manner to input an RGB image (700) and output a label (702) corresponding to a ground truth value paired with the RGB image (700). In one embodiment of the present disclosure, the artificial intelligence model (200a) may be an end-to-end neural network model trained in a manner of minimizing a loss value (730), which is a difference value between a label (720) predicted as a recognition result from the RGB image (700) and a label (702) corresponding to the ground truth value. In the embodiment illustrated in FIG. 7, the artificial intelligence model (200a) is a model trained to recognize coordinate values ​​at which a face is recognized from an input RGB image (700), and the label (702) corresponding to the ground truth value may be, for example, face recognition coordinates. However, it is not limited to this, and the label (702) corresponding to the correct answer value may be a different value depending on the purpose or use of vision recognition by the artificial intelligence model (200a).

[0105] Referring to FIG. 7, the artificial intelligence model (200a) may include a first artificial intelligence model (210), a second artificial intelligence model (220), and a third artificial intelligence model (230). The first artificial intelligence model (210) and the second artificial intelligence model (220) illustrated in FIG. 7 are identical to the first artificial intelligence model (210, see FIG. 6) and the second artificial intelligence model (220, see FIG. 6) illustrated in FIG. 6, and therefore, any redundant description will be omitted. In particular, the RGB image (700), the simulation image (710), the label predicted as a result of vision recognition (720), the loss value (730), and the similarity (740) illustrated in FIG. 7 are identical to the RGB image (600, see FIG. 6), the simulation image (610, see FIG. 6), the label predicted as a result of vision recognition (620, see FIG. 6), the loss value (630, see FIG. 6), and the similarity (640, see FIG. 6) illustrated and described in FIG. 6, except that the reference numbers are different, and therefore, redundant descriptions will be omitted.

[0106] The third artificial intelligence model (230) may be a neural network model trained to reconstruct the original image before it is distorted by reflecting the optical characteristics of the meta-lens (110) when a simulated image (710) output by the first artificial intelligence model (210) is input. In the embodiment illustrated in FIG. 7, the third artificial intelligence model (230) may output a reconstructed image (750) when a simulated image (710) is input. The 'reconstructed image (750)' may be an RGB image predicted to be similar to the RGB image (700), which is the original image before being simulated by reflecting the optical characteristics of the meta-lens (110) by the first artificial intelligence model (210). The third artificial intelligence model (230) may be trained by calculating the difference value between the restored image (750) and the original RGB image (700) applied as input to the artificial intelligence model (200a), and applying the inverse of the difference value as a reconstruction prevention loss (760). In one embodiment of the present disclosure, the third artificial intelligence model (230) may be a deep neural network model trained to minimize the loss value by updating weights through back propagation that applies the reconstruction prevention loss (760). That is, the third artificial intelligence model (230) may be trained to maximize the difference value between the original RGB image (700) and the restored image (750).

[0107] In the embodiment illustrated in FIG. 7, the artificial intelligence model (200a) may further include a third artificial intelligence model (230) trained to output a restored image (750) when a simulation image (710) is input, in addition to the first artificial intelligence model (210) and the second artificial intelligence model (220). The third artificial intelligence model (230) applies the inverse of the difference value between the restored image (750) and the original RGB image (700) as a restoration prevention loss (760), and is trained to minimize the restoration prevention loss (760), so that even if the restored image (750) is obtained from the simulation image (710), the restored image (750) may not be similar to the original RGB image (700). Accordingly, the electronic device (100) according to one embodiment of the present disclosure provides a technical effect of enhancing privacy protection and security, as the original RGB image acquired by the image sensor is not completely restored even if the encoded image acquired through the meta lens is leaked during the inference process using the artificial intelligence model (200a).

[0108] FIG. 8 is a diagram for explaining a method of training an artificial intelligence model (200b) according to one embodiment of the present disclosure.

[0109] Referring to FIG. 8, the artificial intelligence model (200b) may be a neural network model trained in a supervised learning manner to input an RGB image (800) and output a label (802) corresponding to a ground truth value paired with the RGB image (800). In one embodiment of the present disclosure, the artificial intelligence model (200b) may be an end-to-end neural network model trained in a manner of minimizing a loss value (830), which is a difference value between a label (820) predicted as a recognition result from the RGB image (800) and a label (802) corresponding to the ground truth value.

[0110] Referring to FIG. 8, the artificial intelligence model (200b) may include a first artificial intelligence model (210), a second artificial intelligence model (220), and a fourth artificial intelligence model (240). The first artificial intelligence model (210) and the second artificial intelligence model (220) illustrated in FIG. 8 are the same as the first artificial intelligence model (210, see FIG. 6) and the second artificial intelligence model (220, see FIG. 6) illustrated in FIG. 6, and therefore, overlapping descriptions will be omitted. Similarly to FIG. 6, the first artificial intelligence model (210) may be trained to calculate the similarity (840) between an RGB image (800) and a simulation image (810) using a differentiable mathematical model or neural network algorithm that numerically measures the similarity between multiple images, and to apply the calculated similarity (840) as a loss value so as to minimize the loss value. In Fig. 8, the 'similarity (840)' is calculated in the same manner as the method of calculating the similarity (640, see Fig. 6) illustrated and described in Fig. 6, so redundant descriptions are omitted.

[0111] In the embodiment illustrated in FIG. 8, the second artificial intelligence model (220) may be a neural network model trained to output recognition results according to the characteristics of a vision task representing the purpose or use of vision recognition. For example, the second artificial intelligence model (220) may be a neural network model trained to perform a face recognition task that recognizes a face from a simulation image (810) when a simulation image (810) is input and outputs the location coordinate values ​​where the face is recognized. However, the present invention is not limited thereto, and the second artificial intelligence model (220) may also be a deep neural network model trained to perform a vision task of recognizing, classifying, or segmenting an object.

[0112] The fourth artificial intelligence model (240) is a neural network model trained to not output information corresponding to personal identification information. The fourth artificial intelligence model (240) may be a deep neural network model trained to output a label (850) indicating personal identification information as a result of vision recognition when a simulation image (810) is input, calculate a difference value (860) between the label (850) corresponding to the output personal identification information and the label (802) corresponding to the correct answer applied as input to the artificial intelligence model (200b), and apply the inverse of the calculated difference value (860) as a loss value to minimize the loss value. In one embodiment of the present disclosure, the artificial intelligence model (200b) may be an end-to-end neural network model trained to update and optimize the weight values ​​of neural network layers included in the first artificial intelligence model (210) as well as the fourth artificial intelligence model (240) by applying the inverse of the difference value (860) produced by the fourth artificial intelligence model (240) as a loss value.

[0113] The fourth artificial intelligence model (240) may be implemented as, for example, a convolutional neural network (CNN). However, it is not limited thereto, and in one embodiment of the present disclosure, the fourth artificial intelligence model (240) may also be implemented as, for example, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.

[0114] In one embodiment of the present disclosure, 'personal identification information' refers to personal information that can be recognized from an encoded image acquired by the electronic device (100) through a meta lens (110, see FIGS. 1, 3, and 4). The personal identification information may include, for example, at least one of gender, age, race, skin color, hair color, or eye color that can be recognized from the encoded image.

[0115] In one embodiment of the present disclosure, the fourth artificial intelligence model (240) can be trained to output specific personally identifiable information that a user does not want to leak when an encoded image is hijacked or hacked. The fourth artificial intelligence model (240) illustrated in FIG. 8 can be trained not to recognize the gender of a person included in an RGB image (800) applied as input to the artificial intelligence model (200b) when a simulation image (810) is input. That is, the fourth artificial intelligence model (240) can be trained by applying the inverse of the difference value (860) as a loss value so that the difference value (860) between the label (802) corresponding to the correct answer value of the gender (e.g., male) paired with the input RGB image (800) and the label (850) of the gender (e.g., female) predicted according to the vision recognition result is maximized. However, it is not limited thereto, and the fourth artificial intelligence model (240) can be trained not to recognize at least one of the age, race, skin color, hair color, or eye color of a person included in the RGB image (800) applied as input to the artificial intelligence model (200b) when a simulation image (810) is input.

[0116] In the embodiment illustrated in FIG. 8, the artificial intelligence model (200b) may further include, in addition to the first artificial intelligence model (210) and the second artificial intelligence model (220), a fourth artificial intelligence model (240) trained not to output personal identification information when a simulation image (810) is input. The fourth artificial intelligence model (240) is trained to minimize the loss value by applying the inverse of the difference value as the loss value so that the difference value between the label (850) output as personal identification information according to the vision recognition result and the label (802) corresponding to the correct answer value is maximized, so that even if personal identification information is leaked, inaccurate information may be leaked. Therefore, the electronic device (100) according to one embodiment of the present disclosure provides a technical effect of protecting an individual's privacy and enhancing security, as even if an encoded image acquired through a meta-lens is leaked by hijacking or hacking during the inference process using the artificial intelligence model (200b), inaccurate personal identification information is not leaked from the original RGB image acquired by the image sensor.

[0117] FIG. 9 is a diagram for explaining a method of training an artificial intelligence model (200c) according to one embodiment of the present disclosure.

[0118] Referring to FIG. 9, the artificial intelligence model (200c) may be a neural network model trained in a supervised learning manner to input an RGB image (900) and output a label (902) corresponding to the ground truth paired with the RGB image (900). The artificial intelligence model (200c) may include a first artificial intelligence model (210), a second artificial intelligence model (220), a third artificial intelligence model (230), and a fourth artificial intelligence model (240). The first artificial intelligence model (210) and the second artificial intelligence model (220) illustrated in FIG. 9 are identical to the first artificial intelligence model (210, see FIG. 6) and the second artificial intelligence model (220, see FIG. 6) illustrated in FIG. 6, the third artificial intelligence model (230) is identical to the third artificial intelligence model (230, see FIG. 7) illustrated in FIG. 7, and the fourth artificial intelligence model (240) is identical to the fourth artificial intelligence model (240, see FIG. 8) illustrated in FIG. 8, so any duplicate description will be omitted.

[0119] In the embodiment illustrated in FIG. 9, the artificial intelligence model (200c) may be an end-to-end neural network model learned by minimizing a loss value (930) which is a difference value between a label (920) indicating a face recognition coordinate predicted as a recognition result from an RGB image (900) and a label (902) corresponding to the correct value of the face recognition coordinate, minimizing a restoration prevention loss (960) which is the reciprocal of the difference value between a restored image (950) by a third artificial intelligence model (230) and an RGB image (900), and minimizing a loss value by applying the reciprocal of the difference value (980) between a label (970) indicating a gender recognition result output by a fourth artificial intelligence model (240) and a label corresponding to the correct value regarding gender as a loss value.

[0120] In the embodiment illustrated in FIG. 9, the artificial intelligence model (200c) can be trained to obtain highly accurate facial recognition coordinates for face recognition, which is a vision task that a user seeks to recognize from an input image, and to recognize, with low accuracy, gender for personal identification information that the user seeks to prevent from being leaked, such as gender. In addition, the artificial intelligence model (200c) can be trained to output a restored image that is not similar to the input image so that even if an image acquired through an image sensor is distorted by a meta-lens and leaks during the inference process, it is not accurately restored. Therefore, in the electronic device (100) according to one embodiment of the present disclosure, even if an encoded image acquired through a meta-lens is leaked by hijacking or hacking during the inference process using the artificial intelligence model (200c), the original RGB image acquired by the image sensor is not restored, and personal identification information is not unintentionally leaked, thereby providing a technical effect of protecting an individual's privacy and enhancing security.

[0121] FIG. 10 is a flowchart illustrating a method for an electronic device (100) according to one embodiment of the present disclosure to obtain a vision recognition result using an artificial intelligence model changed according to a vision task.

[0122] Steps S1010 and S1020 illustrated in FIG. 10 are steps that specify the operation of step S220 illustrated in FIG. 2. Step S1010 of FIG. 10 may be performed after step S210 illustrated in FIG. 2 is performed.

[0123] FIG. 11 is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to change the configuration of a second artificial intelligence model (220, see FIGS. 6 to 9) according to a vision task.

[0124] Hereinafter, with reference to FIGS. 10 and 11 together, an operation of the electronic device (100) changing the configuration of the second artificial intelligence model (220) according to a vision task will be described.

[0125] In step S1010 of FIG. 10, the electronic device (100) changes at least one of the backbone network and the head network of the artificial intelligence model based on a vision task indicating the purpose or use of vision recognition. Referring also to FIG. 11, the second artificial intelligence model (220) is a vision recognition neural network model trained to output a vision recognition result of an encoded image when an encoded image is input. The second artificial intelligence model (220) may include a backbone network (222) and a head network (224).

[0126] The backbone network (222) is a neural network model trained to extract feature values ​​from an input encoded image and obtain a feature map using the extracted feature values. The backbone network (222) may be implemented using, for example, GoogleNet, ResNet, VGG, DarkNet, ImageNet, or U-Net, but is not limited thereto.

[0127] The head network (224) is a neural network model trained to output a label indicating the recognition result of an encoded image from a feature map output from the backbone network (222). The head network (224) may be implemented using, for example, Embedding & Softmax, Pixel-wise softmax, YOLO, or YOLO & embedding, but is not limited thereto.

[0128] The processor (130, see FIG. 4) of the electronic device (100) can change at least one of the backbone network (222) and the head network (224) of the artificial intelligence model based on the characteristics of the vision task. In the present disclosure, the 'vision task' refers to a task that outputs a vision recognition result from an input image. For example, the vision task may include, but is not limited to, object detection, classification, segmentation, face identification, pose estimation, or depth estimation.

[0129] The plurality of backbone networks (222-1 to 222-n) may be trained models optimized according to the characteristics of different vision tasks. Here, the 'optimized model' refers to a neural network model that outputs prediction results with high accuracy and requires less processing time according to the amount of learning data and the purpose and use of the vision task. For example, the first backbone network (222-1) may be a ResNet optimized for 'recognition', the second backbone network (222-2) may be a VGG optimized for 'classification', and the third backbone network (222-3) may be a U-Net optimized for 'segmentation', but is not limited thereto. In one embodiment of the present disclosure, the processor (130) may select a backbone network optimized according to the purpose or use of the vision task among the plurality of backbone networks (222-1 to 222-n). For example, if the purpose or use of vision recognition is 'detection' of an object, the processor (130) may select a first backbone network (222-1) optimized for 'recognition' among a plurality of backbone networks (222-1 to 222-n). The processor (130) may replace the existing backbone network (222) with the selected first backbone network (222-1) and change the backbone network (222) to the first backbone network (222-1).

[0130] The plurality of head networks (224-1 to 224-n) may be trained models optimized for different vision tasks. For example, the first head network (224-1) may be a pixel-wise softmax optimized for 'segmentation', the second head network (224-2) may be a YOLO optimized for 'recognition', and the third head network (224-3) may be a YOLO & embedding model optimized for 'face recognition', but is not limited thereto. In one embodiment of the present disclosure, the processor (130) may select a head network optimized for the characteristics of the vision task among the plurality of head networks (224-1 to 224-n). For example, if the purpose or use of vision recognition is object detection, the processor (130) may select a second head network (224-2) optimized for 'detection' among a plurality of head networks (224-1 to 224-n). The processor (130) may replace the existing head network (224) with the selected second head network (224-2), thereby changing the head network (224) to the second head network (224-2).

[0131] Referring back to FIG. 10, in step S1020, the electronic device (100) inputs an encoded image into an artificial intelligence model in which at least one of the backbone network and the head network has been changed, and outputs a label indicating the recognition result of the encoded image. Referring also to FIG. 11, the second artificial intelligence model (220) may include a first backbone network (222-1) and a second head network (224-2). When an encoded image is input into the second artificial intelligence model (220), a feature map may be extracted from the encoded image by the first backbone network (222-1). The extracted feature map may be input into the second head network (224-2), and a label indicating the recognition result of the encoded image may be output by the second head network (224-2). In the embodiment illustrated in Fig. 11, since both the first backbone network (222-1) and the second head network (224-2) are neural network models optimized for 'detection', the output label may be a vector value representing the object recognition result.

[0132] FIG. 12 is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to change the configuration of an artificial intelligence model according to a vision task.

[0133] Referring to FIG. 12, the second artificial intelligence model (220) is a vision recognition neural network model trained to output vision recognition results of an encoded image when an encoded image is input. The second artificial intelligence model (220) may include a first backbone network (222-1) and a head network (224). The first backbone network (222-1) and the head network (224) are the same as those described in FIG. 11, and therefore, redundant descriptions are omitted.

[0134] In the embodiment illustrated in FIG. 12, the backbone network of the second artificial intelligence model (220) is determined as the first backbone network (222-1) and may not be changed depending on the vision task. If the vision task changes depending on the purpose or use of vision recognition, the head network (224) may be changed. Each of the multiple head networks (224-1 to 224-n) may be a model that is optimized and learned according to a different vision task. For example, the first head network (224-1) may be a pixel-wise softmax optimized for 'segmentation', the second head network (224-2) may be a YOLO optimized for 'recognition', and the third head network (224-3) may be a YOLO & embedding model optimized for 'face recognition'. However, the present invention is not limited thereto.

[0135] In one embodiment of the present disclosure, the processor (130, see FIG. 4) of the electronic device (100) may select a head network optimized for the purpose or use of vision recognition among the plurality of head networks (224-1 to 224-n) and replace the existing head network (224) with the selected head network. For example, when the purpose or use of vision recognition is object detection, the processor (130) may select a second head network (224-2) optimized for 'detection' among the plurality of head networks (224-1 to 224-n) and replace the existing head network (224) with the selected second head network (224-2).

[0136] When the head network (224) is changed depending on the purpose or use of vision recognition, the model parameters of the first backbone network (222-1) may not be changed regardless of the changed head network (224). For example, the values ​​of the weights of the first backbone network (222-1) may be shared (shared weight) with respect to the head network (224) changed depending on the purpose or use of vision recognition.

[0137] However, it is not limited thereto. In one embodiment of the present disclosure, when the head network (224) is changed, the model parameters of the first backbone network (222-1) may be changed according to the vision task of the changed head network (224). For example, when the purpose or use of vision recognition is 'face recognition', the processor (130) may change the head network (224) to the third head network (224-3) and replace the weight value of the first backbone network (222-1) with a weight value optimized for face recognition.

[0138] In the embodiments illustrated in FIGS. 10 to 12, the electronic device (100) changes at least one of the backbone network (222) and the head network (224) according to a vision task representing the purpose or use of vision recognition, such as 'object recognition', 'classification', 'segmentation', 'face recognition', or 'depth value estimation', thereby providing a technical effect of improving the accuracy of the vision recognition result and shortening the processing time.

[0139] FIG. 13 is a flowchart illustrating a method for an electronic device (100) according to one embodiment of the present disclosure to change the configuration of an artificial intelligence model when a replacement or change of a metalens is recognized.

[0140] The operations according to steps S1310 to S1330 illustrated in FIG. 13 may be performed after step S210 illustrated in FIG. 2 is performed. After the operation according to step S1330 illustrated in FIG. 13 is performed, step S220 of FIG. 2 may be performed.

[0141] FIG. 14 is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to change the configuration of a second artificial intelligence model (220) when a replacement or change of a meta lens (110) is recognized.

[0142] Hereinafter, with reference to FIGS. 13 and 14 together, an operation of changing the configuration of the second artificial intelligence model (220) as the electronic device (100) recognizes the replacement or change of the meta lens (110) will be described.

[0143] In step S1310 of FIG. 13, the electronic device (100) recognizes the replacement or change of the meta-lens. Referring also to FIG. 14, the user can replace or change the meta-lens (110) depending on the purpose or use of vision recognition. The meta-lens may have specific optical characteristics depending on the purpose or use of vision recognition. An encoded image is acquired due to the optical characteristics of the hardware, such as the surface pattern of the meta-lens, and the accuracy of the vision recognition result, such as object recognition, classification, segmentation, face recognition, and situation recognition, may be determined depending on the characteristics of the encoded image. For example, the first meta-lens (110-1) may be a lens optimized for acquiring an encoded image for 'face recognition', and the second meta-lens (110-2) may be a lens optimized for acquiring an encoded image for 'situation recognition'. In the embodiment illustrated in FIG. 14, the user may replace the first meta-lens (110-1) with the second meta-lens (110-2) as the purpose of vision recognition changes.

[0144] The electronic device (100) can recognize the replacement of the metalens by the user. In one embodiment of the present disclosure, the electronic device (100) further includes a metalens profiler model (160), and the processor (130, see FIG. 4) can recognize the replacement of the metalens from the first metalens (110-1) to the second metalens (110-2) using the metalens profiler model (160). The image sensor (120) receives light that is reflected from an object (1400) and passes through the first metalens (110-1) and whose phase is modulated, and when the metalens is replaced with the second metalens (110-2) by a user input, the image sensor (120) can receive light that is reflected from the object (1420) and passes through the second metalens (110-2) and whose phase is modulated. In one embodiment of the present disclosure, the metalens profiler model (160) may be a neural network model trained in a supervised learning manner that applies as input an encoded image obtained by the image sensor (120) by receiving light whose phase is modulated by a specific metalens, and applies a vector value representing a specific metalens as a ground truth. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the metalens has an encoded structure having a unique pattern, and the metalens profiler model (160) may be implemented as a classifier model that identifies a specific metalens through the encoded structure. When the replacement of the metalens is recognized, the metalens profiler model (160) may provide a signal indicating that the replacement of the metalens is recognized to the processor (130).

[0145] Referring back to FIG. 13, in step S1320, the electronic device (100) identifies a vision task corresponding to the replaced or changed meta-lens. Referring also to FIG. 14, the processor (130) can identify a vision task optimized for the replaced meta-lens when the replacement of the meta-lens is recognized. Here, the 'vision task optimized for the meta-lens' may mean a vision task in which the accuracy of the vision recognition result output when the encoded image obtained by the image sensor (120) receiving the phase-modulated light transmitted through a specific meta-lens is input to the second artificial intelligence model (220) is high and the processing time is short. For example, the first meta-lens (110-1) may be a meta-lens optimized for 'face identification' among the vision tasks, and the second meta-lens (110-2) may be a meta-lens optimized for 'situation recognition'.

[0146] Referring back to FIG. 13, in step S1330, the electronic device (100) changes the backbone network and the head network of the artificial intelligence model to backbone networks and head networks optimized for the identified vision task. Referring also to FIG. 14, the second artificial intelligence model (220) may include a first backbone network (222-1) and a first head network (224-1). Specific descriptions of the backbone network and the head network are the same as those described with reference to FIGS. 10 to 12, and thus, redundant descriptions are omitted. In one embodiment of the present disclosure, the first backbone network (222-1) and the first head network (224-1) may be neural network models optimized for performing 'face recognition' among the vision tasks. For example, the first backbone network (222-1) may be GoogleNet, and the first head network (224-1) may be YOLO, but are not limited thereto. The processor (130) can identify the first backbone network (222-1) and the first head network (224-1) included in the second artificial intelligence model (220). In one embodiment of the present disclosure, the processor (130) can identify which vision task-optimized neural network models the identified first backbone network (222-1) and first head network (224-1) are. The processor (130) can replace the first backbone network (222-1) and the first head network (224-1) with the second backbone network (222-2) and the second head network (224-2), which are optimized for the vision task of 'situational awareness' corresponding to the second meta-lens (110-2) replaced by the user. For example, the second backbone network (222-2) can be ResNet, and the second head network (224-2) can be SoftMax, but is not limited thereto.

[0147] Referring again to FIG. 13, the electronic device (100) can input an encoded image into the modified second artificial intelligence model (220) and obtain a vision recognition result through inference using the second artificial intelligence model (220). Referring also to FIG. 14, when the second artificial intelligence model (220) includes the first backbone network (222-1) and the first head network (224-1), when the encoded image is input into the second artificial intelligence model (220), a face recognition result in which the face (1410) is bounding box-processed as an inference result can be obtained. When the second meta lens (110-2) is replaced by the user, the second artificial intelligence model (220) includes a second backbone network (222-2) and a second head network (224-2), and when an encoded image is input to the second artificial intelligence model (220), a label indicating a specific situation (e.g., “Fire” or “Danger”) can be obtained as an inference result.

[0148] In the embodiment illustrated in FIGS. 13 and 14, when a meta lens is changed or replaced by a user, the electronic device (100) automatically selects a backbone network and a head network optimized for a vision task corresponding to the changed or replaced meta lens, and replaces the existing backbone network and head network with the selected backbone network and head network, thereby changing the artificial intelligence model ('second artificial intelligence model (220)' in FIG. 14), thereby providing a technical effect of improving the accuracy of vision recognition results and shortening the processing time.

[0149] FIG. 15 is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to obtain a vision recognition result using a camera system including a meta lens (110), as well as a depth sensor (170) and an audio sensor (180).

[0150] Referring to FIG. 15, the electronic device (100) may further include a depth sensor (170), an audio sensor (180), and a multi-model converter (190), as well as a metalens (110), an image sensor (120), a processor (130, see FIG. 4), and a memory (140, see FIG. 4). The metalens (110) and the image sensor (120) are the same as those described in FIGS. 1, 3, and 4, and thus, redundant descriptions are omitted. In one embodiment of the present disclosure, the artificial intelligence model (200) illustrated in FIG. 15 may be replaced with the artificial intelligence models (200, 200a, 200b, 200c) illustrated and described in FIGS. 6 to 9.

[0151] A depth sensor (170) is a sensor configured to obtain depth information about an object (1500). Here, 'depth information' means information about the distance from the depth sensor (170) to a specific object (1500). In one embodiment of the present disclosure, the depth sensor (170) may be configured as a ToF sensor (Time of flight sensor) that irradiates light to the object (1500) using a light source and obtains depth value information according to the time required for the irradiated light to be reflected from the object (1500) and received by a light receiving sensor. However, the present invention is not limited thereto, and the depth sensor (170) may also be configured as a sensor that obtains depth information using at least one of a structured light method and a stereo image method.

[0152] An audio sensor (180) is a sensor that receives sound emitted by an object (1500) and converts the received sound into an acoustic signal, which is an electrical signal. In one embodiment of the present disclosure, the audio sensor (180) may include a microphone.

[0153] The multi-model converter (190) can obtain depth value information of an object (1500) from a depth sensor (170) and obtain an audio signal from an audio sensor (180). The multi-model converter (190) can embed the depth value information and the audio signal, respectively, and convert them into n-dimensional vector values. The multi-model converter (190) can input the embedding vector into an artificial intelligence model (200).

[0154] The artificial intelligence model (200) can receive an encoded image from the image sensor (120), receive an embedding vector from the multi-model converter (190), and output a vision recognition result based on the encoded image and the embedding vector. In the embodiment illustrated in FIG. 15, the artificial intelligence model (200) can output a label value (1510) (e.g., "Bird") according to the recognition result of the object (1500).

[0155] FIG. 16 is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to obtain a vision recognition result using a camera system including a metalens (110), as well as a depth sensor (170) and an audio sensor (180).

[0156] Referring to FIG. 16, the electronic device (100) may include a meta lens (110), an image sensor (120), a processor (130, see FIG. 4), and a memory (140, see FIG. 4), a depth sensor (170), an audio sensor (180), an artificial intelligence model (200), a depth value-based vision recognition model (250), and an acoustic signal-based vision recognition model (260). The depth sensor (170) and the audio sensor (180) are the same as those described in FIG. 15, and thus, redundant descriptions are omitted. In one embodiment of the present disclosure, the artificial intelligence model (200) illustrated in FIG. 16 may be replaced with the artificial intelligence models (200, 200a, 200b, 200c) illustrated and described in FIGS. 6 to 9.

[0157] A depth-based vision recognition model (depth-based vision algorithm) (250) is a neural network model trained to output a label value regarding the recognition result of a depth value when a depth value is input. In one embodiment of the present disclosure, the depth-based vision recognition model (250) may be a neural network model trained through supervised learning that applies a vector embedding a depth value as an input and applies a label corresponding to the result of vision recognition as a ground truth. In the embodiment illustrated in FIG. 16, when a depth value is input from a depth sensor (170), the depth-based vision recognition model (250) may output a label value (e.g., “Fish”) according to the vision recognition result through inference. The depth-based vision recognition model (250) may provide the output label value to an output integration model (270).

[0158] The sound-based vision recognition model (sound-based vision algorithm) (260) is a neural network model trained to output a label value regarding the recognition result of an acoustic signal when an acoustic signal is input. In one embodiment of the present disclosure, the sound-based vision recognition model (260) may be a neural network model trained through supervised learning that applies a vector embedding an acoustic signal as an input and applies a label corresponding to the vision recognition result of the acoustic signal as a ground truth. In the embodiment illustrated in FIG. 16, the sound-based vision recognition model (260) may output a label value (e.g., "Twitting") according to the vision recognition result through inference when an acoustic signal is input from an audio sensor (180). The sound-based vision recognition model (260) may provide the output label value to the output integration model (270).

[0159] The output aggregator (270) is a model trained to output a final recognition result regarding an object (1600) from input data input from an artificial intelligence model (200), a depth value-based vision recognition model (250), and an acoustic signal-based vision recognition model (260). In one embodiment of the present disclosure, the output aggregator (270) may be a model trained in a rule-based manner. For example, the output aggregator (270) may include a softmax. However, the present invention is not limited thereto, and the output aggregator (270) may be a neural network model trained to embed input data input from an artificial intelligence model (200), a depth value-based vision recognition model (250), and an acoustic signal-based vision recognition model (260) into feature vector values, and to output a ground truth from the feature vector values. In the embodiment illustrated in FIG. 16, the output integration model (270) may receive a recognition result (e.g., "Bird", "Yellow") of an object (1600) from an artificial intelligence model (200), receive a recognition result (e.g., "Fish") of an object (1600) from a depth value-based vision recognition model (250), receive a recognition result (e.g., "Twitting") of an object (1600) from an acoustic signal-based vision recognition model (260), and output a label value (1610) (e.g., "Bird") corresponding to the final recognition result regarding the object (1600).

[0160] In the embodiment illustrated in FIGS. 15 and 16, the electronic device (100) inputs an encoded image acquired through a meta-lens (110) and an image sensor (120) into an artificial intelligence model (200) to obtain a vision recognition result, and outputs a vision recognition result regarding an object (1500, 1600) using depth value information and an audio signal acquired by a depth sensor (170) and an audio sensor (180), respectively, thereby improving the accuracy and performance of vision recognition compared to a system including only a meta-lens (110), an image sensor (120), and an artificial intelligence model (200).

[0161] The present disclosure provides an electronic device (100) that acquires a vision recognition result from an image acquired through a metalens (110). The electronic device (100) according to an embodiment of the present disclosure may include a metalens (110) having a pattern formed on a surface of a plurality of pillars or pins having different shapes, heights, and widths, and having an optical characteristic of modulating the phase of light reflected from an object through the pattern on the surface. The electronic device (100) according to an embodiment of the present disclosure may include an image sensor (120) that receives phase-modulated light reflected from an object and transmitted through the metalens, and converts the received light into an electrical signal to acquire a coded image. The electronic device (100) according to an embodiment of the present disclosure may include a memory (140) that stores one or more instructions; and at least one processor (130). The at least one processor (130) executes the one or more commands so that the electronic device (100) can input an encoded image into an artificial intelligence (AI) model and obtain a label indicating the recognition result of the object through inference using the artificial intelligence model (200). In one embodiment of the present disclosure, the artificial intelligence model (200) may be a neural network model trained to input an RGB image into a model reflecting the optical characteristics of the meta lens (110) to obtain a simulation image, and output a label indicating the ground truth of the RGB image as a recognition result of the obtained simulation image. The artificial intelligence model (200) may be trained to minimize the loss value by applying a similarity value that numerically represents the degree of similarity between the RGB image and the simulation image as a loss value.

[0162] In one embodiment of the present disclosure, the shape, height, and width of a plurality of pillars or pins forming the surface pattern of the meta lens (110) can be formed based on mathematical modeling parameters according to the learning results of the artificial intelligence model (200).

[0163] In one embodiment of the present disclosure, the artificial intelligence model (200) may include a first artificial intelligence model (210) trained to output a simulation image from an RGB image by convolving a point spread function that mathematically models the optical characteristics of a meta lens (110) with an RGB image.

[0164] In one embodiment of the present disclosure, the first artificial intelligence model may be a deep neural network model trained to minimize a loss value by calculating the similarity between an RGB image and a simulation image using a differentiable mathematical model or neural network algorithm that numerically measures the similarity between a plurality of images, and updating weights between layers through back propagation that applies the calculated similarity as a loss value.

[0165] In one embodiment of the present disclosure, the similarity between the RGB image and the simulation image can be calculated through at least one of visual information fidelity (VIF), Learned Perceptual Image Patch Similarity (LPIPS), or a combination thereof.

[0166] In one embodiment of the present disclosure, the artificial intelligence model (200) may further include a second artificial intelligence model trained to reconstruct an original RGB image before it is distorted by the optical characteristics of the meta lens (110) when a simulation image is input.

[0167] In one embodiment of the present disclosure, the second artificial intelligence model may be a deep neural network model trained to minimize a loss value by applying the inverse of the difference value between the restored RGB image and the original RGB image input to the artificial intelligence model as a loss value.

[0168] In one embodiment of the present disclosure, the artificial intelligence model (200) may further include a third artificial intelligence model trained to output a recognition result according to the characteristics of a vision task when an input encoded image is input. The at least one processor (130) may execute one or more commands, so that the electronic device (100) may: input an encoded image to the third artificial intelligence model, and output a label indicating a recognition result according to the characteristics of the vision task through inference using the third artificial intelligence model.

[0169] In one embodiment of the present disclosure, the third artificial intelligence model may be a deep neural network model trained to apply the difference value between the label output as a recognition result and the label indicating the correct answer value paired with the RGB image input to the artificial intelligence model (200) as a loss value, and to minimize the loss value.

[0170] In one embodiment of the present disclosure, the artificial intelligence model (200) may further include a fourth artificial intelligence model trained to output personal identification information recognized from an encoded image when an encoded image is input. The fourth artificial intelligence model may be a deep neural network model trained to calculate a difference value between the personal identification information output as a recognition result and a correct answer value paired with an RGB image input to the artificial intelligence model (200), and to apply the inverse of the calculated difference value as a loss value to minimize the loss value.

[0171] In one embodiment of the present disclosure, the personally identifiable information may include at least one of gender, age, race, skin color, hair color, or eye color that can be recognized from the encoded image.

[0172] In one embodiment of the present disclosure, the artificial intelligence model (200) may further include a metalens profiler model that recognizes replacement or change of the metalens (110) by the user. The at least one processor (130) may execute the one or more commands, so that the electronic device (100) may: identify a vision task corresponding to the replaced or changed metalens (110) when the replacement or change of the metalens (110) is recognized through the metalens profiler model. The at least one processor (130) may execute the one or more commands, so that the electronic device (100) may: change the backbone network and the head network included in the artificial intelligence model (200) to a backbone network and a head network that are predetermined as models optimized for the identified vision task.

[0173] In one embodiment of the present disclosure, the electronic device (100) may further include a depth sensor (170) configured to measure a depth value of an object, and an audio sensor (180) configured to obtain an acoustic signal from the object. The at least one processor (130) may execute the one or more commands, so that the electronic device (100) may: input the depth value of the object measured by the depth sensor (170) and the acoustic signal obtained by the audio sensor (180) into an artificial intelligence model (200), and obtain a label according to the recognition result of the object through inference using the artificial intelligence model (200).

[0174] The present disclosure provides a method for an electronic device (100) to recognize an object from an image acquired through a meta-lens (110). The method may include a step (S210) of receiving phase-modulated light reflected from an object and transmitted through the meta-lens (110), and converting the received light into an electrical signal to acquire a coded image. The method may include a step (S220) of inputting the coded image into an artificial intelligence (AI) model, and acquiring a label representing the object recognition result through inference using the AI ​​model (200).

[0175] In one embodiment of the present disclosure, the artificial intelligence model (200) may include a first artificial intelligence model trained to output a simulation image from an RGB image by convolving a point spread function that mathematically models the optical characteristics of the meta-lens (110) with an RGB image. The first artificial intelligence model may be a differentiable mathematical model that numerically measures the similarity between a plurality of images, or a deep neural network model trained to calculate the similarity between the RGB image and the simulation image using a neural network algorithm, and to minimize the loss value by updating the weights between layers through back propagation that applies the calculated similarity as the loss value.

[0176] In one embodiment of the present disclosure, the similarity between the RGB image and the simulation image can be calculated through at least one of visual information fidelity (VIF), Learned Perceptual Image Patch Similarity (LPIPS), or a combination thereof.

[0177] In one embodiment of the present disclosure, the artificial intelligence model (200) may include a backbone network trained to extract a feature map from an input encoded image and a head network trained to output a label indicating a recognition result of the encoded image from the feature map. The step of obtaining the label (S220) may include a step of changing at least one of the backbone network and the head network based on the purpose or use of a vision task to be recognized using the artificial intelligence model (200) (S1010). The step of obtaining the label (S220) may include a step of inputting an encoded image to the artificial intelligence model (200) in which at least one of the backbone network and the head network has been changed, and outputting a label indicating a recognition result of the encoded image (S1020).

[0178] In one embodiment of the present disclosure, the method may further include a step of recognizing replacement or change of a meta-lens (110) by a user (S1310), and a step of identifying a vision task corresponding to the replaced or changed meta-lens (110) (S1320). The method may further include a step of changing the backbone network and head network of the artificial intelligence model (200) to a predetermined backbone network and head network as models optimized for the identified vision task (S1330).

[0179] In one embodiment of the present disclosure, the method may further include a step of obtaining a depth value of an object from a depth sensor (170), and a step of obtaining an audio signal from the object from an audio sensor (180). In the step of obtaining a label (S220), the electronic device (100) inputs the depth value of the object measured by the depth sensor (170) and the audio signal obtained by the audio sensor (180) into an artificial intelligence model (200), and obtains a label according to the recognition result of the object through inference using the artificial intelligence model (200).

[0180] The present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an electronic device (100) so that the electronic device (100) performs the following operations: receiving phase-modulated light reflected from an object and transmitted through a metalens (110), converting the received light into an electrical signal to obtain a coded image, and inputting the coded image into an artificial intelligence (AI) model and obtaining a label indicating the object recognition result through inference using the AI ​​model (200).

[0181] The program executed by the electronic device (100) described in the present disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0182] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0183] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The media may be readable by a computer, stored in a memory, and executed by a processor.

[0184] A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0185] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0186] A computer program product may include a software program, or a computer-readable storage medium storing the software program. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by the manufacturer of the electronic device (100) or through an electronic marketplace (e.g., the Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a server of the manufacturer of the electronic device (100), a server of the electronic marketplace, or a relay server that temporarily stores the software program.

[0187] In a system comprising an electronic device (100) and / or a server, the computer program product may include a storage medium of the server or the storage medium of the electronic device (100). Alternatively, if there is a third device (e.g., a wearable device) that is communicatively connected to the electronic device (100), the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the electronic device (100) to the third device, or from the third device to the electronic device.

[0188] In this case, either the electronic device (100) or the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the electronic device (100) and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0189] For example, the electronic device (100) may execute a computer program product stored in a memory (130, see FIG. 2) to control another electronic device that is in communication with the electronic device (100) to perform a method according to the disclosed embodiments.

[0190] As another example, a third device may execute a computer program product to control an electronic device in communication with the third device to perform a method according to the disclosed embodiment.

[0191] When a third device executes a computer program product, the third device may download the computer program product from the electronic device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0192] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. In an electronic device (100), A metalens (110) having a pattern formed on a surface consisting of a plurality of pillars or pins having different shapes, heights, and widths, and having optical properties that modulate the phase of light reflected from an object through the pattern on the surface; An image sensor (120) that receives phase-modulated light reflected from an object and transmitted through the metalens (110), and converts the received light into an electrical signal to obtain a coded image; A memory (140) storing one or more instructions; and At least one processor (130); Including, The at least one processor (130) executes the one or more commands, so that the electronic device (100) inputs the encoded image into an artificial intelligence (AI) model (200) and outputs a recognition result of the object through inferencing using the artificial intelligence model (200). The above artificial intelligence model (200) is a neural network model trained to input an RGB image into a model reflecting the optical characteristics of the meta lens (110) to obtain a simulation image, and output a label indicating the correct value (ground truth) of the input RGB image as a recognition result of the obtained simulation image. The above artificial intelligence model (200) is an electronic device (100) that learns to minimize the loss value by applying a similarity value that numerically represents the degree of similarity between the RGB image and the simulation image as a loss value.

2. In paragraph 1, The above artificial intelligence model (200) is An electronic device (100) including a first artificial intelligence model learned to output the simulation image from the RGB image by convolving a point spread function that mathematically models the optical characteristics of the meta lens (110) with the RGB image.

3. In paragraph 2, The above first artificial intelligence model is, An electronic device (100), which is a deep neural network model that is trained to minimize the loss value by calculating the similarity between the RGB image and the simulation image using a differentiable mathematical model or neural network algorithm that numerically measures the similarity between multiple images and updating the weights between layers through back propagation that applies the calculated similarity as a loss value.

4. In paragraph 3, An electronic device (100) in which the similarity between the RGB image and the simulation image is calculated through at least one of visual information fidelity (VIF), Learned Perceptual Image Patch Similarity (LPIPS), or a combination thereof.

5. In any one of clauses 1 to 4, The above artificial intelligence model (200) is An electronic device (100) further comprising a second artificial intelligence model learned to reconstruct an original RGB image before it is distorted by the optical characteristics of the meta lens (110) when the above simulation image is input.

6. In any one of clauses 1 to 5, The above artificial intelligence model (200) is When the above input encoded image is input, a third artificial intelligence model learned to output a recognition result according to the characteristics of the vision task is further included. The electronic device (100) executes the one or more instructions by the at least one processor (130): An electronic device (100) that inputs the encoded image into the third artificial intelligence model and outputs a label indicating a recognition result according to the characteristics of a vision task through inference using the third artificial intelligence model.

7. In any one of clauses 1 to 6, The above artificial intelligence model (200) is When the above input encoded image is input, it further includes a fourth artificial intelligence model trained to output personal identification information recognized from the encoded image, The above fourth artificial intelligence model is, An electronic device (100) which is a deep neural network model that is trained to calculate the difference value between the personal identification information output as the recognition result and the correct answer value paired with the RGB image input to the artificial intelligence model (200), and to minimize the loss value by applying the inverse of the calculated difference value as the loss value.

8. In any one of clauses 1 to 7, The above artificial intelligence model (200) is Further comprising a metalens profiler model that recognizes replacement or change of the metalens (110) by the user, The electronic device (100) executes the one or more instructions by the at least one processor (130): When the replacement or change of the meta lens (110) is recognized through the above meta lens profiler model, the vision task corresponding to the replaced or changed meta lens (110) is identified, An electronic device (100) that changes the backbone network and head network included in the above artificial intelligence model (200) into a pre-determined backbone network and head network as models optimized for the identified vision task.

9. In a method for obtaining an object recognition result from an image obtained through a metalens (110) by an electronic device (100), A step (S210) of receiving phase-modulated light reflected from an object and transmitted through the metalens (110) and converting the received light into an electrical signal to obtain a coded image; and A step (S220) of inputting the encoded image into an artificial intelligence (AI) model and obtaining a label indicating the recognition result of the object through inference using the artificial intelligence model (200); Including, The above artificial intelligence model (200) is a neural network model trained to input an RGB image into a model reflecting the optical characteristics of the meta lens (110) to obtain a simulation image, and output a label indicating the correct value (ground truth) of the input RGB image as a recognition result of the obtained simulation image. The above artificial intelligence model (200) is a method trained to minimize the loss value by applying a similarity value that numerically represents the degree of similarity between the RGB image and the simulation image as a loss value.

10. In paragraph 9, The above artificial intelligence model (200) is It includes a first artificial intelligence model that is learned to output the simulation image from the RGB image by convolving the point spread function that mathematically models the optical characteristics of the above meta lens (110) with the RGB image, The above first artificial intelligence model is, A method, wherein a deep neural network model is trained to minimize the loss value by calculating the similarity between the RGB image and the simulation image using a differentiable mathematical model or neural network algorithm that numerically measures the similarity between multiple images, and updating the weights between layers through back propagation that applies the calculated similarity as a loss value.

11. In clause 10, A method wherein the similarity between the RGB image and the simulation image is calculated through at least one of visual information fidelity (VIF), Learned Perceptual Image Patch Similarity (LPIPS), or a combination thereof.

12. In any one of clauses 9 to 11, The above artificial intelligence model (200) is It includes a backbone network trained to extract a feature map from the input encoded image and a head network trained to output a label indicating the recognition result of the encoded image from the feature map. The step (S220) of obtaining the above label is: A step (S1010) of changing at least one of the backbone network and the head network based on the purpose or use of the vision task to be recognized using the artificial intelligence model (200); and A step (S1020) of inputting the encoded image into the artificial intelligence model (200) in which at least one of the backbone network and the head network has been changed, and outputting a label indicating the recognition result of the encoded image; A method comprising:

13. In any one of paragraphs 9 to 12, Step (S1310) of recognizing replacement or change of the above metal lens (110) by the user; Step (S1320) of identifying a vision task corresponding to the above-mentioned replaced or changed meta lens (110); and Step (S1330) of changing the backbone network and head network of the above artificial intelligence model (200) into a pre-determined backbone network and head network as models optimized for the identified vision task; A method further comprising:

14. In any one of paragraphs 9 to 13, A step of obtaining a depth value of the object from a depth sensor (170); and A step of obtaining an acoustic signal from the object from an audio sensor (180); Including more, The step (S220) of obtaining the above label is: A method of inputting the depth value of the object measured by the depth sensor (170) and the sound signal acquired by the audio sensor (180) into the artificial intelligence model (200), and acquiring a label according to the recognition result of the object through inference using the artificial intelligence model (200).

15. In a computer program product including a computer-readable storage medium, The above storage medium, An operation of receiving phase-modulated light reflected from an object and transmitted through a metalens (110), and converting the received light into an electrical signal to obtain a coded image; and An operation of inputting the encoded image into an artificial intelligence (AI) model and obtaining a label indicating the recognition result of the object through inference using the artificial intelligence model (200); A computer program product comprising instructions executed by an electronic device (100) to perform an operation.

Images