Vision inspection method using in-context learning, and system therefor

Abstract

A vision inspection method using in-context learning, and a system therefor, according to one embodiment of the present disclosure, relate to a method and a system for performing, by using sequence model-based in-context learning, vision inspection on a query image by controlling an operation of a model through a prompt including a small number of images and label examples without updating model parameters.

Description

Vision inspection method and system using in-context learning

[0001] The present disclosure relates to a vision inspection method and a system using in-context learning. More specifically, the invention relates to a method and a system for performing vision inspection on a query image by controlling the operation of a model through a prompt containing a small number of image and label examples without updating model parameters, using sequence model-based in-context learning.

[0002] Recently, the importance of vision inspection technology for quality control is increasing day by day in various industrial sectors, particularly in automated production processes such as smart factories.

[0003] These vision inspection technologies have evolved in the direction of detecting defects with high accuracy by introducing artificial intelligence, particularly deep learning models.

[0004] However, conventional deep learning-based vision inspection technology has a clear limitation in that it requires developing and training individual models for each specific scenario to be solved.

[0005] For example, if a new type of defect occurs or parts of the products under inspection are changed, the entire process of collecting large-scale training data tailored to the scenario, retraining the model, and deploying it to the field must be repeated.

[0006] Since this process entails significant time and cost, there is a problem in that it is difficult to respond flexibly to the modern industrial environment, which involves multi-product, small-batch production or frequent process changes.

[0007] Furthermore, in actual industrial settings, it is always difficult to secure a sufficient amount of labeled data, and the labels on the acquired data are often mixed in various forms, such as class labels, bounding boxes, and segmentation, making it difficult to utilize them in an integrated manner.

[0008] To address these issues, there is a growing need for a new vision inspection framework that possesses the versatility and flexibility to rapidly adapt to new environments or requirements with only a limited number of examples, moving away from the approach of developing task-dependent models every time.

[0009] One embodiment of the present disclosure is devised to solve the problems of the prior art as described above, and aims to provide a method and system for performing vision inspection on a query image by controlling the operation of a model through a prompt containing a small number of image and label examples without updating model parameters using sequence model-based in-context learning.

[0010] However, the technical problems to be solved by the present disclosure and the embodiments thereof are not limited to the technical problems described above, and other technical problems may exist.

[0011] A vision inspection method using in-context learning according to one embodiment of the present disclosure comprises, in a method executed by a computer, a step in which at least one processor of the computer accesses at least one memory; - wherein the at least one memory includes a memory containing at least one task prompt defining an example of an inspection rule and a memory containing at least one query image which is an inspection target to be determined according to the rule, and the at least one processor loads the at least one task prompt and the at least one query image from the at least one memory; the at least one processor generates an embedding for the loaded at least one task prompt and an embedding for the at least one query image; and the at least one processor inputs the generated embedding for the at least one task prompt and the embedding for the at least one query image into at least one artificial intelligence model that performs in-context learning. The method comprises: a step in which the at least one processor interprets the context according to the embedding of the at least one task prompt through the at least one artificial intelligence model and infers the label of the at least one query image; and a step in which the at least one processor manifests the output data according to the inference.

[0012] In another aspect, the above-mentioned in-context learning is characterized by learning and inferring new tasks based on the context according to a given prompt.

[0013] In another aspect, the task prompt refers to an image-label pair that defines the content and criteria of a task to be performed by at least one artificial intelligence model at the time of inference, and the query image refers to a target image to be examined by applying the criteria of the task prompt.

[0014] In another aspect, the above-mentioned at least one processor further includes the step of collecting the at least one task prompt and the at least one query image by interacting with at least one of a user interface or an automated production line system.

[0015] In another aspect, the step of generating the embedding includes performing data processing to generate an embedding for the task prompt and data processing to generate an embedding for the query image sequentially or in parallel.

[0016] In another aspect, the step of generating the embedding includes generating an embedding for the task prompt through an image-label encoder that converts an image of the prompt data and a label corresponding to the image into a single embedding vector that can be processed by the at least one artificial intelligence model.

[0017] In another aspect, the step of generating the embedding further includes the step of generating an embedding for the query image through an image encoder that converts a given image into an embedding vector that can be processed by at least one artificial intelligence model.

[0018] In another aspect, the step of inferring the label of the query image includes the step of dynamically interpreting the rules of the current task based on the context according to the prompt embedding sequence, which is a list of embeddings for the task prompt.

[0019] In another aspect, the step of inferring the label of the query image further includes the step of inferring the label of the query image by applying the rules of the dynamically interpreted current task to the embeddings for the query image.

[0020] In another aspect, the step of implementing output data according to the above inference includes the step of processing the output data into a form suitable for the purpose of the task.

[0021] In another aspect, the step of implementing output data based on the above inference further includes the step of expressing the output data in different ways according to various service scenarios.

[0022] In another aspect, the step of implementing output data according to the above inference includes the step of ingesting the output data into at least one subsequent processing component, and the step of controlling the at least one subsequent processing component to represent the output data through at least one interface.

[0023] In another aspect, the above at least one processor further includes the step of training the above at least one artificial intelligence model.

[0024] In another aspect, the step of training at least one artificial intelligence model includes the step of iteratively optimizing various vision inspection scenarios for the entire architecture of at least one artificial intelligence model.

[0025] In another aspect, the step of training the at least one artificial intelligence model comprises a prompt dataset including a plurality of image-label pairs ( A test dataset including ) and a plurality of image-label pairs paired with the prompt dataset ( In-context learning episodes based on ) , The method further includes the step of calculating the loss of at least one artificial intelligence model based on ), and the step of updating the parameters of at least one artificial intelligence model in a direction that minimizes the calculated loss.

[0026] Meanwhile, a vision inspection system using in-context learning according to one embodiment of the present disclosure comprises at least one processor; and at least one memory storing at least one instruction that performs the following when executed by the at least one processor; wherein the at least one instruction comprises the step of the at least one processor accessing the at least one memory, - wherein the at least one memory includes a memory containing at least one task prompt defining an example of a check rule and a memory containing at least one query image which is a check target to be determined according to the rule, and the step of the at least one processor loading the at least one task prompt and the at least one query image from the at least one memory, the step of the at least one processor generating an embedding for the loaded at least one task prompt and an embedding for the at least one query image, the step of the at least one processor inputting the generated embedding for the at least one task prompt and the embedding for the at least one query image into at least one artificial intelligence model that performs in-context learning, and the step of the at least one processor, according to the embedding for the at least one task prompt through the at least one artificial intelligence model It includes a step of interpreting a context and inferring a label of at least one query image, and an instruction in which at least one processor performs a step of manifesting output data according to the inference.

[0027] In another aspect, a vision inspection system using in-context learning according to one embodiment of the present disclosure comprises: a plurality of neurons configured in an array comprising at least one register, at least one programmable logic, and at least one input interface; a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; and at least one routing network that controls the data flow between the plurality of neurons; wherein each of the plurality of neurons further comprises a Field Programmable Gate Array (FPGA) implementation for a predetermined artificial neural network that is connected to at least one other neuron through the routing network to establish a transmission path for the weights.

[0028] In another aspect, a vision inspection system using in-context learning according to one embodiment of the present disclosure comprises: a plurality of neurons organized into an array comprising at least one register, at least one microprocessor, and at least one input; and a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; wherein each of the plurality of neurons further comprises an Application Specific Integrated Circuit (ASIC) for a predetermined artificial neural network connected to at least one other neuron through any one of the plurality of synapse circuits.

[0029] A vision inspection method and system using in-context learning according to one embodiment of the present disclosure performs vision inspection on a query image by controlling the operation of the model through a prompt containing a small number of image and label examples without updating model parameters using in-context learning based on a sequence model, thereby implementing a general-purpose model applicable to various vision inspection tasks, significantly reducing the cost and time associated with developing and maintaining specialized models for each vision inspection task, and maximizing the efficiency of the inspection process by responding quickly and flexibly to environmental changes such as the occurrence of new types of defects or process changes.

[0030] However, the effects obtainable in the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly understood from the description below.

[0031] FIG. 1 illustrates an example of a block diagram of a computing system implementing an in-context learning-based vision inspection service according to one embodiment of the present disclosure.

[0032] FIG. 2 illustrates an example of a block diagram of a computing device implementing an in-context learning-based vision inspection service according to one embodiment of the present disclosure.

[0033] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device implementing an in-context learning-based vision inspection service according to one embodiment of the present disclosure.

[0034] FIG. 4 illustrates an example of a flowchart for explaining a vision inspection method using in-context learning according to one embodiment of the present disclosure.

[0035] FIG. 5 illustrates an example of a drawing for explaining in-context learning according to one embodiment of the present disclosure.

[0036] FIG. 6 illustrates an example of a conceptual diagram for explaining a vision inspection method using in-context learning according to one embodiment of the present disclosure.

[0037] As the present disclosure is capable of various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present disclosure, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Additionally, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Furthermore, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the present disclosure is not necessarily limited to what is depicted.

[0038] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0039]

[0040] [Exemplary system providing in-context learning-based vision inspection services]

[0041] Hereinafter, an exemplary system for implementing an in-context learning-based vision inspection service that performs vision inspection on query images by controlling the model's behavior through a prompt containing a small number of image and label examples without updating model parameters, using sequence model-based in-context learning, will be described in detail with reference to the attached drawings.

[0042] FIG. 1 illustrates an example of a block diagram of a computing system implementing an in-context learning-based vision inspection service according to one embodiment of the present disclosure.

[0043] Referring to FIG. 1, a computing system (1000) implementing the in-context learning-based vision inspection service of the present disclosure includes a user computing device (110), a server computing system (130), and a training computing system (150), and each device and system is connected to communicate via a network (170).

[0044] According to one embodiment of the present disclosure, ~~~ (title of the invention) may be implemented and provided locally by a user computing device (110), 2) implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), and 3) implemented and provided by the user computing device (110) and the server computing system (130) in conjunction with each other.

[0045] In this embodiment, the user computing device (110) and / or the server computing system (130) can train a machine learning model (120 and / or 140) through interaction with a training computing system (150) that is communicatedly connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0046] And at this time, the artificial intelligence model can be 1) trained directly locally by a user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training and learning techniques. It may also be implemented by transmitting the artificial intelligence model trained by the training computing system (150) to the user computing device (110) and / or the server computing system (130) through the network (170) to provide / update it.

[0047] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110).

[0048] - User Computing Device (110: User Computing Device)

[0049] The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a PDA (personal digital assistants), a PMP (portable multimedia player), a desktop, a wearable device, an embedded computing device and / or a tablet PC.

[0050] Additionally, in the embodiment, the user computing device (110) may further include a predetermined server computing device that provides an in-context learning-based vision inspection service environment.

[0051] This user computing device (110) includes at least one processor (111) and memory (112).

[0052] Here, the processor (111) of the user computing device (110) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0053] In particular, according to the embodiment, this processor (111) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which is a hardware technology for implementing a certain digital circuit.

[0054] Here, a Field Programmable Gate Array (FPGA) implementation can refer to a flexible digital circuit that is programmable according to user needs.

[0055] As an example, a field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results or state information of operations to support synchronized operation of the FPGA, programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits configurable according to user needs, and an input interface that receives signals from external devices or sensors and transmits them to internal circuits as a channel for receiving data from outside the FPGA.

[0056] Through the combination of the above components, a field-programmable gate array implementation can provide flexible and various types of digital circuits.

[0057] Meanwhile, an Application-Specific Integrated Circuit (ASIC) can refer to a custom integrated circuit that is fixedly designed to perform a specific use or function.

[0058] As an example, the application-dedicated integrated circuit may include a register, which is a small memory device for temporarily storing and managing data and supports the rapid processing of ASIC operations by storing intermediate calculation results or state information; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and coordinates the operation of the entire system by performing various operations or generating control signals when necessary; and an input block, which is an interface for receiving data from the outside, which receives data to be processed by the ASIC and transmits it internally, and receives various input data through connections with sensors or external devices.

[0059] Through the combination of the components mentioned above, an application-specific integrated circuit can perform specific purpose tasks in an optimized manner.

[0060] Returning to the point, the memory (112) of the user computing device (110) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc., and may include web storage of a server that performs memory storage functions on the internet. This memory (112) may store data (113) and instructions (114) necessary for the at least one processor (111) to perform functional operations, such as training an artificial intelligence model or performing vision inspection using in-context learning through an artificial intelligence model.

[0061] In one embodiment, the user computing device (110) can perform various deep learning for in-context learning-based vision inspection services by linking with a deep-learning neural network.

[0062] Here, the deep learning neural network according to the embodiment may include a Convolutional Neural Network (CNN), R-CNN (Regions with CNN features), Fast R-CNN, Faster R-CNN, Mask R-CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing the embodiments described below, and the embodiments of the present disclosure do not limit or restrict such deep learning neural networks themselves.

[0063] At this time, according to the embodiment, the deep learning neural network may be installed directly on the server computing system (130) or operate as a separate device from the server computing system (130) to perform deep learning for the in-context learning-based vision inspection service.

[0064] Additionally, in one embodiment, the user computing device (110) can store at least one machine learning model (120).

[0065] For example, the user computing device (110) may be composed of various machine learning models, such as multiple neural networks (e.g., deep neural networks) that perform a vision inspection method using in-context learning based on structured / quantitative data, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0066] For example, machine learning models may include linear regression, decision trees, random forests, gradient boosting pre-trained language models or / and deep learning models. And neural networks may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or / and other forms of neural networks.

[0067] Additionally, according to an embodiment, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process for a vision inspection method using in-context learning through a large-scale language model (LLM).

[0068] In one embodiment, a user computing device (110) receives at least one machine learning model (120) from a server computing system (130) via a network (170), stores it in memory (112), and then executes the stored machine learning model (120) by a processor (111) to perform vision inspection using in-context learning.

[0069] In another embodiment, the user computing device (110) can provide an in-context learning-based vision inspection service to the user by performing operations through a machine learning model (140) including at least one machine learning model (140) in conjunction with a server computing system (130) and communicating related data to the outside.

[0070] For example, a user computing device (110) can perform an in-context learning-based vision inspection service by having a server computing system (130) provide an output for the user's input using a machine learning model (140) via the web.

[0071] Additionally, the artificial intelligence model can be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0072] Additionally, the user computing device (110) may include at least one input component (121) that detects user input.

[0073] For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) that detects a touch of the user's input medium (e.g., a finger or a stylus), an image sensor that detects the user's motion input, a microphone that detects the user's voice input, a button, a mouse and / or a keyboard, etc.

[0074] Here, the image sensor may include an image processing module. Specifically, the image sensor may process still images or video obtained by an image sensor device (e.g., CMOS or CCD).

[0075] In addition, the image sensor can process a still image or video acquired through the image sensor device using an image recognition process (e.g., OCR, etc.) and / or an image processing module to extract necessary information and transmit the extracted information to a processor.

[0076] Additionally, the input component (121) can receive input from an external controller (e.g., mouse, keyboard, etc.) based on an interface module, and in this case, may include an external output device (e.g., speaker).

[0077] At this time, the interface module may be configured to include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0078] In addition, the external output device may include a display system that outputs various information related to an in-context learning-based vision inspection service as a graphic image.

[0079] Such a display system may be implemented by including at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0080] Meanwhile, the user computing device (110) including the above-described components may further perform at least some of the functional operations performed by the server computing system (130) described later.

[0081] -Server Computing System (130: Server Computing System)

[0082] The server computing system (130) can perform a series of processes to provide in-context learning-based vision inspection services.

[0083] Specifically, in an embodiment, the server computing system (130) can provide the in-context learning-based vision inspection service by exchanging data necessary to enable the in-context learning-based vision inspection service process to be run on an external device, such as a user computing device (110), with said external device.

[0084] More specifically, in an embodiment, the server computing system (130) can provide an environment in which an application can run on a user computing device (110).

[0085] To this end, the server computing system (130) may include an application program, data and / or instructions for the application to operate, and may transmit and receive various data based thereon with the external device.

[0086] Additionally, the server computing system (130) includes at least one processor (131) and memory (132).

[0087] Here, the processor (131) of the server computing system (130) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0088] In particular, depending on the embodiment, such a processor (131) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which are hardware technologies for implementing a specific digital circuit. A detailed description thereof is omitted by applying the description of the FPGA and ASIC mentioned above.

[0089] And the memory (132) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (132) may store data (133) and instructions (134) necessary for the processor (131) to perform functional operations, such as training an artificial intelligence model or performing vision inspection using in-context learning through an artificial intelligence model.

[0090] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate a plurality of computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include a plurality of computing devices connected to a network (170).

[0091] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer non-linear model as the machine learning model (140). Exemplary neural networks may include a feed-forward neural network, a deep neural network, a recurrent neural network, and a convolutional neural network.

[0092] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, data store) which is a storage for continuously storing and managing raw data that forms the basis of an in-context learning-based vision inspection service.

[0093] Such data stores may include various forms of data storage, ranging from file systems to cloud storage. For example, a data store may include at least one database among a relational database that uses a structured query language (SQL) to define and manipulate data, a NoSQL database designed for flexibility and scalability to process unstructured and semi-structured data, a data warehouse optimized for querying and analysis by centralizing large volumes of data from multiple sources as a system used for reporting and data analysis, a data warehouse that stores large volumes of raw data in basic formats such as structured data, semi-structured data, and unstructured data, and a local storage device or Network Attached Storage (NAS) that stores data in files in a format generally accessible by a computer operating system.

[0094] - Training Computing System (150: Training Computing System)

[0095] The training computing system (150) includes at least one processor (151) and memory (152).

[0096] Here, the processor (151) of the training computing system (150) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0097] In particular, depending on the embodiment, this processor (151) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which are hardware technologies for implementing a specific digital circuit. A detailed description thereof is omitted by applying the description of the FPGA and ASIC mentioned above.

[0098] And the memory (152) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0099] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as back propagation of error (according to the framework illustrated in FIG. 3).

[0100] For example, such a model trainer (160) can perform updates to one or more parameters of a machine learning model (120 and / or 140) for an in-context learning-based vision inspection service in a backpropagation manner based on a defined loss function.

[0101] In some embodiments, performing backpropagation of the error may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight devaluation, dropout and / or knowledge distillation, etc.) to improve the generalization ability of the machine learning model (120 and / or 140) being trained.

[0102] Additionally, the model trainer (160) can train a machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data of different forms, such as images, audio samples and / or text, for example. Examples of image types that may be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images and / or various other forms of images.

[0103] These training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0104] And the model trainer (160) includes computer logic that is utilized to provide the desired function.

[0105] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one embodiment, the model trainer (160) may include a program file stored in a storage device, be loaded into memory (152), and be executed by one or more processors (151). In another embodiment, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0106] Network (170) includes, but is not limited to, 3GPP (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth network, satellite broadcasting network, analog broadcasting network and / or DMB (Digital Multimedia Broadcasting) network.

[0107] Generally, communication through the network (170) can be performed using any type of wired and / or wireless connection through various communication protocols (e.g., TCP / IP, HTTP, SMTP and / or FTP, etc.), encodings or formats (e.g., HTML and / or XML, etc.), and / or protection schemes (e.g., VPN, Secure HTTP and / or SSL, etc.).

[0108] FIG. 2 illustrates an example of a block diagram of a computing device implementing an in-context learning-based vision inspection service according to one embodiment of the present disclosure.

[0109] Including FIG. 2, the computing device (100) included in the user computing device (110), server computing system (130), and training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing application (e.g., Detection, Classification and / or Segmentation, etc.), a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application and / or a chat-bot application, etc.

[0110] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and by storing and operating the trained artificial intelligence model, it may provide output data according to a predetermined input data (e.g., a query image).

[0111] Each application of the computing device (100) can communicate with a number of other components of the computing device (100), such as, for example, at least one sensor, a context manager, a device state component, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0112] FIG. 3 illustrates an example of a block diagram in another aspect of a computing device implementing an in-context learning-based vision inspection service according to one embodiment of the present disclosure.

[0113] Referring to FIG. 3, the computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application can communicate with a central intelligence layer. For example, applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application and / or a browser application, etc. In one embodiment, each application can communicate with the central intelligence layer (and a model stored therein) using an API (e.g., an API common across all applications).

[0114] The central intelligence layer may include a number of machine learning models. For example, as illustrated in FIG. 3, at least some of the machine learning models may be provided for each application and managed by the central intelligence layer. In another embodiment, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model for all applications. In some embodiments, the central intelligence layer may be included within the operating system of the computing device (200) or otherwise implemented.

[0115] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data store for the computing device (200). As illustrated in FIG. 3, the central device data layer can communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0116] The technology described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from said systems. It will be recognized that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, division of tasks, and functionality between and from components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components operating in combination. Databases and applications may be implemented in a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0117]

[0118] Meanwhile, a computing system (1000) according to one embodiment of the present disclosure may include at least one image encoder, an image-label encoder and / or an in-context learning sequence model.

[0119] [Image Encoder (IE)]

[0120] Specifically, an image encoder (IE) according to one embodiment of the present disclosure can receive a predetermined image (in the embodiment, a query image) and convert it into an embedding vector in a form that can be processed (interpreted) by an in-context learning sequence model to be described later.

[0121] Here, the query image according to the embodiment may refer to an actual image of the object to be inspected that is entered into the system to receive an inspection judgment.

[0122] That is, in the embodiment, a prompt data set consisting of a plurality of images and corresponding label pairs can serve as an example that provides a 'judgment criterion' for the model, and the query image can represent actual data that must perform a final judgment, such as whether it is a good product or a defective product, by applying the judgment criterion.

[0123] Returning to the above, the image encoder (IE) may include a model pre-trained with a large dataset, such as a Convolutional Neural Network (CNN) or a Vision Transformer, and may also include a model that has been fine-tuned to enhance feature extraction performance for specific inspection purposes as needed.

[0124] Through this, the image encoder (IE) can convert various forms of visual information into normalized data that a sequence model can understand, and provide a foundation for effectively extracting key features necessary for inference.

[0125]

[0126] [Image Label Encoder (ILE)]

[0127] Additionally, an image-label encoder (ILE) according to one embodiment of the present disclosure can perform the role of combining at least one prompt data, i.e., at least one image and a corresponding label pair, to convert it into a single embedding vector that can be processed (interpreted) by an in-context learning sequence model described below.

[0128] Here, the label according to the embodiment can express various types of good or defective products included in the image and various levels of precision in combination, without the need to be unified into a single form or precision.

[0129] For example, the label may include a combination of labels having different types and precision, such as an image label specifying the type for the entire matching image (e.g., good or defective), a bounding box label specifying the approximate location of a specific area within the image (e.g., defective area) using a rectangle and specifying the type for said specified area, and / or a segmentation label precisely defining the boundary of a specific area within the image (e.g., defective area) in pixel units and specifying the type for said defined area.

[0130] Returning to the point, the image-label encoder (ILE) can also be implemented to improve the parameter efficiency of the entire system by internally reusing the structure of the image encoder (IE).

[0131] Through this, the image-label encoder (ILE) can flexibly respond to unstructured data environments and, at the same time, generate the context of the rules for a task (i.e., inspection in the examples) using only a few examples, thereby playing a pivotal role in realizing in-context learning, which is the core of the present disclosure.

[0132]

[0133] [In-Context Learning Sequence Model (ICSM)]

[0134] In addition, an in-context learning sequence model (ICSM) according to one embodiment of the present disclosure can receive an embedding of a prompt dataset extracted from the image-label encoder (ILE) and an embedding of a query image extracted from the image encoder (IE) as input, and dynamically interpret the context of a given prompt to infer the label of the query image.

[0135] That is, the In Context Learning Sequence Model (ICSM) can learn and understand the context of the rules for a task (i.e., inspection in the embodiment) from multiple examples provided as prompts (i.e., multiple image-label pairs in the embodiment) without a separate parameter update process, and interpret the relationship with the query image based on this to output the final task execution results, such as classification, detection, and / or segmentation.

[0136] In other words, the in-context learning sequence model (ICSM) according to the embodiment is characterized by changing the decision rule based only on the content of the prompt without directly updating the model parameters, and can be implemented by utilizing an architecture such as a Transformer, a state space model, and / or a recurrent neural network.

[0137] In addition, the above In-Context Learning Sequence Model (ICSM) can be designed to improve computational efficiency by reusing intermediate result values ​​from the prompt processing process during inference.

[0138] Through this, the In-Context Learning Sequence Model (ICSM) can rapidly respond to new defects or environmental changes without separate retraining, and provide the versatility and flexibility to perform various inspection tasks with a single model.

[0139]

[0140] [Vision inspection method using in-context learning]

[0141] Hereinafter, a method for implementing an in-context learning-based vision inspection service according to one embodiment of the present disclosure, in which a computing system (1000) performs vision inspection on a query image by controlling the operation of a model through a prompt containing a few images and label examples without updating model parameters using in-context learning based on a sequence model, will be described in detail with reference to the attached drawings.

[0142] FIG. 4 illustrates an example of a flowchart for explaining a vision inspection method using in-context learning according to one embodiment of the present disclosure.

[0143] Referring to FIG. 4, a method for providing an in-context learning-based vision inspection service according to one embodiment of the present disclosure may include a step of performing model training (S101), a step of acquiring inspection request data (S103), a step of generating an embedding based on the acquired inspection request data (S105), a step of performing in-context inference based on the generated embedding (S107), and a step of providing output data based on the performed in-context inference (S109).

[0144] Specifically, a computing system (1000) according to one embodiment of the present disclosure can perform model learning. (S101)

[0145] Specifically, in an embodiment, the computing system (1000) can perform learning on an in-context learning sequence model (ICSM).

[0146] For reference, in-context learning can refer to a method of learning and inferring new tasks based on descriptions and / or example information within the context input to the model, without directly updating the model's parameters.

[0147] Here, context can refer to the previous input sequence information that the sequence model refers to when generating output.

[0148] FIG. 5 illustrates an example of a drawing for explaining in-context learning according to one embodiment of the present disclosure.

[0149] For example, referring to FIG. 5, the computing system (1000) can control the operation so that the model performs translation for a new word ('cheese') without a separate tuning process by providing the model with the task description 'Translate English to French:' and some translation examples ('sea otter => loutre de mer', etc.) as context.

[0150] More specifically, as an example, a computing system (1000) can perform pre-training in a manner that iteratively optimizes various vision inspection scenarios (episodes) for the entire model architecture so that the in-context learning sequence model (ICSM) learns how to perform the in-context learning described above based on a given prompt.

[0151] In other words, the computing system (1000) can perform pre-learning on the entire model architecture in a meta-learning manner that iteratively optimizes various vision inspection scenarios (episodes) to give prompt-based in-context learning capabilities to the in-context learning sequence model (ICSM).

[0152] At this time, the computing system (1000) implements the entire model architecture in a structure capable of end-to-end learning, and can integrally tune the parameters of the related components through gradient descent-based optimization.

[0153] Specifically, the computing system (1000) can acquire a predetermined prompt data set as training data.

[0154] Here, the prompt data set according to the embodiment may refer to data consisting of a plurality of image-label pairs (i.e., a prompt image and a corresponding correct label) provided (input) to the model to inform the rules of the vision inspection task currently to be performed.

[0155] These prompt datasets serve as 'context' and / or 'examples' for in-context learning, and depending on the context, the training set ( It may also be referred to as ).

[0156] Additionally, the computing system (1000) can additionally acquire a test data set paired with the prompt data set as training data.

[0157] Here, the test dataset according to the embodiment may include one or more image-label pairs (i.e., a test image and its corresponding ground truth label) for evaluating the inference performance of a model for a task defined through a prompt dataset, and depending on the context, a test set ( It can be referred to as ).

[0158] Additionally, the computing system (1000) comprises an in-context learning episode (E:) configured based on the prompt data set and test data set obtained as above. , By using ), the model's loss can be calculated according to the following [Equation 1] and the model parameters can be updated in a direction that minimizes it.

[0159] [Mathematical Formula 1]

[0160]

[0161] More specifically, the computing system (1000) has an objective function according to [Equation 1] In-Context Learning Sequence Model (ICSM) aiming to minimize the expected value You can update the parameters of.

[0162] In detail, the computing system (1000) can sample a predetermined in-context learning episode from the learning data.

[0163] At this time, the in-context learning episode may include a training set that serves as a prompt and a test set that is evaluated.

[0164] Additionally, the computing system (1000) is an in-context learning sequence model (ICSM) Using the training set of the corresponding in-context learning episode as the context, each test image included in the test set Prediction regarding It can be made to output.

[0165] And the computing system (1000) is the actual correct answer label of the prediction. Individual loss compared to It can produce.

[0166] In an embodiment, the computing system (1000) can calculate the individual losses as described above for all test data within an in-context learning episode and sum the calculated individual losses to calculate the total loss for the in-context learning episode.

[0167] Continuing, the computing system (1000) repeatedly performs the loss calculation process based on the in-context learning episode described above for various in-context learning episodes, and through an optimization algorithm such as gradient descent, an in-context learning sequence model (ICSM) in a direction that minimizes the expected value (average) of the calculated loss The parameters of can be updated gradually.

[0168] Meanwhile, according to an embodiment, the computing system (1000) may perform training on an image encoder (IE) and / or an image-label encoder (ILE).

[0169] Specifically, in the case of the image encoder (IE) and / or image-label encoder (ILE), it is possible to utilize a general-purpose model pre-trained with a large dataset, so training on it may not be essential, but fine-tuning to enhance performance for a specific inspection domain or end-to-end training with the in-context learning sequence model (ICSM) may be performed as needed.

[0170] That is, according to an embodiment, the computing system (1000) may optimize the parameters of all components together by performing the aforementioned episode-based meta-learning in the same way on the entire model architecture including the image encoder (IE), image-label encoder (ILE), and in-context learning sequence model (ICSM).

[0171] In this way, in the embodiment, the computing system (1000) can implement a general-purpose inspection model that can immediately respond with only a few examples without a separate retraining process when a new inspection task occurs in the future by pre-training the model on how to learn from prompts through various episode-based meta-learning.

[0172] FIG. 6 illustrates an example of a conceptual diagram for explaining a vision inspection method using in-context learning according to one embodiment of the present disclosure.

[0173] Additionally, with reference to FIG. 6, a computing system (1000) according to one embodiment of the present disclosure can perform a vision inspection task based on a newly given prompt and / or query image using a model that has been trained through the above-described step S101.

[0174] Specifically, a computing system (1000) according to one embodiment of the present disclosure can acquire inspection request data. (S103)

[0175] Specifically, in an embodiment, at least one processor of a computing system (1000) can access at least one in-memory data structure.

[0176] Here, the data structure may include at least one task prompt defining an example of an inspection rule and at least one query image being an inspection target to be determined according to the rule.

[0177] And at least one processor of the computing system (1000) can load the inspection request data from the at least one memory and perform preparation for a subsequent step.

[0178] Here, the inspection request data according to the embodiment may refer to a set of data input into the system to perform a specific vision inspection task.

[0179] In an embodiment, the inspection request data may include at least one 'Task Prompt' defining an example of an inspection rule and at least one 'Query Image' data which is an inspection target to be determined according to the rule.

[0180] At this time, the above 'task prompt' and 'query image' may refer to data used at the actual inference time that is distinct from the prompt data and / or test data used in the learning phase of S101 described above.

[0181] Specifically, the above task prompt may refer to an image-label pair that defines the content and criteria of the task the model must perform at the time of actual inference.

[0182] In addition, the above query image may refer to a target image to be actually judged (inspected) as good / defective, etc. by applying the criteria of the above task prompt.

[0183] Specifically, in the embodiment, the computing system (1000) can acquire (collect) inspection request data through various paths.

[0184] In an example, the computing system (1000) can obtain inspection request data through a user interface.

[0185] As a specific example, the computing system (1000) can obtain a task prompt by receiving an uploaded image and label information specified therefor based on user input based on the interface, and can obtain a query image, which is an image to be inspected (determined) through the user input.

[0186] In another embodiment, the computing system (1000) can obtain inspection request data by linking with an automated production line system.

[0187] As a specific example, the computing system (1000) can load a predefined task prompt for a specific production process from a storage.

[0188] For example, the computing system (1000) can set up a task prompt containing normal / defective examples of the 'A model motor' for the 'A model motor' production line by loading it from a storage such as a database or file server.

[0189] The task prompt loaded in this way can be continuously used as a criterion for inspection judgment while the production process is in progress.

[0190] Afterwards, the computing system (1000) can perform an inspection using a query image obtained in real time.

[0191] For example, when a certain condition (e.g., a condition that a product on a conveyor belt is accurately positioned at an inspection point) is met, a sensor of the process system can detect this and trigger a camera, and the triggered camera can take an image of the product and transmit it to a computing system (1000) as a query image.

[0192] Thus, the computing system (1000) can obtain inspection request data (i.e., task prompt and query image) through linkage with an automated production line.

[0193] In another embodiment, the computing system (1000) can obtain inspection request data in a batch processing manner by linking with a specific database.

[0194] As a specific example, a computing system (1000) may receive a specific task prompt and / or a list and / or path of multiple query images to which it is applied from a predetermined database (e.g., local storage, network file server and / or task management system, etc.), sequentially load each query image, and perform inspections according to the task prompt in batches.

[0195] Additionally, a computing system (1000) according to one embodiment of the present disclosure can generate an embedding based on acquired inspection request data. (S105)

[0196] Specifically, in the embodiment, the computing system (1000) can generate an embedding for each of the data within the inspection request data obtained through step S103.

[0197] That is, the computing system (1000) can perform data processing for each of at least one task prompt and query image included in the inspection request data to generate an embedding for each of the task prompt and query image.

[0198] At this time, since data processing for generating embeddings of the task prompt and query image can be performed independently, the computing system (1000) according to the embodiment can perform (implement) data processing for generating embeddings of the task prompt and query image sequentially or in parallel.

[0199] More specifically, in an embodiment, the computing system (1000) can 1) generate an embedding for a task prompt of inspection request data.

[0200] That is, in the embodiment, the computing system (1000) can generate an embedding for each of at least one task prompt within the inspection request data.

[0201] Specifically, the computing system (1000) can work in conjunction with an image-label encoder (ILE) to convert each image-label pair, each containing at least one task prompt in the inspection request data, into a single embedding vector in a form that can be interpreted by an in-context learning sequence model (ICSM).

[0202] In this process, the computing system (1000) can fuse the visual information of each image and the semantic information of each corresponding label, and the sequence of at least one transformed embedding vector (PE: hereinafter, task prompt embedding vector) can form a context for in-context learning.

[0203] That is, the computing system (1000) can obtain a context according to a sequence of at least one task prompt embedding vector (PE) (hereinafter, prompt embedding sequence).

[0204] At this time, according to the embodiment, when the computing system (1000) examines multiple query images for the same task prompt, it may increase inference efficiency by caching and reusing the context created as above.

[0205] Meanwhile, in the embodiment, the computing system (1000) can 2) generate an embedding for a query image of the inspection request data.

[0206] That is, in the embodiment, the computing system (1000) can generate an embedding for each of at least one query image within the inspection request data.

[0207] The following description is based on an example using a single query image for effective explanation, but is not limited thereto.

[0208] Specifically, the computing system (1000) can convert the query image contained in the inspection request data into an embedding vector (QE: hereinafter, query image embedding vector) in a form that can be interpreted by an in-context learning sequence model (ICSM).

[0209] Through this, the computing system (1000) can represent the unique visual features of the query image as compressed numerical data, and the feature extraction method used at this time applies the same method used when the image encoder (IE) was pre-trained, thereby maintaining consistency with the pre-trained feature space and enabling comparison in a consistent feature space.

[0210] Additionally, a computing system (1000) according to one embodiment of the present disclosure can perform in-context inference based on generated embeddings. (S107)

[0211] Specifically, in an embodiment, the computing system (1000) can perform in-context inference using an in-context learning sequence model (ICSM) based on at least one task prompt embedding vector (PE) and a query image embedding vector (QE) generated through step S105.

[0212] In other words, in the embodiment, the computing system (1000) can perform in-context inference using an in-context learning sequence model (ICSM) based on a context (context) and a query image embedding vector (QE) according to a sequence of at least one task prompt embedding vector (PE) generated through step S105 (i.e., prompt embedding sequence).

[0213] Here, in-context inference according to the embodiment may mean that an in-context learning sequence model (ICSM) dynamically interprets the rules of the current task from a given context (i.e., context according to the prompt embedding sequence) without directly updating its own parameters, and applies the interpreted rules to the embedding vector of a given query image (i.e., query image embedding vector (QE)) to infer the final inspection result.

[0214] In detail, in an embodiment, the computing system (1000) can input a prompt embedding sequence and a query image embedding vector (QE) into an in-context learning sequence model (ICSM).

[0215] At this time, the in-context learning sequence model (ICSM) according to the embodiment can dynamically interpret and learn the rules of the current task from a given context according to a prompt embedding sequence through internal operations such as an attention mechanism.

[0216] As an example, an in-context learning sequence model (ICSM) can instantly establish judgment criteria for a task by analyzing the difference in visual features between a 'normal (OK)' example and a 'bad (NG)' example provided as context, without separate parameter updates.

[0217] And the In-Context Learning Sequence Model (ICSM) can infer the label for the corresponding query image by applying the judgment criteria established as above to the query image embedding vector (QE).

[0218] Specifically, an In-Context Learning Sequence Model (ICSM) can output an inspection result for a query image (e.g., 'normal' or 'scratch defect') by comparing and analyzing which example within the context (e.g., normal or a specific type of defect) the features of the corresponding query image have the highest similarity to.

[0219] In this way, the computing system (1000) in the embodiment can immediately define and update the behavior of the in-context learning sequence model (ICSM) through a small number of examples given at the time of actual inference without updating the parameters of the pre-trained model.

[0220] Thus, the computing system (1000) can implement vision inspection that responds very quickly and flexibly to changes in the inspection environment, such as new products, new types of defects and / or changes in lighting, without a high-cost relearning process.

[0221] Additionally, a computing system (1000) according to one embodiment of the present disclosure may provide output data based on in-context inference performed. (S109)

[0222] Specifically, in the embodiment, the computing system (1000) can provide output data (hereinafter, inference output data) based on in-context inference performed in step S107 in a predetermined manner.

[0223] Here, the inference output data according to the embodiment may include the final inspection result determined for the query image based on the criteria of the given task prompt through in-context inference.

[0224] For example, the inference output data may include image labels for the entire query image (e.g., 'good' or 'bad'), confidence scores or probability values ​​for each type of defect, bounding box coordinates indicating the location of a specific area within the image (e.g., a defective area), and / or segmentation mask data defining the boundaries of the area in pixels.

[0225] More specifically, as an example, the computing system (1000) can process and provide inference output data in a form suitable for the purpose of the task.

[0226] At this time, in the embodiment, the computing system (1000) can input the inference output data generated as above into at least one subsequent processing component.

[0227] Here, the subsequent processing component may refer to a functional unit that receives data generated by at least one processor and performs a predetermined subsequent processing, and this may be implemented in hardware, software and / or a combination thereof.

[0228] For example, the subsequent processing component may include a user interface creation component for manifesting inference output data through a user interface, a data storage component for storing inference output data in a specific format in a database, and / or a visualization processing component for visualizing inference output data.

[0229] At this time, the subsequent processing component according to the embodiment can process and manifest the inference output data in a predetermined form (e.g., a form such as a table and / or a graph displayed through a graphical user interface (GUI) of a user computing device (110).

[0230] Returning to the point, as a specific example, the computing system (1000) can provide the inference output data after processing, such as converting it into a specific form (e.g., text, image and / or audio, etc.) or converting it into a standardized data format such as JSON (JavaScript Object Notation) or XML (eXtensible Markup Language) for interoperability with other systems.

[0231] In this case, in the embodiment, the computing system (1000) can provide the above-mentioned inference output data to various service applications and / or systems so that it can be implemented (utilized) in different ways according to different service scenarios.

[0232] For example, the computing system (1000) can transmit inference output data determined as ‘scratch defects’, etc., to the factory automation system according to the manufacturing process and smart factory service scenario.

[0233] Accordingly, the factory automation system that receives the inference output data can perform actions to improve the automation of the manufacturing process and yield, such as controlling a robot arm to immediately remove defective products from the production line or recording the type of defect and the time of occurrence in a quality control database to utilize for production process improvement.

[0234] Therefore, the computing system (1000) can rapidly adapt to newly occurring types of defects and implement an intelligent factory automation and quality control system.

[0235] As another example, the computing system (1000) can transmit inference output data determined as ‘packaging defects’, etc., to the logistics management system according to the logistics / distribution service scenario.

[0236] Through this, the logistics management system can perform actions to improve logistics processing efficiency and manage quality, such as automatically sorting the product to a repackaging line or sending warning notifications to a manager.

[0237] In this way, the computing system (1000) can ensure the continuity and accuracy of the logistics process by enabling the rapid introduction of new inspection rules with only a few examples, even in an environment where labels or packaging formats are frequently changed.

[0238] As another example, the computing system (1000) can transmit inference output data inferred as 'suspected lesion', etc., to a medical imaging information system (PACS) according to a healthcare / medical image analysis service scenario.

[0239] In this way, the medical imaging information system can assist medical professionals in making rapid and accurate diagnoses by performing pre-processing for computer-aided diagnosis, such as visually highlighting specific areas of a medical image and increasing the reading priority of the image based on the received inference output data.

[0240] Accordingly, the computing system (1000) can effectively assist the medical staff in their diagnosis by responding quickly with only prompt-based supplementation without large-scale relearning, even when environmental changes occur, such as when a new type of lesion appears or the settings of medical equipment are changed.

[0241] As described above, a computing system (1000) according to one embodiment of the present disclosure can rapidly define or update the operation of an inspection model with only a few examples (prompts) without the process of retraining the entire model by introducing the in-context learning capability of a sequence model into the field of vision inspection.

[0242] Thus, the computing system (1000) can secure high flexibility to respond quickly to environmental changes such as the occurrence of new types of defects or changes in processes, and at the same time, implement versatility to process various inspection tasks (e.g., classification, outlier detection and / or segmentation) as a single model, thereby drastically reducing development and maintenance costs compared to existing technology and maximizing the efficiency and speed of vision inspection in various industrial fields.

[0243] Thus, a computing system (1000) according to one embodiment of the present disclosure can create high technical synergy by organically combining with various next-generation technologies.

[0244] For example, a computing system (1000) can operate as an intelligent agent (AI Agent) that can immediately modify its operation rules with only a few examples (prompts).

[0245] That is, the computing system (1000) can interpret prompts without a separate relearning process to quickly absorb new knowledge and thereby implement the characteristics of an intelligent agent that makes flexible decisions.

[0246] As another example, the computing system (1000) can implement its internal architecture as a chain of experts.

[0247] In detail, the computing system (1000) may have a structure in which an image encoder (IE) capable of functioning as a visual feature extraction expert, an image-label encoder (ILE) capable of functioning as a prompt processing expert, and an in-context learning sequence model (ICSM) capable of functioning as a comprehensive inference expert collaborate sequentially to solve complex vision inspection problems.

[0248] As another example, a computing system (1000) can perform stepwise reasoning similar to a chain of thoughts in the reasoning process.

[0249] That is, the computing system (1000) can construct a context by interpreting examples provided as prompts step-by-step through an in-context learning sequence model (ICSM), and perform a logical reasoning process leading to a final judgment based on this.

[0250] As another example, through the features described above, the computing system (1000) can function as a core technology of a Smart Factory.

[0251] Specifically, the computing system (1000) can contribute to maximizing productivity and flexibility in a smart factory environment where production lines or items change frequently, by immediately redefining inspection tasks with only prompt modifications.

[0252] In conclusion, the computing system (1000) according to one embodiment of the present disclosure can easily create synergy with next-generation technologies such as the intelligent prompt response capability of an AI agent, the role division capability according to the Chain of Expert structure, the context interpretation capability according to the Chain of Thought process, and the ability to respond to environmental changes of a Smart Factory, through the core feature of an in-context vision inspection framework that can flexibly change the operation of a model with only a few examples (prompts).

[0253] Ultimately, a computing system (1000) according to one embodiment of the present disclosure can organically combine the technologies described above to overcome the limitations of existing vision inspection and provide an integrated solution for a next-generation intelligent automation system.

[0254]

[0255] The embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present disclosure, and vice versa.

[0256] The specific embodiments described in this disclosure are examples and do not limit the scope of this disclosure in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of this disclosure.

[0257] Furthermore, although the detailed description of the present disclosure has been explained with reference to preferred embodiments of the present disclosure, those skilled in the art or those with ordinary knowledge in the art will understand that the present disclosure can be modified and changed in various ways without departing from the spirit and technical scope of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the present disclosure should not be limited to the contents described in the detailed description of the specification but should be determined by the claims.

[0258]

[0259] The present disclosure relates to a vision inspection method and system using in-context learning, and since it is applicable to the artificial intelligence industry, it has industrial applicability.

Claims

1. In a method executed by a computer, A step in which at least one processor of the above computer accesses at least one memory; - wherein the at least one memory includes a memory containing at least one task prompt defining an example of a check rule and a memory containing at least one query image which is a check target to be determined according to the rule, and The step of the at least one processor loading the at least one task prompt and the at least one query image from the at least one memory; The above at least one processor generates an embedding for the loaded at least one task prompt and an embedding for the at least one query image; The step of the above at least one processor inputting the embeddings for the generated at least one task prompt and the embeddings for the at least one query image into at least one artificial intelligence model that performs in-context learning; The above at least one processor interprets the context according to the embedding for the at least one task prompt through the at least one artificial intelligence model and infers the label of the at least one query image; and A method comprising the step of at least one processor manifesting output data according to the inference.

2. In Paragraph 1, The above in-context learning is, A method characterized by learning and inferring a new task based on the context according to a given prompt.

3. In Paragraph 1, The above task prompt is, It refers to an image-label pair that defines the content and criteria of a task to be performed by at least one artificial intelligence model at the time of inference, and The above query image is, A method for indicating a target image to be inspected by applying the criteria of the above task prompt.

4. In Paragraph 1, A method further comprising the step of the above-mentioned at least one processor collecting the above-mentioned at least one task prompt and the above-mentioned at least one query image by linking with at least one of a user interface or an automated production line system.

5. In Paragraph 1, The step of generating the above embedding is, A method comprising the step of performing data processing to generate an embedding for the above task prompt and data processing to generate an embedding for the above query image sequentially or in parallel.

6. In Paragraph 5, The step of generating the above embedding is, A method comprising the step of generating an embedding for the task prompt through an Image Label Encoder that converts an image of the prompt data and a label corresponding to the image into a single embedding vector that can be processed by at least one artificial intelligence model.

7. In Paragraph 6, The step of generating the above embedding is, A method further comprising the step of generating an embedding for the query image through an image encoder that converts a given image into an embedding vector that can be processed by at least one artificial intelligence model.

8. In Paragraph 1, The step of inferring the label of the above query image is, A method comprising the step of dynamically interpreting the rules of the current task based on the context according to a prompt embedding sequence, which is a list of embeddings for the task prompt.

9. In Paragraph 8, The step of inferring the label of the above query image is, A method further comprising the step of inferring the label of the query image by applying the rules of the dynamically interpreted current task to the embeddings of the query image.

10. In Paragraph 1, The step of implementing output data based on the above inference is, A method comprising the step of processing the above output data into a form suitable for the purpose of the task.

11. In Paragraph 10, The step of implementing output data based on the above inference is, A method further comprising the step of expressing the above output data in different ways according to various service scenarios.

12. In Paragraph 1, The step of implementing output data based on the above inference is, A step of ingesting the above output data into at least one subsequent processing component, and A method comprising the step of controlling at least one subsequent processing component to express the output data through at least one interface.

13. In Paragraph 1, A method comprising the step of the above-mentioned at least one processor training the above-mentioned at least one artificial intelligence model.

14. In Paragraph 13, The step of training at least one artificial intelligence model is, A method comprising the step of iteratively optimizing various vision inspection scenarios for the entire architecture of at least one artificial intelligence model.

15. In Paragraph 14, The step of training at least one artificial intelligence model is, A prompt dataset containing multiple image-label pairs ( A test dataset including ) and a plurality of image-label pairs paired with the prompt dataset ( In-context learning episodes based on ) , A step of calculating the loss of at least one artificial intelligence model based on ), and A method further comprising the step of updating the parameters of at least one artificial intelligence model in a direction that minimizes the loss calculated above.

16. At least one processor; and It includes at least one memory that stores at least one instruction that performs the following when executed by the above at least one processor; and The above at least one instruction is, The above-mentioned at least one processor has the step of accessing the above-mentioned at least one memory, - wherein the above-mentioned at least one memory includes a memory containing at least one task prompt defining an example of a check rule and a memory containing at least one query image which is a check target to be determined according to the rule, and The above at least one processor loads the at least one task prompt and the at least one query image from the at least one memory, and The above at least one processor generates an embedding for the loaded at least one task prompt and an embedding for the at least one query image, and The above at least one processor inputs the embeddings for the generated at least one task prompt and the embeddings for the at least one query image into at least one artificial intelligence model that performs in-context learning, and The above at least one processor interprets the context according to the embedding of the at least one task prompt through the at least one artificial intelligence model and infers the label of the at least one query image, and A system comprising at least one processor including instructions that perform the step of manifesting output data according to the inference.

17. In Paragraph 16, A plurality of neurons comprising an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; and at least one routing network that controls the data flow between the plurality of neurons; A system further comprising a Field Programmable Gate Array (FPGA) implementation for a predetermined artificial neural network, wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to establish a transmission path for the weights.

18. In Paragraph 16, A plurality of neurons organized into an array comprising at least one register, at least one microprocessor, and at least one input; and a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; comprising A system further comprising an Application Specific Integrated Circuit (ASIC) for a predetermined artificial neural network, wherein each of the plurality of neurons is connected to at least one other neuron through any one of the plurality of synaptic circuits.

Images