Method and system for analyzing image on basis of separate interpretation for spatial feature and temporal feature

Abstract

An embodiment provides a method comprising the steps of: receiving an image for analysis and storing the image in at least one memory; loading the image for analysis from the at least one memory; generating, by at least one processor, an analysis result for the image for analysis by using at least one artificial intelligence model using the image for analysis as an input thereto, wherein the at least one artificial intelligence model is pre-trained to output the analysis result for the image by performing a predetermined operation using a disentangling mask for controlling the interaction between a spatial token including spatial information on the image and a non-spatial token including temporal information or language information; and ingesting the analysis result into at least one subsequent processing component.

Description

Image analysis method and system based on separate interpretation of spatial and temporal features

[0001] The present disclosure relates to an image analysis method and system based on separate analysis of spatial features and temporal features, and more specifically, to an image analysis method and system based on separate analysis of spatial features and temporal features that can increase the accuracy and reliability of spatial information processing.

[0002] Recently, in the field of artificial intelligence, Large-Scale Language Models (LLMs) and Visual-Language Models (VLMs) have demonstrated innovative performance in various natural language processing and multimodal tasks, such as text generation, translation, and image recognition. However, these models inherently have fundamental limitations in their ability to process spatial information and represent it linguistically accurately.

[0003] The spatial interpretation limitations of LLMs manifest specifically in two aspects. First is the problem of converting coordinates into linguistic representations. LLMs struggle to accurately convert numerical spatial information, such as spatial coordinates (x, y, z) or bounding boxes, into natural language positional representations that humans can understand. For example, GPT-4V exhibits the problem of outputting coordinate values ​​as they are, which clearly demonstrates the lack of ability in LLMs to interpret spatial relationships and express them in language.

[0004] Second is the issue of spatiotemporal hallucination. When analyzing video or complex image scenes, VLM suffers from the problem of relying on "spurious patterns," which incorrectly link spatial features (scenes) with temporal features (actions). For example, there have been reported cases where the system generated an incorrect scene-action combination of "a person skiing in the snow" in footage of a snow-covered slope, even though no skiers were present. This hallucination is a critical issue that undermines the accuracy and consistency of analysis results.

[0005] These limitations in spatial information interpretation are severely restricting the application scope of LLMs. In particular, they have a decisive impact on the utilization and reliability of models in industrial sectors where accurate processing of spatial information is essential, such as autonomous driving, robot control, medical image analysis, and architectural design and construction management. Therefore, technologies that accurately interpret spatial information extracted from images or videos without distortion and reliably represent it hold significant industrial value.

[0006] Previously, methods such as introducing structured spatial representations or training spatial information processing networks and language models separately have been proposed. However, these methods still fail to perfectly resolve the mapping between spatial information and text representations, and struggle particularly with handling unstructured spatial relationships or complex spatiotemporal confusion. Therefore, a new technical approach is required to fundamentally improve the accuracy and reliability of spatial interpretation in LLM by clearly separating and controlling spatial, temporal, and linguistic features.

[0007] According to various embodiments of the present disclosure, the present invention aims to provide an image analysis method and system based on separate analysis of spatial and temporal features that can mitigate the halucination phenomenon caused by inaccurate interpretation of spatial positional relationships during image data analysis and improve the accuracy and consistency of analysis results.

[0008] According to various embodiments of the present disclosure, the present invention aims to provide an image analysis method and system based on separate interpretation of spatial and temporal features that can reduce the error rate of complex context inference by sensitively reflecting the spatial positional relationship between tokens (patches) within an image through token rotation-based encoding when performing position encoding included in an input sequence.

[0009] However, the technical problems that the various embodiments of the present disclosure aim to solve are not limited to the technical problems described above, and other technical problems may exist.

[0010] One embodiment is,

[0011] A method performed by a computing device comprises the steps of: receiving an image for analysis and storing it in at least one memory; loading the image for analysis from the at least one memory; generating an analysis result for the image for analysis using at least one artificial intelligence model that takes the image for analysis as input by at least one processor, wherein the at least one artificial intelligence model is pre-trained to output an analysis result for the image by performing a predetermined operation using a disentangling mask for controlling the interaction between a spatial token containing spatial information for the image and a non-spatial token containing temporal information or linguistic information; and inputting the analysis result to at least one subsequent processing component.

[0012] In another aspect, the analysis result may include at least one of a natural language description including spatial-temporal relationships and context for the image for analysis and information regarding object coordinates.

[0013] In another aspect, the method may further include the step of the at least one subsequent processing component using the analysis result to provide at least one service among autonomous driving, robot control, medical image analysis, architectural design, and construction management.

[0014] In another aspect, the method may further include the step of the at least one subsequent processing component manifesting the analysis result through at least one user interface.

[0015] In another aspect, the step of expressing the analysis result through at least one user interface may include providing an image through the user interface in which different visual effects are applied to a plurality of regions of an image related to the analysis result, based on a weight matrix calculated through a predetermined operation using the separation mask for the spatial token and the non-spatial token.

[0016] In another aspect, the at least one artificial intelligence model can be pre-trained by applying different attention mechanisms to the spatial token and the non-spatial token, respectively, using the separation mask.

[0017] In another aspect, the above-mentioned at least one artificial intelligence model may apply different attention mechanisms to the spatial token and the non-spatial token, respectively, by applying bidirectional attention to the spatial token and unidirectional attention to the non-spatial token.

[0018] In another aspect, the at least one artificial intelligence model is pre-trained by a learning method by the at least one processor, and the learning method may include the steps of: extracting a vision feature map from a training image; generating a global token, a spatial token, and a temporal token for the training image using the vision feature map; generating a text token from a text input; constructing an input sequence including the global token, the spatial token, the temporal token, and the text token; encoding position information by applying a harmonic rotation position embedding to each token of the input sequence; performing an attention operation with the input sequence encoded with position information as input, applying the separation mask to control the interaction between the spatial token and the non-spatial token; generating an output result by performing a non-linear transformation on the result of the attention operation; and updating the parameters of the at least one artificial intelligence model so as to minimize the loss based on the output result and the ground truth data.

[0019] In another aspect, the attention operation with the separation mask applied may include a bidirectional attention operation for the spatial token, a unidirectional attention operation for the time token, and a bidirectional attention operation for the text token.

[0020] In another aspect, the attention operation may include an operation that optionally restricts the interaction in which the space token looks up the time token or the time token looks up the space token.

[0021] In another aspect, the at least one processor may generate the spatial token by performing spatial pooling on the vision feature map, and generate the temporal token by performing frame difference and temporal pooling on the vision feature map.

[0022] In another aspect, the step of encoding location information for each token of the input sequence may include, for each token in the input sequence, assigning a balanced location identifier based on a local order and a unique location identifier based on a global order; determining a rotation angle for each token based on the balanced location identifier and the unique location identifier; and encoding location information for each token by performing a rotation operation on each token in the input sequence using the rotation angle determined for each token.

[0023] In another aspect, the step of generating the output result may include the step of generating the output result by performing a non-linear transformation using a multilayer perceptron (MLP) on the attention operation result.

[0024] In another aspect, the method further comprises the steps of generating a spatial descriptive token corresponding to information specifying a predetermined space for the training image, generating a language token including language information describing the predetermined space, and generating a contextual spatial token by performing an operation to fuse spatial meaning through the interaction of the spatial descriptive token and the language token, and the step of configuring the input sequence may include the step of configuring an input sequence further including the global token, spatial token, time token, and text token and the contextual spatial token.

[0025] One embodiment is,

[0026] A system is provided comprising at least one memory and at least one processor that reads at least one instruction stored in said memory and executes an image analysis method based on the separation and interpretation of spatial features and temporal features, wherein the at least one instruction comprises the steps of receiving an image for analysis and storing it in at least one memory; loading the image for analysis from said at least one memory; generating an analysis result for said image for analysis using at least one artificial intelligence model that takes the image for analysis as input by the at least one processor, wherein the at least one artificial intelligence model is pre-trained to output an analysis result for said image by performing a predetermined operation using a disentangling mask for controlling the interaction between a spatial token containing spatial information about an image and a non-spatial token containing temporal information or linguistic information; and inputting the analysis result to at least one subsequent processing component.

[0027] In another aspect, the system may further include a Field Programmable Gate Array (FPGA) implementation for a predetermined artificial neural network comprising a plurality of neurons arranged in an array including at least one register, at least one programmable logic, and at least one input interface, a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons, and at least one routing network that controls the data flow between the plurality of neurons, wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to establish a transmission path for the weights.

[0028] In another aspect, the system may further include a plurality of neurons organized into an array comprising at least one register, at least one microprocessor, and at least one input, and a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons, wherein each of the plurality of neurons may further include an Application Specific Integrated Circuit (ASIC) for a predetermined artificial neural network connected to at least one other neuron through any one of the plurality of synapse circuits.

[0029] According to various embodiments of the present disclosure, errors in analysis inference (illusion) caused by inaccurate interpretation of spatial positional relationships during the analysis of high-resolution images or image patch sequences can be effectively mitigated, and the accuracy and consistency of sequence data analysis can be improved.

[0030] According to various embodiments of the present disclosure, by performing a RoPE-based rotation operation that sensitively reflects the relative distance between each token included in an input sequence, a large language model can accurately identify complex spatiotemporal relationships within the sequence and significantly reduce errors (illusion) in analytical reasoning.

[0031] Since the various embodiments of the present disclosure are based on the separate analysis of spatial features and the corresponding temporal flow, they can be flexibly applied with high scalability to various domains and environments dealing with complex spatiotemporal data, such as video analysis, medical image analysis (3D / 4D), and autonomous driving.

[0032] According to various embodiments of the present disclosure, by generating a response output based on a contextual space token created by fusing information specifying a specific space for an image with linguistic features describing the same specific space, precise inference without error is possible even for complex linguistic instructions combining temporal elements and spatial locations.

[0033] However, the effects obtainable through the various embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly understood from the description below.

[0034] FIG. 1 illustrates an example of a block diagram of a computing system implementing an image analysis service based on separate analysis of spatial features and temporal features according to one embodiment.

[0035] FIG. 2 briefly illustrates the structure of a neuromorphic circuit that may be included in a processor according to one embodiment.

[0036] FIG. 3 is a block diagram of a computing device implementing an image analysis service based on separate analysis of spatial features and temporal features according to one embodiment.

[0037] FIG. 4 is a block diagram of a computing device implementing an image analysis service based on separate analysis of spatial features and temporal features according to another embodiment.

[0038] FIG. 5 is a block diagram of a computing device implementing an image analysis service based on separate analysis of spatial features and temporal features according to another embodiment.

[0039] FIG. 6 illustrates an exemplary configuration of an analysis architecture that performs image analysis based on separate analysis of spatial features and temporal features according to one embodiment.

[0040] FIG. 7 is intended to illustrate a method for encoding location information for an input sequence of a large language model according to one embodiment.

[0041] FIG. 8 is a block diagram illustrating the configuration of a space-language mapping module according to one embodiment.

[0042] FIG. 9 is intended to explain the function of a space-language mapping operation module according to one embodiment.

[0043] FIG. 10 is a flowchart of an image analysis method according to one embodiment.

[0044] FIG. 11 is a flowchart of a learning method for at least one artificial intelligence model used in the method of FIG. 10.

[0045] FIG. 12 is a flowchart illustrating the steps that may be included in the step of encoding location information of the method of FIG. 11.

[0046] FIG. 13 is a flowchart illustrating the steps for generating contextual space tokens through vision-language mapping that the method of FIG. 11 may further include.

[0047] As various modifications can be made to the various embodiments of the present disclosure, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the various embodiments of the present disclosure, and the methods for achieving them, will become clear by referring to the embodiments described in detail below together with the drawings. However, the various embodiments of the present disclosure are not limited to the embodiments disclosed below but can be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Also, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Additionally, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the various embodiments of the present disclosure are not necessarily limited to those depicted.

[0048] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0049]

[0050] - Image analysis system using an artificial intelligence model (1000)

[0051] An image analysis system (1000) according to one embodiment can solve the long-term dependency problem that occurs when a transformer-based model processes long sequence image data (e.g., long-duration video) and improve analysis accuracy and consistency. The system (1000) according to one embodiment aims to overcome the limitation that the encoding method for tokens included in existing location history sequences may cause errors in analysis inference (hallucinations) because it fails to effectively reflect complex spatiotemporal relationships.

[0052] The system (1000) can perform a Disentangled Spatio-Temporal (DST) attention operation to efficiently analyze input sequence data input into a language model. The DST attention operation can improve the accuracy and consistency of the analysis by separating and processing the spatial and temporal features of the input sequence individually, thereby preventing unnecessary data entanglement between temporal and spatial information.

[0053] Additionally, the system (1000) may utilize a structured positional encoding technique based on RoPE (Rotary Positional Embedding). This separates and interprets input sequence data into spatial and temporal features, and for the input sequence, positional information of each token is assigned in two dimensions: a balanced positional identifier (based on local order) and a unique positional identifier (based on global order). By combining these two identifiers to determine the rotation angle for the token and performing a rotation operation that applies this to the token, the relative distance between tokens can be sensitively reflected even in long sequences.

[0054] As a result, the system (1000) can more clearly analyze and encode the spatiotemporal features of complexly intertwined images to strengthen the inference basis of the language model. This effectively reduces the halucination phenomenon that occurs when scene and behavior information are inconsistent, thereby lowering the error rate of analysis inference and enabling consistent and reliable results in various sequence data analysis fields, such as video analysis.

[0055] FIG. 1 illustrates an example of a block diagram of a computing system (1000) that implements an image analysis service based on separate analysis of spatial features and temporal features according to one embodiment.

[0056] Referring to FIG. 1, a computing system (1000) implementing an image analysis service based on separate analysis of spatial features and temporal features according to one embodiment includes a user computing device (110), a server computing system (130), and a training computing system (150), and the devices can communicate through a network (170).

[0057] An image analysis method based on separate analysis of spatial features and temporal features according to one embodiment may be implemented and provided locally by a user computing device (110), implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or implemented and provided by the user computing device (110) and the server computing system (130) in conjunction with each other.

[0058] In this embodiment, the user computing device (110) and / or the server computing system (130) can train a machine learning model (120 and / or 140) through interaction with a training computing system (150) that is communicatedly connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0059] And at this time, the artificial intelligence model can be 1) trained directly locally by a user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training and learning techniques. It may also be implemented by transmitting the artificial intelligence model trained by the training computing system (150) to the user computing device (110) and / or the server computing system (130) through the network (170) to provide / update it.

[0060] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110).

[0061]

[0062] - User Computing Device (110: User Computing Device)

[0063] The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a PDA (personal digital assistants), a PMP (portable multimedia player), a desktop, a wearable device, an embedded computing device, a tablet PC, an augmented reality (VR) device, and / or a virtual reality (AR) device.

[0064] The user computing device (110) may include at least one processor (111) and memory (112). Here, the processor (111) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0065] In particular, according to the embodiment, this processor (111) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which is a hardware technology for implementing a certain digital circuit.

[0066] Here, a field programmable gate array (FPGA) can refer to a flexible digital circuit that is programmable according to user needs.

[0067] As an example, a field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results or state information of operations to support synchronized operation of the FPGA, programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits configurable according to user needs, and an input interface that receives signals from external devices or sensors and transmits them to internal circuits as a channel for receiving data from outside the FPGA.

[0068] Through the combination of the above components, a field-programmable gate array implementation can provide flexible and various types of digital circuits.

[0069] Meanwhile, an Application-Specific Integrated Circuit (ASIC) can refer to a custom integrated circuit that is fixedly designed to perform a specific use or function.

[0070] As an example, the application-dedicated integrated circuit may include a register, which is a small memory device for temporarily storing and managing data and supports the rapid processing of ASIC operations by storing intermediate calculation results or state information; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and coordinates the operation of the entire system by performing various operations or generating control signals when necessary; and an input block, which is an interface for receiving data from the outside, which receives data to be processed by the ASIC and transmits it internally, and receives various input data through connections with sensors or external devices.

[0071] Through the combination of the components mentioned above, an application-specific integrated circuit can perform specific purpose tasks in an optimized manner.

[0072] For example, ASICs can have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits.

[0073] FIG. 2 briefly illustrates the structure of a neuromorphic circuit (300) that may be included in a processor (111, 131, 151) according to one embodiment.

[0074] Referring to FIG. 2, for example, a neuromorphic circuit (300) may include a plurality of presynaptic neuron circuits (310), a plurality of presynaptic lines (311) extending laterally from the plurality of presynaptic neuron circuits (310), a plurality of postsynaptic neuron circuits (320), a plurality of postsynaptic lines (321) extending longitudinally from the plurality of postsynaptic neuron circuits (320), and a plurality of synaptic circuits (330) provided at the intersection of the plurality of presynaptic lines (311) and the plurality of postsynaptic lines (321).

[0075] A plurality of free synaptic neuron circuits (310) can transmit signals input from the outside in the form of electrical signals to a plurality of synaptic circuits (330) through a plurality of free synaptic lines (311).

[0076] Additionally, a plurality of post-synaptic neuron circuits (320) can receive electrical signals from a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0077] Furthermore, multiple post-synaptic neuron circuits (320) may transmit electrical signals to multiple synaptic circuits (330) through multiple post-synaptic lines (321).

[0078] A plurality of synapse circuits (330) can store weights included in layers constituting a neural network system implemented by a neuromorphic circuit (300) and perform a predetermined operation based on the weights and input data.

[0079] For example, each of the plurality of synaptic circuits (330) may include a resistive memory cell having a variable resistance. In this case, the resistance value of the plurality of synaptic circuits (330) changes by a voltage applied through the plurality of presynaptic neuron circuits (310) or the plurality of postsynaptic neuron circuits (320), and can store weight data according to this resistance change.

[0080] The neuromorphic circuit (300) is formed by mimicking the structure of neurons and synapses, which are essential elements of the human brain. When a deep neural network (DNN) is realized using the neuromorphic circuit (300), the data processing speed can be improved and power consumption can be reduced compared to when the existing von Neumann structure is utilized.

[0081] The memory (112) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs memory storage functions on the internet. This memory (112) may store data (113) and instructions (114) necessary for the at least one processor (111) to perform functional operations such as training an artificial intelligence model or performing image analysis through an artificial intelligence model.

[0082] In one embodiment, the user computing device (110) can store at least one machine learning model (120).

[0083] For example, the machine learning model (120) may be various machine learning models, such as multiple neural networks (e.g., deep neural networks) for performing an image analysis method based on separate interpretation of spatial features and temporal features, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0084] For example, machine learning models may include linear regression, decision trees, random forests, gradient-boosting pre-trained language models or / and deep learning models. And neural networks may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or / and other forms of neural networks.

[0085] Additionally, according to various embodiments, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process for an image analysis method based on separate analysis of spatial features and temporal features through a large language model (LLM).

[0086] In one embodiment, a user computing device (110) receives at least one machine learning model (120) from a server computing system (130) through a network (170), stores it in memory (112), and then executes the stored machine learning model (120) through a processor (111) to perform an operation for image analysis based on separate interpretation of spatial features and temporal features.

[0087] In another embodiment, the server computing system (130) includes at least one machine learning model (140) and performs operations through the machine learning model (140), and can provide an image analysis service based on separate analysis of spatial features and temporal features to the user by communicating with the user computing device (110) and related data.

[0088] For example, a user computing device (110) can perform an image analysis service based on separate analysis of spatial and temporal features in such a way that a server computing system (130) provides an output for the user's input using a machine learning model (140) via the web.

[0089] Additionally, the artificial intelligence model can be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0090] Additionally, the user computing device (110) may include at least one input component (121) for detecting user input. For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) for detecting a touch of a user input medium (e.g., a finger or a stylus), an image sensor for detecting user motion input, a microphone for detecting user voice input, a button, a mouse and / or a keyboard, etc. Additionally, the user input component (121) may include an interface and an external controller when receiving input to an external controller (e.g., a mouse and / or a keyboard, etc.) through an interface.

[0091]

[0092] -Server Computing System (130: Server Computing System)

[0093] The server computing system (130) can perform a series of processes to provide an image analysis service based on separate interpretation of spatial features and temporal features.

[0094] Specifically, in an embodiment, the server computing system (130) can provide an image analysis service based on separate interpretation of spatial features and temporal features by exchanging data necessary to enable an image analysis service process based on separate interpretation of spatial features and temporal features to be driven from an external device such as a user computing device (110).

[0095] More specifically, in an embodiment, the server computing system (130) can provide an environment in which an application for providing an image analysis service based on separate analysis of spatial features and temporal features on a user computing device (110) can operate.

[0096] To this end, the server computing system (130) may include an application program, data and / or instructions, etc. for the application to operate, and may transmit and receive various data based thereon with the external device.

[0097] A server computing system (130) may include at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or electrical units for performing other functions, or a plurality of electrically connected processors.

[0098] For example, ASICs may have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits (see Fig. 2).

[0099] And the memory (132) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (132) may store data (133) and instructions (134) necessary for the processor (131) to perform functional operations, such as training an artificial intelligence model or executing an image analysis method based on the separate interpretation of spatial and temporal features through the artificial intelligence model.

[0100] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate a plurality of computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include a plurality of computing devices connected to a network (170).

[0101] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer non-linear model as the machine learning model (140). Exemplary neural networks may include a feed-forward neural network, a deep neural network, a recurrent neural network, and a convolutional neural network.

[0102] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, data store) which is a storage for continuously storing and managing raw data that forms the basis of an image analysis service based on separate analysis of spatial features and temporal features.

[0103] Such data stores may include various forms of data storage, ranging from file systems to cloud storage. For example, a data store may include at least one database among a relational database that uses a structured query language (SQL) to define and manipulate data, a NoSQL database designed for flexibility and scalability to process unstructured and semi-structured data, a data warehouse optimized for querying and analysis by centralizing large volumes of data from multiple sources as a system used for reporting and data analysis, a data warehouse that stores large volumes of raw data in basic formats such as structured data, semi-structured data, and unstructured data, and a local storage device or Network Attached Storage (NAS) that stores data in files in a format generally accessible by a computer operating system.

[0104]

[0105] - Training Computing System (150: Training Computing System)

[0106] The training computing system (150) may include at least one processor (151) and memory (152). Here, the processor (151) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions, or a plurality of electrically connected processors.

[0107] For example, ASICs may have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits (see Fig. 2).

[0108] And the memory (152) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc. and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0109] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in a user computing device (110) and / or a server computing system (130) using various training or learning techniques, such as back propagation of error.

[0110] For example, such a model trainer (160) can perform backpropagation updates to one or more parameters of a machine learning model (120 and / or 140) for an image analysis service based on separate interpretation of spatial and temporal features based on a defined loss function.

[0111] In some embodiments, performing backpropagation of the error may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight devaluation, dropout and / or knowledge distillation, etc.) to improve the generalization ability of the machine learning model (120 and / or 140) being trained.

[0112] For example, a model trainer (160) can train a machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data of different forms, such as, for example, images, audio samples and / or text.

[0113] Examples of image types that can be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0114] These training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0115] And the model trainer (160) includes computer logic that is utilized to provide the desired function.

[0116] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one embodiment, the model trainer (160) may include a program file stored in a storage device, be loaded into memory (152), and be executed by one or more processors (151). In another embodiment, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0117] Network (170) includes, but is not limited to, 3GPP (3rd Generation Partnership Project) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth network, satellite broadcasting network, analog broadcasting network and / or DMB (Digital Multimedia Broadcasting) network.

[0118] Generally, communication through the network (170) can be performed using any type of wired and / or wireless connection through various communication protocols (e.g., TCP / IP, HTTP, SMTP and / or FTP, etc.), encodings or formats (e.g., HTML and / or XML, etc.), and / or protection schemes (e.g., VPN, Secure HTTP and / or SSL, etc.).

[0119] FIG. 3 is a block diagram of a computing device (100) that implements an image analysis service based on separate analysis of spatial features and temporal features according to one embodiment.

[0120] Referring to FIG. 3, the computing device (100) included in the user computing device (110), server computing system (130), and training computing system (150) includes a plurality of applications (e.g., applications 1 to N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing application (e.g., detection, classification, and / or segmentation, etc.), a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and / or a chat-bot application.

[0121] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and by storing and operating the trained artificial intelligence model, it may provide output data according to a predetermined input data.

[0122] Each application of the computing device (100) can communicate with a number of other components of the computing device (100), such as, for example, at least one sensor, a context manager, a device state component, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0123] FIG. 4 is a block diagram of a computing device (200) implementing an image analysis service based on separate analysis of spatial features and temporal features according to another embodiment.

[0124] Referring to FIG. 4, the computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application can communicate with a central intelligence layer. For example, applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application and / or a browser application. In one embodiment, each application can communicate with the central intelligence layer (and a model stored therein) using an API (e.g., a common API across all applications).

[0125] The central intelligence layer may include a number of machine learning models. For example, as illustrated in FIG. 4, at least some of the machine learning models may be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model for all applications. In some embodiments, the central intelligence layer may be included within the operating system of the computing device (200) or otherwise implemented.

[0126] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data store for the computing device (200). As illustrated in FIG. 4, the central device data layer can communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0127] The technology described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from said systems. It will be recognized that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, division of tasks, and functionality between and from components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components operating in combination. Databases and applications may be implemented in a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0128] FIG. 5 is a block diagram of a computing device (1000) implementing an image analysis service based on separate interpretation of spatial features and temporal features according to another embodiment. FIG. 6 illustrates an exemplary configuration of an analysis architecture that performs image analysis based on separate interpretation of spatial features and temporal features according to an embodiment. FIG. 7 is intended to explain a method for encoding location information for an input sequence of a large language model according to an embodiment. FIG. 8 is a block diagram illustrating the configuration of a spatial-language mapping module (17) according to an embodiment. FIG. 9 is intended to explain the function of a spatial-language mapping operation module (22) according to an embodiment.

[0129] Referring to FIG. 5, the computing device (400) included in the user computing device (110), server computing system (130) and training computing system (150) may include a vision feature extraction module (10), a global token generation module (11), a spatial token generation module (12), a time token generation module (13), a text token generation module (14), a prompt configuration module (15), a large language model training module (16), and a spatial-language mapping module (17).

[0130] Additionally, referring to FIG. 6, the image analysis model framework used by a computing device (400) according to one embodiment for image analysis may include the architecture of a multimodal video-language model (VLM) that performs complex spatiotemporal inference by integrating and processing input video data and text data.

[0131] This architecture processes images using a vision encoder (e.g., CLIP) to extract visual features, which can then be separated into spatial and temporal tokens through processes such as spatial pooling and temporal pooling. The text tokens generated by passing these prepared visual tokens and text provided by the user through a tokenizer and a text embedding module can then be input into a large language model. Subsequently, positional information can be encoded by applying Harmonic-Rotary Position Embedding to the input sequence, and a specific data fusion operation (DST Attention) can be performed to separate spatiotemporal information.

[0132] The vision feature extraction module (10) can perform the role of extracting visual features necessary for subsequent analysis and language model training from input video data. The visual features extracted by the vision feature extraction module (10) may have the form of a vision feature map.

[0133] The vision feature extraction module (10) can process input image frames to generate a token embedding that compresses the content and meaning of the image. The vision feature extraction module (10) can utilize a pre-trained encoder, and, referring to FIG. 6, can utilize a model such as CLIP (Contrastive Language-Image Pre-training).

[0134] The vision features extracted by the vision feature extraction module (10) can have a high-dimensional vector form that captures meaningful information such as objects, actions, and scenes within the image, going beyond mere pixel information of the image. These features can then be transmitted to the spatial token generation module (12) and the temporal token generation module (13) to be separated and processed into tokens containing spatiotemporal information.

[0135] The global token generation module (11) can generate a global token that summarizes and represents the entire content of the input image based on the visual features of the image obtained from the vision feature extraction module (10).

[0136] The global token generation module (11) can be designed to represent the macroscopic context or key content that penetrates the entire sequence in a compressed form, rather than being limited to the local features of each frame or patch of the image. Referring to FIG. 6, for example, the global token generation module (11) can generate CLS Tokens (Classification Tokens) that can function as a vector containing comprehensive information of the entire image.

[0137] Global tokens can be generated through a pooling process of extracted visual features. This pooling process, including weight-based summing, maximum value extraction, or average calculation, condenses the most important information from numerous spatial and temporal tokens to produce a single or small number of global token vectors. The global tokens generated through the global token generation module (11) can then be combined with spatial and temporal tokens and input into a large language model.

[0138] Global tokens serve as representatives containing the high-level context of an image, encouraging the model to consider the overall situation when deriving answers to complex queries (e.g., prompts) or image analysis results. This enables the model to perform consistent inference within the overall context rather than relying solely on partial features, thereby playing a key role in improving the accuracy and reliability of the analysis.

[0139] The spatial token generation module (12) can perform the role of separating and encoding spatial information within each frame of the input image based on visual features extracted through the vision feature extraction module (10). The spatial token generation module (12) can generate spatial tokens that can identify spatial correlations within the same point in time (frame).

[0140] Referring to FIG. 6, the spatial token generation module (12) can generate spatial tokens by applying spatial pooling to the extracted vision feature maps, and thus the spatial tokens can include details related to the local context within each frame and the shape and location of the object.

[0141] When the generated spatial tokens are subsequently processed through a disentangled spatial and temporal attention mask in a large language model, they can be controlled to perform attention operations only with other spatial tokens within the same frame, without performing attention operations with tokens from other time frames. By processing spatial information separately from temporal information in this way, the model can prevent unnecessary data entanglement that occurs during the analysis of complex sequence data and clearly and efficiently interpret the spatial context within a single time point.

[0142] The time token generation module (13) can separate and encode temporal flow information of an input image sequence based on visual features obtained from the vision feature extraction module (10). The time token generation module (13) can generate temporal tokens capable of capturing movement or long-term dependencies over time at the same spatial location.

[0143] Referring to FIG. 6, the time token generation module (13) can generate time tokens by applying temporal pooling to the extracted features of the extracted vision feature map, thereby compressing and including information about the temporal relationship between each frame and changes in the object in the time tokens.

[0144] When the generated time tokens are processed through a Disentangled Spatial & Temporal Attention Mask, they can be controlled to perform attention operations only with tokens of the same spatial location in different time frames, rather than with other spatial tokens within the same time frame. By processing temporal information separately from spatial information in this way, the model can effectively identify long-term temporal changes and the progression of behavior; in particular, this can mitigate the long-term dependency issues that occur in sequence data analysis and improve the efficiency and consistency of the analysis.

[0145] The text token generation module (14) can generate text tokens by processing and encoding language features, namely user prompts or descriptive text, that are input to the image analysis system (1000). The text token generation module (14) can prepare the system (1000) to fuse visual information and language information in a multi-modal environment to perform inference.

[0146] Referring to FIG. 6, for example, a text token generation module (14) can apply a tokenizer to input text to separate the sentence into individual words or subwords and convert them into text embeddings. Through this embedding process, a high-dimensional vector containing semantic information and grammatical context of the words can be generated.

[0147] The generated text tokens are subsequently input into a large language model and can be used in attention operations along with spatial and temporal tokens extracted from the image. Through this operation, the model can accurately map which spatiotemporal elements of the image the user's question (language) relates to and generate complex inference results.

[0148] The input sequence configuration module (15) can combine and arrange various types of tokens extracted and generated within the system (1000) into a single integrated sequence that fits the input format of a large language model. The input sequence configuration module (15) receives global tokens, spatial tokens, temporal tokens, and text tokens, determines the order among the tokens, and arranges them into a single long vector sequence. In this way, the input sequence configuration module (15) clearly configures the input sequence and transmits it to the large language model learning module (16) to enable efficient learning and inference.

[0149] The large language model learning module (16) can pre-train at least one artificial intelligence model to maximize the accuracy and consistency of image analysis by integrating and analyzing input visual and linguistic features and mitigating errors (hallucinations) in spatiotemporal relationship inference.

[0150] The large language model training module (16) can sequentially perform structured position encoding and Disentangled Spatio-Temporal (DST) attention operations on global tokens, spatial tokens, time tokens, and text tokens received from the input sequence configuration module (15).

[0151] The integrated input sequence passed to the large language model learning module (16) may first undergo a step of encoding position information. Through this process, structured spatiotemporal position information can be reflected for tokens based on Harmonic-Rotary Position Embedding.

[0152] For example, referring to Fig. 7, a Balance Position Id based on local order and a Distinct Position Id based on global order may be assigned to each token.

[0153] Here, local order refers to the relative order within the token type group to which the token belongs (time token group, space token group, etc.), and global order may refer to the absolute order of the token relative to the entire input sequence.

[0154] For example, referring to Fig. 7, the first token generated within the time token group has a balanced position identifier of '1' according to local order, and the fourth spatial token generated throughout the input sequence may have a unique position identifier of '4' according to global order.

[0155] In this manner, the balanced position identifier and unique position identifier assigned to each token can serve as criteria for determining the rotation angle to effectively inject spatiotemporal position information into the token's embedding vector. The position information obtained from these two identifiers is combined to calculate the final rotation angle for each token, and this angle can be used to perform rotation operations based on Harmonic-Rotary Position Embedding. In this process, the balanced identifier provides a rotation component that contributes to relative and local relationships within the token group, while the unique identifier provides a rotation component that reflects global position information based on the entire sequence.

[0156] The actual rotation operation can be performed by applying a single rotation operation to the embedding vector of the corresponding token using a rotation angle determined by integrating information obtained from two identifiers. Through this operation, the token embedding can simultaneously include two different types of location information (local order and global order). This enables large language models to accurately recognize relationships between tokens that are complexly intertwined in space and time, and consequently, can mitigate hallucinations that occur during image analysis and dramatically improve the consistency and precision of inference.

[0157] Tokens encoded with location information can be processed through Disentangled Spatial & Temporal (DST) attention operations, thereby preventing unnecessary entanglement between spatiotemporal information and maximizing analysis efficiency.

[0158] The DST attention operation operates based on a Disentangled Spatial & Temporal Attention Mask. It applies causal attention, which refers only to the previous time point, to temporal tokens, and allows bidirectional attention, which refers to the entire context, to spatial tokens and text tokens, thereby enabling control of the optimal operation method for each token type.

[0159] Attention operations can be performed by calculating an attention score through the inner product between the query and key of tokens that reflect location information. This score is applied to the Sotmax function and converted into weights in the form of a probability distribution. Additionally, tokens included in the input sequence can be transformed into value (V) vectors through a separate linear transformation module (Value Projector). In this case, the ground truth (GT) data can be provided in the form of correct labels or descriptive text for the input images, and the final output of the large language model can be compared with this ground truth data to calculate the loss function.

[0160] The attention weights passed through the Softmax function are multiplied by the value (V) of the corresponding token and weightedly summed to generate the final output vector, which can be input into a Multi-Layer Perceptron (MLP). The MLP can refine the feature vector through non-linear transformations.

[0161] During the backpropagation process, parameters can be adjusted to minimize the calculated loss. At this time, the parameters of pre-trained feature extractors, such as vision encoders (CLIP) and text embedding models, are frozen, while the parameters of modules responsible for spatiotemporal relationship inference—such as various projectors that generate spatial and temporal tokens from vision feature maps, DST attention blocks, and MLPs—can be updated.

[0162] Through these sophisticated DST attention operations and structured positional encoding processes, the large language model can clearly learn complex spatial relationships within images and associations between visual and linguistic elements. This enables the model to infer precisely and error-free even for complex linguistic instructions that combine visual elements and spatial locations, thereby serving as a foundation for providing highly reliable image analysis results that significantly reduce errors (hallucinations) in analytical inference.

[0163] The spatial-language mapping module (17) may use a Spatial-Language Mapping Neural Module (SLMM) that performs operations to fuse spatial meanings by integrating and aligning visual spatial information extracted from an image and the user's linguistic instructions.

[0164] The space-language mapping module (17) can generate contextual space tokens that enable precise inference without error even for complex language instructions that combine visual changes and spatial locations, by going beyond simply connecting features of text and images, through the interaction of a space description token corresponding to information specifying space (e.g., bounding box information, coordinate information, etc.) and a language token containing language information describing a predetermined space.

[0165] The spatial-language mapping module (17) may include various internal modules for such complex mapping operations. For example, referring to FIG. 8, it may be configured around a spatial-language mapping operation module (22) that fuses the information, and includes a spatial information encoding module (20) and a language information encoding module (21) that process visual features and language features, respectively. Additionally, the spatial-language mapping module (17) may include a spatial information transformation module (23) and a language information transformation module (24) that improve the features of spatial information and language information so that the model can interpret them easily, thereby increasing the accuracy of the mapping operation.

[0166] Furthermore, the spatial-language mapping module (17) may include modules to support complex context and knowledge-based reasoning. For example, the spatial-language mapping module (17) may include a prompt reconstruction module (25) to increase the depth of reasoning by utilizing external knowledge, such as a spatial knowledge graph database (27) and a domain-specific spatial knowledge database (28), and to resolve ambiguity in the input prompt.

[0167] Finally, the space-language mapping module (17) integrates visual, linguistic, and knowledge information through the multimodal context encoding module (26) to complete a contextual space token, thereby laying the foundation for the system (1000) to finally output an accurate response.

[0168] The process of generating contextual space tokens (CSI) within the space-language mapping module (17) can be achieved through the organic interaction of the space information encoding module (20), the language information encoding module (21), and the space-language mapping operation module (22).

[0169] The space-language mapping module (17) can fuse the meaning of location information (spatial information), such as bounding boxes / coordinates specifying the space, and text (language information) describing the space through cross-attention.

[0170] Referring to FIG. 9, for example, a spatial information encoding module (20) can generate spatial description tokens (SI) based on spatial information provided by a user. Here, spatial information may refer to structured numerical information that specifies the location of an object within an image, such as bounding box coordinates, the center point of an object, and the size of an area. The spatial description tokens (SI) may correspond to a set of vectors that encode location information specifying such space.

[0171] Additionally, the language information encoding module (21) can process the user's prompt text to generate a language token (LI). The language token (LI) may correspond to a set of vectors that encode the meaning and context of the text describing the space.

[0172] The spatial-language mapping operation module (22) can perform a cross-attention operation that analyzes the interaction between spatial descriptive tokens (SI) and language tokens (LI) to generate contextual spatial tokens (CSI). This process fuses the meanings of spatial descriptive tokens (SI) corresponding to location information such as bounding boxes / coordinates that specify the space and language tokens (LI) corresponding to information regarding text indications, and provides this to the learning process of a large language model, thereby enabling the large language model to accurately identify the spatial meaning indicated by the text within the visual features.

[0173] In the cross-attention operation, spatial descriptive tokens (SIs) act as queries, while language tokens (LIs) can serve as keys and values. This corresponds to a process where spatial descriptive tokens (SIs) reference language tokens (LIs) to re-evaluate their importance and meaning. This operation is similar to the process of finding the answer to the question, "Which part of the image does the text describe?", and a high weight (attention score) can be assigned to areas where the meaning of the text matches spatial features.

[0174] Contextual Spatial Tokens (CSIs) generated as a result of the cross-attention operation correspond to a new set of tokens enhanced by the fusion of spatial information and the context of linguistic references. CSIs can contain vectors in which the importance and meaning of visual elements are reinterpreted according to specific text references. For example, through the text "red object in the top-left corner," CSIs can be used to improve inference accuracy by maximizing the importance of spatial tokens related to location information regarding the top-left corner area over all other spatial features.

[0175] The spatial information transformation module (23) is responsible for directly transforming the spatial information of an object appearing in an image, which can contribute to securing the robustness of the model. The spatial information transformation module (23) can generate transformed spatial information that can be recognized at various viewpoints or under various conditions by applying transformations such as rotation, enlargement (reduction), and movement based on input spatial information (bounding box coordinates, center point, etc.).

[0176] For example, by generating new spatial information by rotating or enlarging the coordinates of an object in an image, the model can be supported in learning to maintain consistency even in the deformed form of the object when it is subsequently passed to the spatial-language mapping operation module (22). The spatial information transformation module (23) can consequently contribute to increasing the accuracy of the mapping operation by improving spatial features so that the model can interpret them easily.

[0177] The language information transformation module (24) can perform the role of adjusting or refining the features of language tokens according to the transformation applied by the spatial information transformation module (23). This can contribute to preventing inference errors or hallucinations caused by image transformation and increasing the accuracy and consistency of spatial information processing in the large language model by ensuring that the model accurately reflects the impact of spatial information on language expression.

[0178] For example, when the spatial information of an object within an input image undergoes transformations such as rotation, enlargement, or reduction, the language information transformation module (24) can adjust or refine the features of the language tokens to match the transformed spatial conditions. If the coordinates of "a box in the top-left corner" are changed by the transformation module (23) to indicate "a bottom-right corner," the language information transformation module (24) can automatically adjust the meaning of the original language token "top-left corner" to a text feature indicating "a bottom-right corner."

[0179] In this manner, the transformed spatial descriptive tokens (SI) and the adjusted language tokens (LI) can be provided as targets for cross-attention operations of the spatial-language mapping operation module (22). That is, the spatial-language mapping operation module (22) performs cross-attention operations using these transformed tokens as queries and key / value pairs, thereby generating a final contextual spatial token (CSI) with ensured robustness. This process ensures that the large language model accurately fuses linguistic instructions and spatial meanings without error, even with respect to transformed spatial information, and can ultimately be provided to the learning process of the large language model.

[0180] The multimodal context encoding module (26) can independently integrate and process visual information and linguistic information within the spatial-language mapping module (17) to complete an optimized multimodal representation that can be utilized by a large language model. This can complement or process the primary fusion results of the spatial-language mapping operation module (22) in parallel, thereby contributing to complex spatial context inference and improving the accuracy of the information. The multimodal context encoding module (26) can improve the accuracy and consistency of inference by going beyond simply combining features and deeply considering the influence of visual and linguistic information on each other.

[0181] The multimodal context encoding module (26) can receive visual tokens and language tokens separately from the space-language mapping operation module (22) and perform independent multimodal fusion. The multimodal context encoding module (26) can encode complex context information.

[0182] For example, a multimodal context encoding module (26) can generate a multimodal representation represented as a visual-language integrated embedding by integrating an image (visual) embedding and a text (language) embedding. This may include a process of aligning vectors through a linear transformation or an initial fusion operation so that features of different modalities exist in a common semantic space.

[0183] Additionally, the multimodal context encoding module (26) can perform attention operations on the integrated multimodal representation to identify the relationship between specific regions of the image and the text, and reflect the contextual meaning in spatial information processing. Through this process, visual features important to text instructions are emphasized, and the accuracy of inference can be improved.

[0184] Furthermore, the multimodal context encoding module (26) may include a network capable of inferring spatial relationships between objects within an image and reflecting them in the final text representation. This can help interpret complex relationships, such as "A is to the left of B," beyond simple object detection.

[0185] The multimodal context encoding module (26) can add depth to reasoning by integrating external knowledge, such as a spatial knowledge graph database (27) and a domain-specific spatial knowledge database (28). By combining or referencing structured spatial relationships extracted from external knowledge into contextual spatial tokens (CSI), the multimodal context encoding module (26) can help prevent the large language model from making errors in reasoning that requires unstructured spatial relationships or expertise in a specific domain. This integration can contribute to solving problems that limit the use of large language models, particularly in applications where the accuracy of spatial information is critical, such as medical image analysis, autonomous driving, and robot control.

[0186] The multimodal context encoding module (26) can contribute to increasing the accuracy and consistency of spatial information processing, beyond simply processing text and visual information in parallel in a large language model.

[0187] The spatial knowledge graph database (27) may include a structured external knowledge-based database that is utilized to process unstructured spatial relationships and improve the spatial interpretation capabilities of a large language model. The spatial knowledge graph database (27) may include a knowledge graph constructed by expressing spatial relationships and characteristics extracted from various sources, such as text, images, and sensor data, in the form of nodes and edges.

[0188] The spatial-language mapping module (17) can improve spatial accuracy and consistency in the inference of the large language model by inferring spatial relationships between objects within an image by referring to this database and reflecting this in language expressions. This can help the large language model to understand complex spatial contexts contextually by utilizing structured spatial knowledge, rather than relying simply on visual information.

[0189] A domain-specific spatial knowledge database (28) can be constructed to effectively integrate specialized and highly specific spatial information of a specific application domain (e.g., medical, architectural, autonomous driving, etc.) into a large language model. This enables the processing of spatial information that takes into account domain-specific characteristics that are difficult to resolve with general spatial relationships alone, and this can contribute to overcoming the spatial interpretation limitations of the large language model.

[0190] For example, spatial knowledge applicable only to a specific domain, such as the anatomical positional relationships of specific lesions in the medical domain or the spatial characteristics of specific road structures in the autonomous driving domain, can be provided in a structured form. By utilizing the domain-specific spatial knowledge database (28), the system (1000) can minimize spatial reasoning errors occurring in the domain and achieve accuracy that meets the requirements of actual application fields.

[0191]

[0192] - Image analysis using an artificial intelligence model (S100)

[0193] An image analysis method (S100) using an artificial intelligence model according to one embodiment can improve the inference errors (illusion phenomena) and efficiency issues inherent in existing complex image data analysis systems. The method (S100) goes beyond simply analyzing images and enables intelligent multimodal inference that interprets a user's linguistic instructions by accurately connecting them to the spatiotemporal features of the image. For example, the method (S100) can increase the reliability of the analysis by applying a new encoding technique that helps the artificial intelligence more clearly recognize complex positional relationships or temporal changes within the image.

[0194] The method (S100) can separate and reconstruct spatial information extracted from an image and the user's linguistic information and provide them to an artificial intelligence model. Through this process, the artificial intelligence model can accurately learn information about where, when, and in what relationship a specific object is located within the overall flow of the image. For example, it can respond precisely without error even to complex linguistic instructions that combine time and location, such as "a red object moving in the top-left corner."

[0195] The method (S100) helps an artificial intelligence model deeply understand the context of data, thereby effectively reducing the illusion of inferring information that does not actually exist, and provides high scalability and flexibility that allows it to be stably applied to new types of data or long video sequences without additional learning, thus providing highly reliable analysis results and excellent utility value in various application fields such as medical image analysis, autonomous driving, and robot control.

[0196] FIG. 10 is a flowchart of an image analysis method (S100) according to one embodiment. FIG. 11 is a flowchart of a learning method for at least one artificial intelligence model used in the method (S100) of FIG. 10. FIG. 12 is a flowchart for explaining steps that may be included in the step (S1059) of encoding location information of the method of FIG. 11. FIG. 13 is a flowchart for explaining steps for generating contextual spatial tokens through vision-language mapping that the method of FIG. 11 may further include.

[0197] Referring to FIG. 10, an image analysis method (S100) using an artificial intelligence model according to one embodiment may include the steps of receiving an image for analysis and storing it in at least one memory (S101), loading the image for analysis from the at least one memory (S103), generating at least one analysis result for the image for analysis using at least one artificial intelligence model that takes the image for analysis as input by at least one processor (S105), and inputting the analysis result to at least one subsequent processing component (S107).

[0198] In one embodiment, the method (S100) may be performed by a processor (131) included in a server computing system (130). However, it is not limited thereto, and at least a part of the method (S100) may be performed by a processor (111) of a user computing device (110) or a processor (151) of a training computing system (150), and another part may be performed by a processor (131) included in a server computing system (130).

[0199] For convenience of explanation, the following description describes a processor (131) included in a server computing system (130) performing the method (S100).

[0200] In step (S101), the processor (131) can receive an image for analysis through a user computing device (110) and store it in at least one memory. Here, the image for analysis is data to be processed by an image analysis system (1000) using a large language model, and may include various types of sequence data containing complex spatiotemporal features, such as video frame sequences, high-resolution single images, or medical images. This process is responsible for the initial acquisition of data for subsequent analysis and stable storage within the system, and the received image may be processed in a vision feature extraction module (10) before being input into a large language model learning module (16).

[0201] The received image for analysis can be stored in memory (RAM, ROM, etc.) or a storage device (SSD, disk, etc.) to ensure data integrity and can be prepared so that the processor (131) can efficiently load the image for analysis in a subsequent step (S103). This step (S101) signifies the starting point of a preprocessing process for accurately extracting and analyzing spatiotemporal features of the image, and corresponds to an initial stage in which the data is converted to a state accessible to the computing device of the system (1000).

[0202] In step (S103), the processor (131) may load an image for analysis stored in at least one memory for subsequent processing. Step (S103) may include loading data from a storage location into the processor's (131) workspace (e.g., cache or main memory) so that the processor (131) can directly access and process the image for analysis.

[0203] In step (S105), the processor (131) can perform a series of core processes to generate an analysis result for an image for analysis using an artificial intelligence model. Step (S105) can extract spatial and temporal visual features from the image for analysis, integrate and analyze them together with the linguistic features of text data through a Disentangled Spatio-Temporal (DST) attention operation, and then finally generate an output result.

[0204] In this case, the analysis result may include at least one of a natural language description including spatial-temporal relationships and context for the image for analysis and information regarding object coordinates.

[0205] Referring to FIG. 11, for example, step (S105) may include: a step of extracting a vision feature map from a training image (S1051); a step of generating a global token, a spatial token, and a temporal token for the training image using the vision feature map (S1053); a step of generating a text token from a text input (S1055); a step of constructing an input sequence including the global token, the spatial token, the temporal token, and the text token (S1057); a step of encoding position information by applying a harmonic rotation position embedding to each token of the input sequence (S1059); a step of performing an attention operation with the input sequence encoded with position information, applying the separation mask to control the interaction between the spatial token and the non-spatial token (S1061); a step of generating an output result by performing a non-linear transformation on the result of the attention operation (S1063); and a step of updating the parameters of the at least one artificial intelligence model so as to minimize the loss based on the output result and the correct answer data (S1065).

[0206] In step (S1051), the processor (131) can extract a Vision Feature Map from a training image. This process may include processing the input image for analysis using a Vision Encoder, compressing the image's content (objects, shapes, scenes, etc.), and generating a feature map in the form of a high-dimensional vector containing meaning. This Vision Feature Map can be used as basic data for generating global tokens, spatial tokens, and temporal tokens in the next step (S1053).

[0207] In step (S1053), the processor (131) can generate global tokens, spatial tokens, and temporal tokens based on the vision feature map extracted in step (S1051). This process may mean separating and encoding the features of the vision modality into independent token forms.

[0208] For example, global tokens contain macroscopic context or scene information for the entire image, while spatial tokens represent static spatial features such as the location, boundaries, and shapes of objects within the image. Additionally, temporal tokens can encode dynamic temporal features, such as object movement or state changes, within a video sequence.

[0209] By preparing each feature in the form of a separated token in this way, the interaction between spatial and temporal features is precisely controlled through the DST attention operation described later, thereby enabling the large language model to effectively infer complex spatiotemporal contexts.

[0210] In step (S1055), the processor (131) can generate text tokens from the text input. This process may include separating the prompt or query text entered by the user into individual words or semantic units using a tokenizer and converting them into text embeddings. The generated text tokens are prepared in the form of high-dimensional vectors containing linguistic meaning and context, and can be used as input data for a DST attention operation along with visual tokens (global, spatial, and temporal tokens) in a subsequent step.

[0211] In step (S1057), the processor (131) can combine all tokens generated in the previous steps (S1053 and S1055), respectively, to form a final input sequence for a Disentangled Spatio-Temporal (DST) attention operation.

[0212] This input sequence may consist of a Global token, a Spatial token, a Temporal token, and a Language token. The processor (131) can arrange these separate tokens into a single integrated sequence.

[0213] Through step (S1057), data can be arranged in the most efficient form so that the DST operation can simultaneously reference spatial, temporal, and linguistic features and analyze their interaction. This lays the foundation for precisely processing complex multimodal information in separate dimensions.

[0214] In step (S1059), the processor (131) can encode position information by applying a Harmonic-Rotary Position Embedding (RoPE) to each token of the input sequence. This process clearly reflects the relative positional relationships between tokens in a vector space, which helps the large language model accurately grasp spatiotemporal context and reduce hallucinations.

[0215] Referring to FIG. 12, for example, step (S1059) may include: a step (S191) of assigning a balanced position identifier based on a local order and a unique position identifier based on a global order to each token in the input sequence; a step (S193) of determining a rotation angle for each token based on the balanced position identifier and the unique position identifier; and a step (S195) of encoding position information for each token by performing a rotation operation on each token in the input sequence using the rotation angle determined for each token.

[0216] In step (S191), the processor (131) may assign a balanced location identifier based on local order and a unique location identifier based on global order to each token in the input sequence. The balanced location identifier indicates the relative order within the token type group (space, time, text) to which the token belongs and contributes to understanding the local context. The unique location identifier indicates the absolute order of the corresponding token relative to the entire input sequence and contributes to understanding the macroscopic context and long-term location.

[0217] These two identifiers define the spatiotemporal location of the token in a dual manner, and can subsequently be used as a criterion for determining the rotation angle.

[0218] In step (S193), the processor (131) can determine the rotation angle for each token based on the balanced position identifier and unique position identifier assigned to each token in step (S191). In this process, the information from the two identifiers can be integrated to calculate the final angle at which the token must rotate in the embedding space. This angle contains relative distance information regarding how far the token is from other tokens and is uniquely determined for each token, which can be used to increase the precision of spatiotemporal relationship inference.

[0219] In step (S195), the processor (131) can finally encode position information by performing a rotary operation on each token in the input sequence using the rotation angle determined per token in step (S193). This rotary operation transforms the embedding vector of the token to effectively inject position information, thereby enabling accurate spatiotemporal relationship inference during subsequent attention operations and improving the accuracy of image analysis.

[0220] In step (S1061), the processor (131) may perform a Disentangled Spatio-Temporal (DST) attention operation using an input sequence in which location information is encoded as input. This operation may apply a disentangling mask to control the interaction between spatial tokens and temporal tokens. The disentangling mask allows causal attention for temporal tokens and bidirectional attention for spatial tokens and text tokens, and can prevent unnecessary entanglement between spatial information and temporal information.

[0221] Through this, the model can efficiently process spatiotemporal information by separating it, mitigate long-term dependency issues, and improve the accuracy and consistency of the analysis.

[0222] In step (S1063), the processor (131) can generate a final output result by performing a non-linear transformation on the result of the attention operation from the previous step (S1061). This non-linear transformation can be performed, for example, through a Multi-Layer Perceptron (MLP) module, and the MLP module can receive the feature vector enhanced through the attention operation as input, refine the features into a form suitable for the final inference purpose (e.g., classification, response generation, object detection), and transform them into a high dimension.

[0223] The output result generated in this step (S1063) may include information (e.g., text answer, location / description of an object in the image) that the image analysis system (1000) ultimately provides to the user.

[0224] In step (S1065), the processor (131) can update the parameters of the artificial intelligence model to minimize the loss based on the generated output result and ground truth data.

[0225] This process is carried out through the backpropagation algorithm, and the model's internal weights (parameters) can be adjusted to reduce the error (loss) between the output results and the ground truth data. Parameter updates can be performed on modules responsible for spatiotemporal relationship inference, such as DST attention blocks, MLPs, and projectors, for example; through this, the model's performance can be progressively improved and its ability to mitigate hallucinations can be enhanced.

[0226] In step (S107), the processor (131) may input the analysis result generated using the artificial intelligence model in step (S105) into at least one subsequent processing component. This analysis result is final output information obtained through inference of a large language model (LLM) and may include text descriptions of complex images, location and attribute information of objects, or inferred spatial relationships.

[0227] Step (S107) may correspond to a step of stably and efficiently transmitting the results to the next step of the system (1000) so that they can be utilized in actual application fields after the analysis of the artificial intelligence model is completed.

[0228] In this case, at least one subsequent processing component may use the analysis results to provide at least one service among autonomous driving, robot control, medical image analysis, architectural design, and construction management. For example, spatial relationship inference information derived from the analysis results may be utilized for path planning in an autonomous driving system or as direct data for workspace recognition and command execution in a robot control system.

[0229] Here, at least one subsequent processing component may include various types of systems or modules that receive an analysis result (e.g., spatiotemporal relationship inference result) generated through an artificial intelligence model in step (S105) and provide a final service or perform additional system tasks based thereon. The subsequent processing component may utilize the analysis result in accordance with the requirements of the actual application domain and may be implemented in the form of a module inside the computing device (400) or an external application system depending on the system architecture.

[0230] If the follow-up processing component is implemented as a module within the computing device (400), it can perform the role of delivering the analysis results to the end user or integrating them with system data. For example, it may include a module that manifests the analysis results through at least one user interface (UI) in the form of a text report, a warning message, or a visualized form. Additionally, it includes a database storage system for long-term preservation of analyzed spatial relationships and inference results, and a communication / transmission module for securely transmitting data to other internal modules or external systems. These internal modules can contribute to converting the LLM analysis results into a user-friendly form and ensuring data integrity within the system.

[0231] The follow-up processing component may be an external application system that provides services for specific purposes by utilizing the analysis results. This can be applied in domains where the accuracy of spatial information is critical, such as autonomous driving, robot control, medical image analysis, architectural design, and construction management. For example, autonomous driving systems utilize precise spatiotemporal relationship information derived from analysis results to improve the accuracy of collision risk assessment and path planning, while robot control systems execute commands without errors based on information regarding the location and movement of objects within the workspace. In this way, external application systems can use highly refined multimodal analysis results as the basis for final decisions and actions.

[0232] Additionally, at least one follow-up processing component can manifest the analysis results through at least one user interface (UI). This means displaying the analysis results on a screen so that the user can visually check and interact with them, or outputting them in the form of specific warning messages or reports. Through this, the final result of the image analysis system (1000) can be delivered to the user, thereby maximizing its usefulness.

[0233] In this case, for example, the process of at least one subsequent processing component expressing the analysis results through at least one user interface may be based on a weight matrix calculated through a predetermined operation using a separation mask for spatial tokens and non-spatial tokens.

[0234] For example, at least one follow-up processing component can provide an image through a user interface with different visual effects applied to multiple regions of the image related to the analysis result.

[0235] In other words, by analyzing the interaction strength (weight matrix) between tokens during DST attention operations and applying specific emphasis effects (e.g., highlights, color changes, focusing, etc.) only to image regions highly relevant to the user's query (e.g., objects to be analyzed), the complex inference results of a large language model can be conveyed to the user intuitively and clearly.

[0236] Meanwhile, the method (S100) may further include steps for generating contextual space tokens through vision-language mapping.

[0237] For example, referring to FIG. 13, the method may include the step of generating a spatial description token corresponding to information specifying a predetermined space for the training image (S111), the step of generating a language token including language information describing the predetermined space (S113), and the step of generating a contextual spatial token by performing an operation to fuse spatial meaning through the interaction of the spatial description token and the language token (S115).

[0238] In step (S111), the processor (131) may generate a spatial description token (SI) corresponding to information specifying a specific space for a training image. Here, the information specifying the space may include structured numerical information specifying the location of an object within the image, such as bounding box coordinates, the center point of the object, and the size of the area. The spatial description token (SI) is a set of vectors encoding this location information and can serve as a query in a spatial-language mapping operation.

[0239] In step (S113), the processor (131) may generate a language token (LI) containing language information describing the aforementioned space. This process may include separating words or phrases (e.g., "top left corner," "red object") containing spatial meaning from a user's prompt or text instruction through a tokenizer and converting them into text embeddings. The language token (LI) may serve as a key and value in a space-language mapping operation to provide linguistic context to the space description token (SI).

[0240] In step (S115), the processor (131) can generate a contextual spatial token (CSI) by performing an operation that fuses spatial meaning through the interaction of a spatial descriptive token (SI) and a language token (LI). This operation can be performed in such a way that the spatial descriptive token (SI) updates itself by referencing the language token (LI) through cross-attention. The contextual spatial token (CSI) contains spatial information whose importance is reinterpreted by linguistic instructions, which can help a large language model accurately identify the spatial meaning indicated by the text within the visual features.

[0241] Various embodiments of the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the various embodiments of the present disclosure, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the various embodiments of the present disclosure, and vice versa.

[0242] The specific embodiments described in this disclosure are exemplary and do not limit the scope of the various embodiments of this disclosure in any way. For the sake of brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are exemplary representations of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of the various embodiments of this disclosure.

[0243] Furthermore, although the detailed description of the present disclosure has been described with reference to preferred embodiments of the present disclosure, those skilled in the art or those with ordinary knowledge in the art will understand that various modifications and changes can be made to the various embodiments of the present disclosure without departing from the spirit and technical scope of the various embodiments of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the various embodiments of the present disclosure should not be limited to the contents described in the detailed description of the specification but should be determined by the claims.

[0244] Various embodiments of the present disclosure have industrial applicability in that they can mitigate the halucination phenomenon caused by inaccurate interpretation of spatial positional relationships during image data analysis in various industrial fields and improve the accuracy and consistency of analysis results.

Claims

1. A method performed by a computing device, A step of receiving an image for analysis and storing it in at least one memory; A step of loading the analysis image from at least one memory; A step of generating an analysis result for an analysis image using at least one artificial intelligence model that takes the analysis image as input, wherein at least one processor is pre-trained to output an analysis result for an image by performing a predetermined operation using a disentangling mask to control the interaction between a spatial token containing spatial information about the image and a non-spatial token containing temporal information or linguistic information, and A method comprising the step of inputting the above analysis result into at least one subsequent processing component.

2. In Paragraph 1, A method comprising at least one of the above analysis results, a natural language description including spatial-temporal relationships and contexts for the image for analysis, and information regarding object coordinates.

3. In Paragraph 1, A method further comprising the step of the above at least one subsequent processing component using the analysis result to provide at least one service among autonomous driving, robot control, medical image analysis, architectural design, and construction management.

4. In Paragraph 1, A method further comprising the step of the above at least one subsequent processing component manifesting the analysis result through at least one user interface.

5. In Paragraph 4, The step of expressing the above analysis results through at least one user interface is, A method comprising: providing, through the user interface, an image having different visual effects applied to a plurality of regions of an image related to the analysis result, based on a weight matrix calculated through a predetermined operation using the separation mask for the spatial token and the non-spatial token.

6. In Paragraph 1, A method in which at least one artificial intelligence model is pre-trained by applying different attention mechanisms to the spatial token and the non-spatial token, respectively, using the separation mask.

7. In Paragraph 6, A method in which at least one artificial intelligence model applies different attention mechanisms to the spatial token and the non-spatial token, respectively, wherein bidirectional attention is applied to the spatial token and unidirectional attention is applied to the non-spatial token.

8. In Paragraph 1, The above at least one artificial intelligence model is pre-trained by a learning method by the above at least one processor, and The above learning method is, Step of extracting a vision feature map from a training image; A step of generating global tokens, spatial tokens, and temporal tokens for the training image using the vision feature map; Step of generating text tokens from text input; A step of constructing an input sequence including the above global token, space token, time token, and text token; A step of encoding position information by applying harmonic rotation position embeddings to each token of the above input sequence; A step of performing an attention operation applying the separation mask to control the interaction between the spatial token and the non-spatial token, using the input sequence in which the above location information is encoded as input; A step of generating an output result by performing a non-linear transformation on the above attention operation result; and A method comprising the step of updating the parameters of at least one artificial intelligence model so as to minimize loss based on the output result and the correct answer data.

9. In Paragraph 8, A method in which an attention operation applying the above separation mask includes a bidirectional attention operation for the spatial token, a unidirectional attention operation for the time token, and a bidirectional attention operation for the text token.

10. In Paragraph 9 A method comprising the above attention operation, which optionally restricts the interaction in which the spatial token looks up the time token or the time token looks up the spatial token.

11. In Paragraph 8, A method wherein at least one processor performs spatial pooling on the vision feature map to generate the spatial token, and performs frame difference and temporal pooling on the vision feature map to generate the temporal token.

12. In Paragraph 8, The step of encoding location information for each token of the above input sequence is: For each token in the above input sequence, a balanced location identifier based on local order and a unique location identifier based on global order are assigned; A step of determining a rotation angle for each token based on the above-mentioned balance position identifier and unique position identifier; and A method comprising: a step of encoding position information for each token by performing a rotation operation for each token in the input sequence using a rotation angle determined for each token.

13. In Paragraph 8, The step of generating the above output result is, A method comprising the step of generating the output result by performing a non-linear transformation using a multilayer perceptron (MLP) on the result of the attention operation.

14. In Paragraph 8, A step of generating a spatial description token corresponding to information specifying a predetermined space for the above-mentioned training image; A step of generating a language token including language information describing the above-mentioned predetermined space; and The method further includes the step of generating a contextual space token by performing an operation that fuses spatial meaning through the interaction of the spatial descriptive token and the language token; The step of configuring the above input sequence is, A method comprising the step of constructing an input sequence further including the above global token, spatial token, time token, and text token and the above contextual spatial token.

15. At least one memory; and At least one processor that reads at least one instruction stored in the memory and executes an image analysis method based on separate interpretation of spatial features and temporal features; comprising The above at least one instruction is, A step of receiving an image for analysis and storing it in at least one memory; A step of loading the analysis image from at least one memory; A step of generating an analysis result for an analysis image using at least one artificial intelligence model that takes the analysis image as input, wherein at least one processor is pre-trained to output an analysis result for an image by performing a predetermined operation using a disentangling mask to control the interaction between a spatial token containing spatial information about the image and a non-spatial token containing temporal information or linguistic information, and A system comprising a command to perform the step of ingesting the above analysis result into at least one subsequent processing component.

16. In Paragraph 15, A plurality of neurons comprising an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; and at least one routing network that controls the data flow between the plurality of neurons; comprising A system further comprising a Field Programmable Gate Array (FPGA) implementation for a predetermined artificial neural network, wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to establish a transmission path for the weights.

17. In Paragraph 15, A plurality of neurons organized into an array comprising at least one register, at least one microprocessor, and at least one input; and a plurality of synapse circuits storing synapse weights that regulate the connection strength between the plurality of neurons; comprising A system comprising an Application Specific Integrated Circuit (ASIC) for a predetermined artificial neural network, wherein each of the plurality of neurons is connected to at least one other neuron through any one of the plurality of synaptic circuits.

Images