Using solid evidence to improve visual reasoning

Abstract

A processor-implemented method for generating robust reasoning for a visual reasoning task includes receiving an interleaved sequence of image and text information through a first artificial neural network (ANN). The first ANN extracts grid features from the images of the interleaved sequence of image and text information to generate a representation of the interleaved sequence of image and text information based on grid features. A second ANN maps the grid features to text regions. The second ANN extracts visual information from the interleaved sequence of image and text information based on grid features in the text regions. The second ANN determines reasoning based on the visual information. The visual information includes one or more lower-level surrogate tasks.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims priority to U.S. Patent Application No. 18 / 500,986, filed on 2 November 2023, entitled "USING GROUNDED RATIONALES TO IMPROVE VISUAL REASONING," claiming the benefit of U.S. Patent Provisional Application No. 63 / 467,159, filed on 17 May 2023, the entirety of which disclosures are expressly incorporated herein by reference.

[0002] The aspects of this disclosure generally relate to artificial neural networks, and more specifically to improving visual reasoning using solid evidence. [Background technology]

[0003] An artificial neural network may comprise an interconnected group of artificial neurons (e.g., neuron models). An artificial neural network may be a computing device, or represented as a method to be executed by a computing device. Convolutional neural networks (CNNs) are a type of feedforward artificial neural network. A convolutional neural network may contain a collection of neurons, each having a receptive field, tiling the input space together. Convolutional neural networks, such as deep convolutional neural networks, have numerous applications. These neural network architectures are used in a variety of technologies, including image recognition, speech recognition, acoustic scene classification, keyword spotting, autonomous driving, and other classification tasks.

[0004] Given the numerous useful applications of neural networks, there is a growing demand for their use in solving increasingly complex problems in further areas of application. One challenge in expanding the use of neural networks is the training process. Neural networks use large amounts of training data to learn each task. Obtaining large training datasets can be costly and time-consuming. Attempts have been made to improve the training of neural network models with fewer training examples. One area of ​​exploration is inference. Neural network inference attempts to more closely mimic the way humans learn information through deductive reasoning and other methods. Neural inference attempts to enhance neural model learning by utilizing data examples within the training dataset, as well as relevant information such as contextual information.

[0005] Visual reasoning tasks can measure the ability of learning processes to infer causal relationships, detect interactions between objects, and understand temporal dynamics based on visual cues. That is, humans can solve problems through multi-step reasoning processes in which they can extract visual information step by step by paying attention to it. However, the information density of the visual domain makes it difficult to enhance neural network training using such information. [Overview of the project]

[0006] Each aspect of this disclosure is described in its independent claims. Some aspects of this disclosure are described in its dependent claims.

[0007] In some aspects of this disclosure, a processor-implemented method includes receiving an interleaved sequence of image and text information by a first artificial neural network (ANN). The processor-implemented method also includes extracting grid features of the images in the interleaved sequence of image and text information by the first ANN in order to generate a representation of the interleaved sequence of image and text information based on grid features. The processor-implemented method further includes mapping the grid features to text regions by a second ANN. The processor-implemented method further includes extracting visual information of the interleaved sequence of image and text information by the second ANN based on grid features in the text regions. The processor-implemented method further includes determining the basis by the second ANN based on the visual information of the interleaved sequence of image and text information. The visual information includes one or more lower-level surrogate tasks.

[0008] Various aspects of this disclosure relate to an apparatus including means for receiving an interleaved sequence of image and text information by a first artificial neural network (ANN). The apparatus also includes means for extracting grid features of images in the interleaved sequence of image and text information by the first ANN in order to generate a representation of the interleaved sequence of image and text information based on grid features. The apparatus further includes means for mapping the grid features to text regions by a second ANN. The apparatus further includes means for extracting visual information of the interleaved sequence of image and text information by the second ANN based on grid features in the text regions. The apparatus further includes means for determining grounds by the second ANN based on the visual information of the interleaved sequence of image and text information. The visual information includes one or more lower-level surrogate tasks.

[0009] In some aspects of this disclosure, a non-temporary computer-readable medium recording non-temporary program code is disclosed. The program code is executed by a processor and includes program code for receiving an interleaved sequence of image and text information by a first artificial neural network (ANN). The program code also includes program code for the first ANN to extract grid features of the images in the interleaved sequence of image and text information in order to generate a representation of the interleaved sequence of image and text information based on grid features. The program code further includes program code for a second ANN to map the grid features to text regions. The program code further includes program code for the second ANN to extract visual information of the interleaved sequence of image and text information based on grid features in the text regions. The program code further includes program code for the second ANN to determine grounds based on the visual information of the interleaved sequence of image and text information. The visual information includes one or more lower-level surrogate tasks.

[0010] Various aspects of this disclosure relate to a device having at least one memory and one or more processors coupled to at least one memory. The processors are configured to receive interleaved sequences of image and text information by a first artificial neural network (ANN). The processors are also configured to extract grid features of images in the interleaved sequences of image and text information by the first ANN in order to generate a representation of the interleaved sequences of image and text information based on grid features. The processors are further configured by a second ANN to map grid features to text regions. The processors are further configured by the second ANN to extract visual information of the interleaved sequences of image and text information based on grid features in the text regions. Furthermore, the processors are configured by the second ANN to determine grounds based on the visual information of the interleaved sequences of image and text information, which includes one or more lower-level surrogate tasks.

[0011] Additional features and advantages of this disclosure are described below. Those skilled in the art will understand that this disclosure can be readily used as a basis for modifying or designing other structures to accomplish the same purposes of this disclosure. They will also recognize that such equivalent structures would not deviate from the teachings of this disclosure as described in the appended claims. Novel features that are considered unique to this disclosure will be better understood, along with further purposes and advantages, with respect to both their organization and operation, by considering the following descriptions in relation to the appended figures. However, it should be clearly understood that each of the figures is provided for illustrative and explanatory purposes only and is not intended to define the limits of this disclosure. [Brief explanation of the drawing]

[0012] The features, properties, and advantages of this disclosure will become more apparent from the detailed description below when read in conjunction with the drawings, which use similar reference numerals throughout to identify corresponding elements.

[0013] [Figure 1] This figure shows exemplary implementations of neural networks using a system-on-a-chip (SOC) including a general-purpose processor, according to several aspects of this disclosure. [Figure 2A] This figure shows neural networks in various aspects of the present disclosure. [Figure 2B] This figure shows neural networks in various aspects of the present disclosure. [Figure 2C] This figure shows neural networks in various aspects of the present disclosure. [Figure 2D] This figure shows exemplary deep convolutional networks (DCNs) according to various aspects of the present disclosure. [Figure 3] This block diagram shows exemplary deep convolutional networks (DCNs) according to various aspects of the present disclosure. [Figure 4] This block diagram shows exemplary software architectures that allow for the modularization of artificial intelligence (AI) functions, according to various aspects of this disclosure. [Figure 5] This block diagram shows exemplary architectures of a multimodal large language model (LLM) according to various aspects of the present disclosure. [Figure 6] This flowchart illustrates processor-implemented methods for determining the basis for a visual inference task using an artificial neural network (ANN) according to various aspects of this disclosure. [Modes for carrying out the invention]

[0014] The embodiments for implementing the invention described below with reference to the accompanying drawings explain various configurations and do not represent the only configuration in which the concepts described can be practiced. The embodiments for implementing the invention include specific details aimed at providing a complete understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some examples, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0015] Based on the teachings, those skilled in the art should understand that the scope of the present disclosure is intended to encompass any aspect of the present disclosure, regardless of whether it is implemented independently of any other aspect of the present disclosure or in combination with any other aspect of the present disclosure. For example, an apparatus may be implemented using any number of the described aspects, or a method may be practiced. In addition, the scope of the present disclosure is intended to include such apparatus or methods practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure disclosed may be embodied by one or more elements of the claims.

[0016] The word "exemplary" is used to mean "serving as an example, instance, or illustration." Any aspect described as "exemplary" should not necessarily be construed as being more preferable or advantageous than other aspects.

[0017] Certain aspects are described, but many variations and substitutions of these aspects fall within the scope of this disclosure. Some benefits and advantages of the preferred aspects are stated, but the scope of this disclosure is not limited to specific benefits, uses, or purposes. Rather, the aspects of this disclosure are to be widely applicable to different technologies, system configurations, networks, and protocols, some of which are shown by way of example in the following description of the figures and preferred aspects. The embodiments and drawings for carrying out the invention are not limiting but are merely illustrative of this disclosure, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0018] As described, visual reasoning tasks can measure the ability of a learning process to infer causal relationships, detect object interactions, and understand temporal dynamics based on visual cues. Visual reasoning tasks can include tasks such as tracking an object with a largely occluded field of view or answering questions involving visual reasoning. Visual reasoning tasks can include revealing causal structures, such as which object activates a machine. Due to the information density of the visual area, visual reasoning tasks can be difficult to perform.

[0019] Autoregressive large language models (LLMs) have shown positive results for various reasoning tasks such as grade school arithmetic problems and the Law School Admission Test (LSAT). However, language models for these problems only process text data to perform reasoning and generate solutions for the target tasks. Many real-world scenarios utilize reasoning across complex domains with various heterogeneous sensory inputs (e.g., perceptual cues and language).

[0020] Recently, multimodal LLMs have been proposed to solve multimodal tasks by modeling information from both text and visual domains. However, such conventional multimodal LLMs may only work well for tasks involving global visual-text relationships, such as caption generation and dialogue. However, conventional multimodal LLMs may fail for tasks that require highly detailed visual reasoning and a detailed understanding of spatiotemporal relationships between objects in a scene.

[0021] Therefore, in order to address these and other challenges, aspects of this disclosure focus on using artificial neural networks to generate reasoning for visual reasoning tasks.

[0022] Certain aspects of the subject matter described herein may be implemented to achieve one or more of the following potential benefits. In some examples, the techniques described may be usefully applied to visual reasoning tasks, including tracking objects in a heavily obstructed field of view, determining causal structures (e.g., determining which object activates a machine in an interleaved sequence of image and text data), and answering questions that utilize visual reasoning. Furthermore, the techniques described may reduce memory consumption and latency.

[0023] Figure 1 shows an exemplary implementation of a system-on-a-chip (SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to determine the basis for a visual inference task using an artificial neural network. Variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., a weighted neural network), delays, frequency bin information, and task information may be stored in memory blocks associated with a neural processing unit (NPU) 108, memory blocks associated with the CPU 102, memory blocks associated with a graphics processing unit (GPU) 104, memory blocks associated with a digital signal processor (DSP) 106, memory block 118, or distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from program memory associated with the CPU 102 or from memory block 118.

[0024] The SOC100 may also include a GPU104, a DSP106, a connectivity block 110 which may include 5G connectivity, 4G LTE connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and additional processing blocks adapted to specific functions, such as a multimedia processor 112 which can detect and recognize gestures. In one implementation, the NPU108 is implemented in the CPU102, DSP106, and / or GPU104. The SOC100 may also include a navigation module 120 which may include a sensor processor 114, image signal processors (ISPs) 116, and / or a global positioning system.

[0025] The SOC100 may be based on the ARM instruction set. In aspects of this disclosure, instructions loaded into the general-purpose processor 102 may include code for receiving an interleaved sequence of image and text information by an artificial neural network (ANN). The general-purpose processor 102 may also include code for extracting grid features of the image in the interleaved sequence of image and text information by a first ANN in order to generate a representation of the interleaved sequence of image and text information based on grid features. The general-purpose processor 102 may also include code for mapping the grid features to text regions by a second ANN. The general-purpose processor 102 may also include code for extracting visual information of the interleaved sequence of image and text information by a second ANN based on grid features in the text regions. The general-purpose processor 102 may include code for determining grounds by a second ANN based on the visual information. The visual information includes one or more lower-level surrogate tasks.

[0026] Deep learning architectures can perform object recognition tasks by learning to represent inputs at successively higher levels of abstraction within each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck in traditional machine learning. Before the advent of deep learning, machine learning methods for object recognition problems relied heavily on human-designed features, perhaps in combination with shallow classifiers. A shallow classifier could be, for example, a two-class linear classifier where a weighted sum of feature vector components is compared to a threshold to predict which class an input belongs to. Human-designed features could be templates or kernels adapted to a specific problem domain by engineers with domain expertise. In contrast, deep learning architectures can represent similar features to those that human engineers can design, but can learn through training. Furthermore, deep networks can learn to represent and recognize novel types of features that humans may not have considered.

[0027] Deep learning architectures can learn feature hierarchies. When presented with visual data, for example, the first layer may learn to recognize relatively simple features such as edges in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at a specific frequency. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0028] Deep learning architectures can perform particularly well when applied to problems with natural hierarchical structures. For example, classifying motorized vehicles may benefit from first learning to recognize wheels, windshields, and other features. These features can then be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0029] Neural networks can be designed using various connection patterns. In a feedforward network, information is passed from lower layers to higher layers, with each neuron in a given layer communicating with neurons in higher layers. As mentioned above, a hierarchical representation can be constructed within the consecutive layers of a feedforward network. Neural networks can also have recursive or feedback (also called top-down) connections. In recursive connections, the output from a neuron in a given layer can be transmitted to another neuron in the same layer. Recursive architectures can be useful when recognizing patterns that span two or more chunks of input data delivered to the neural network in a sequence. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be useful when the recognition of a higher-level concept can help identify specific lower-level features of the input.

[0030] The connections between layers of a neural network may be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, neurons in the first layer may transmit their output to any neuron in the second layer, and as a result, each neuron in the second layer receives input from any neuron in the first layer. Figure 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, neurons in the first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in the layer has the same or similar connection pattern but has a connection strength that can have different values ​​(e.g., 210, 212, 214, and 216). Since higher-layer neurons within a given region can receive inputs that, through training, are tuned to the characteristics of a limited portion of the total inputs to the network, local connectivity patterns can create spatially distinct receptive fields within the higher layers.

[0031] An example of a locally connected neural network is a convolutional neural network. Figure 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strength associated with the input for each neuron in the second layer (e.g., 208) is shared. Convolutional neural networks may be suitable for problems where the spatial location of the input is meaningful.

[0032] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of DCN200, designed to recognize visual features from an image 226 input from an image capture device 230, such as an in-vehicle camera. In this example, DCN200 can be trained to identify traffic signs and the numbers written on them. Of course, DCN200 may also be trained for other tasks, such as identifying lane markings or traffic signals.

[0033] DCN200 can be trained using supervised learning. During training, DCN200 may be presented with images such as image 226 of a speed limit sign, and then a forward pass may be computed to produce output 222. DCN200 may include a feature extraction section and a classification section. Upon receiving image 226, the convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to produce a first set 218 of feature maps. For example, the convolutional kernel for convolutional layer 232 may be a 5x5 kernel that produces a 28x28 feature map. In this example, four different convolutional kernels were applied to image 226 in convolutional layer 232, since four different feature maps are produced in the first set 218 of feature maps. Convolutional kernels are also sometimes called filters or convolutional filters.

[0034] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218; that is, the size of the second set of feature maps 220, such as 14x14, is smaller than the size of the first set of feature maps 218, such as 28x28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0035] In the example in Figure 2D, a second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may contain a number corresponding to a possible feature of image 226, such as "label", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Therefore, the output 222 of DCN200 may be the probability that image 226 contains one or more features.

[0036] In this example, the probabilities in output 222 for "label" and "60" are higher than the probabilities for other outputs in output 222 such as "30", "40", "50", "70", "80", "90", and "100". Before training, the outputs 222 generated by DCN200 are likely to be inaccurate. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth of image 226 (e.g., "label" and "60"). The weights of DCN200 can then be adjusted so that the output 222 of DCN200 is closer to the target output.

[0037] To adjust the weights, the learning algorithm can calculate a gradient vector for the weights. The gradient can indicate the amount by which the error will increase or decrease if the weights are adjusted. In the top layer, the gradient can directly correspond to the weight values ​​connecting the activated neurons in the second-to-last layer to the neurons in the output layer. In lower layers, the gradient may depend on the weight values ​​and the error gradient calculated in the upper layers. The weights can then be adjusted to minimize the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward path" through the neural network.

[0038] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate for the entire system stops decreasing or until the error rate reaches a target level. After training, DCN200 may be presented with new images, and a forward pass through DCN200 may yield an output 222 that can be considered DCN200's inference or prediction.

[0039] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs can be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn a probability distribution over a set of inputs. Because RBMs can learn a probability distribution without any information about which class each input should belong to, they are frequently used in unsupervised learning. Using a hybrid paradigm of unsupervised and supervised learning, the lower RBM of a DBN can be trained unsupervised and can function as a feature extractor, while the upper RBM can be trained supervised (on a combined distribution of inputs from previous layers and target classes) and can function as a classifier.

[0040] DCNs are convolutional networks composed of additional pooling and normalization layers. DCNs achieve state-of-the-art performance for a wide range of tasks. DCNs can be trained using supervised learning, where both the input and output objectives are known for a large number of samples, and the network weights are modified using gradient descent.

[0041] A DCN may also be a feedforward network. In addition, as described above, connections from neurons in the first layer of a DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of a DCN can be leveraged for high-speed processing. The computational load of a DCN can be much smaller than, for example, the computational load of a neural network of similar size that includes recursive or feedback connections.

[0042] The processing of each layer of a convolutional network may be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on that input may be considered three-dimensional, having two spatial dimensions along the image axes and a third dimension capturing color information. The output of the convolutional connections may be considered as forming a feature map in subsequent layers, where each element of the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map may be further processed using rectification, max(0,x), or other nonlinearities. Values ​​from adjacent neurons may be further pooled, which corresponds to downsampling and can result in additional local invariance and dimensionality reduction. Normalization corresponding to whitening may also be applied through lateral inhibition between neurons in the feature map.

[0043] Figure 3 is a block diagram of DCN350. DCN350 may include several different types of layers based on connectivity and weight sharing. As shown in Figure 3, DCN350 includes convolutional blocks 354A and 354B. Each of convolutional blocks 354A and 354B may consist of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0044] Although only two of the convolutional blocks 354A and 354B are shown, this disclosure is not limited in that way, and instead, any number of convolutional blocks 354A and 354B may be included in the DCN350 according to design preferences.

[0045] The convolutional layer 356 may include one or more convolutional filters that can be applied to the input data to generate a feature map. The normalization layer 358 may normalize the output of the convolutional filters. For example, the normalization layer 358 may provide whitening or lateral suppression. The maximal pooling layer 360 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0046] For example, the DCN's parallel filter bank may be loaded onto the CPU 102 or GPU 104 of the SOC 100 (e.g., Figure 1) to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the DCN 350 may have access to other processing blocks that may reside on the SOC 100, such as the sensor processor 114 and the navigation module 120, which are dedicated to sensors and navigation, respectively.

[0047] DCN350 may also include one or more fully connected layers 362 (FC1 and FC2). DCN350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of DCN350 are weights (not shown) to be updated. The output of each layer (e.g., 356, 358, 360, 362, 364) may serve as input to one of the subsequent layers (e.g., 356, 358, 360, 362, 364) in DCN350 to learn a hierarchical feature representation from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) supplied in the first of the convolutional blocks 354A. The output of DCN350 is a classification score 366 for the input data 352. The classification score 366 may also be a set of probabilities, each of which is the probability that the input data contains a feature from the set of features.

[0048] Figure 4 is a block diagram showing an exemplary software architecture 400 in which artificial intelligence (AI) functions can be modularized. Using this architecture 400, an application can be designed that enables various processing blocks of the SOC 420 (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) to support the determination of the basis for a visual inference task using an artificial neural network for an AI application 402, according to aspects of this disclosure. The architecture 400 may be included in a computing device such as a smartphone, for example.

[0049] The AI ​​application 402 may be configured to invoke functions defined in user space 404, for example, which can provide scene detection and recognition indicating the location where a computing device, including architecture 400, is currently operating. The AI ​​application 402 may configure its microphone and camera differently depending on whether the scene to be recognized is an office, an auditorium, a restaurant, or an outdoor setting such as a lake. The AI ​​application 402 may make requests to compiled program code associated with libraries defined in the AI ​​function application programming interface (API) 406. These requests may ultimately rely on the output of a deep neural network configured to provide inference responses based on video and positioning data, for example.

[0050] The AI ​​application 402 may also have further access to a runtime engine 408, which may be compiled code of a runtime framework. The AI ​​application 402 may prompt the runtime engine 408 to perform inference, for example, at specific time intervals or triggered by events detected by the user interface of the AI ​​application 402. The runtime engine 408 may then signal an operating system in the operating system (OS) space 410, such as a kernel 412 running on the SOC 420, when it is ready to provide an inference response. In some examples, kernel 412 may be a LINUX kernel. The operating system may then cause a continuous relaxation of quantization to be performed on the CPU 422, DSP 424, GPU 426, NPU 428, or any combination thereof. The CPU 422 may be accessed directly by the operating system, while the other processing blocks may be accessed through drivers, such as drivers 414, 416, or 418 for the DSP 424, GPU 426, or NPU 428, respectively. In an exemplary example, the deep neural network may be configured to run on a combination of processing blocks such as CPU422, DSP424, and GPU426, or it may run on NPU428.

[0051] Aspects of this disclosure relate to generating evidence for a visual reasoning task using an artificial neural network (ANN). Aspects of this disclosure may break down a complex visual reasoning task into simplified steps. The ANN model may be trained on a surrogate task in such a way that it generates evidence about spatial and / or spatiotemporal relationships present in a stream of data containing visual information.

[0052] According to aspects of this disclosure, a large-scale language model (LLM) can be trained to solve visual reasoning tasks by breaking down complex visual reasoning problems into simpler ones in a stepwise process, while simultaneously mapping important visual information from the visual domain to the text domain. To guide the LLM in extracting relevant spatial and temporal visual features, spatial grid-level visual features may be used in conjunction with evidence-based training.

[0053] In some embodiments, high-level symbolic inference capabilities in neural networks can take the form of emergent techniques. Rather than being addressed through architectural considerations (e.g., by having dedicated processing modules for certain subtasks), these capabilities can emerge in ANNs in response to training on many important related training tasks. According to embodiments of this disclosure, high-level inference and perception capabilities can be utilized as a suitable set of important surrogate tasks for training an ANN. Perception capabilities can expand the set of tasks that a given ANN architecture can solve. For example, perception capabilities can enable computationally simple ANN architectures to perform a wide range of tasks, and in some embodiments, complex tasks.

[0054] In some embodiments, artificial neural network architectures can be designed to map to power-efficient accelerator hardware. For example, an ANN that processes input through relatively simple and homogeneous computations (e.g., a large array of convolutional operations combined with a cross-attention layer to map to a language model) can be trained to perform complex perceptual inference tasks. Thus, embodiments of this disclosure can reduce the demand for use case information that involves complex compiler functions and extensive operator support.

[0055] Consider a visual reasoning problem where the goal is to correctly answer whether a query object will activate a "Blicket" detector. The blicket detector emits light and plays music when an object is placed on it. Humans may solve this problem through a multi-stage reasoning process by paying attention to visual information and extracting it step by step, using lower-level visual abilities such as object recognition and re-identification. For example, one strategy a human may follow to solve such a problem is to first read the question, examine the scene, create an overview of the current object, as well as any important lower-level visual information, remember the important information along the way, and finally state the answer based on the extracted information. Such a reasoning process may help address both the complexity of the task and the need to filter rich visual data for important information. In short, such a reasoning process can be thought of as involving intermediate subtasks that may be called "look, remember, reason (LRR)," namely, searching for important visual cues, remembering the important cues along the way, and aggregating the collected information to reason and determine the answer. In various aspects of this disclosure, a unimodal large language model (LLM) for text may be augmented to perform general-purpose multimodal visual reasoning by extending the LLM with lower levels of visual aptitude.

[0056] For example, a ready-made LLM may possess lower levels of visual competence to solve a wide range of visual reasoning tasks. In some embodiments, the LLM may be indirectly trained using surrogate tasks expressed in natural language for generating important evidence, which may be based on visual input and follow a “see, remember, reason” reasoning process. Adapter modules may be provided to enable top-down attention controlled by the LLM. Thus, an LLM configured to be considered a general-purpose LRR model can perform a variety of visual reasoning tasks, including (but not limited to) spatial reasoning, temporal reasoning, and causal visual reasoning.

[0057] To enable visual reasoning by utilizing highly expressive LLMs, a “See, Remember, Reason (LRR)” framework is proposed. The LRR model may be based on a pre-trained LLM backbone with an additional cross-attention layer to enable processing of multimodal inputs. Evidence derived from multimodal inputs may be used to address the challenges presented by visual reasoning problems. Unlike conventional approaches, the evidence may additionally include lower-level visual surrogate tasks expressed in natural language to enable the visual reasoning task. The evidence may be supported by a top-down attention mechanism that allows higher-level concepts to modulate perceptual pathways.

[0058] Figure 5 is a block diagram showing exemplary architectures of a multimodal LLM model 500 according to various aspects of the present disclosure. Referring to Figure 5, the multimodal LLM model (which may also be called a “see, remember, infer (LRR)” model) 500 may include a convolutional neural network (CNN) 502 and an LLM 506.

[0059] The LRR model 500 can be implemented using an autoregressive framework. The LRR model 500 can be constructed using the parameter θ. The LRR model 500 can receive interleaved streams of visual input 520 and text input 522. The visual input 520 is,

number

number

[0060] In some embodiments, CNN 502 may include, for example, a residual network (ResNet). CNN 502 may receive a visual input I of an interleaved sequence of visual inputs 520 and / or text inputs 522. CNN 502 may extract low-level grid-level visual features from the visual input I (520) to store low-level visual information. CNN 502 may be configured to ensure that the LRR model 500 can be broadly applicable across various visual inference problems. CNN 502 may be configured to handle the visual input sequence

number

number

number

number

[0061] LLM 506 may be, for example, a pre-trained language model, or even a ready-made LLM. The parameters of LLM 506 may be initialized from the pre-trained LLM to leverage the existing inference capabilities of such a language model. However, in some embodiments, LLM 506 may be trained on text alone. Visual inference tasks may rely on extracting visual information about the spatial and temporal relationships between objects in a visual input (e.g., a scene). In a multimodal setting, information from the visual input I(520) may be mapped to the text-based representation space of LLM 506. One challenge raised in multimodal visual inference methods is that the information density of images can be very high compared to text (e.g., tokens).

[0062] LLM 506 may include a self-attention block 508 and a top-down attention block 510. Self-attention block 508 may include self-attention layers 514a and 514b. Self-attention block 508 may receive text input 522 (e.g., tokens) in the first self-attention layer 514a. Self-attention layers 514a and 514b of LLM 506 may process the input text 522 sequentially by applying self-attention. Self-attention can be thought of as a mechanism that identifies parts of the input that deserve attention or attention by comparing each element of the input (e.g., multiscale visual features) with other elements of the input (e.g., other multiscale visual features). In other words, self-attention allows elements of the input to interact with each other.

[0063] The first self-attention layer 514a may be called the first embedding layer of LLM 506 and may encode text input 522 (e.g., tokens), while subsequent layers of LLM 506 may encode increasingly richer and more information-density representations, and then increasingly global information. Thus, higher embedding layers in the hierarchy of the top-down attention mechanism may guide the information extraction process from the visual input I(520).

[0064] The top-down attention block 510 may function as a mechanism to enable the LLM 506 to directly extract lower-level visual information from grid-level features of the visual input I(520). The top-down attention block 510 may include cross-attention layers 516a, 516b. The cross-attention layers 516a, 516b may be interleaved between self-attention layers in higher layers of the LLM 506 (e.g., after the self-attention block 508). The cross-attention layers 516a, 516b are hidden states {h 1 ,...,h m You may also utilize a rich hierarchical representation encoded in},

number

[0065] The LRR model 500 utilizes cross-attention layers 516a and 516b at higher levels {k,...,m} of the hierarchical representation space of LLM 506 to extract grid-level visual features by CNN 502.

number

number

number

number

number

[0066] In some embodiments, positional embedding is used to store spatial information for each grid element.

number

Number

Number

Number

Number

Number

Number

Number

number

[0067] Therefore, the LRR model 500 can leverage the flexibility of the language model to represent a variety of lower-level visual tasks through language in a general setting. Unlike conventional visual reasoning approaches, aspects of this disclosure can utilize lower-level visual information to generate grounds for visual reasoning tasks. Lower-level visual information may include surrogate tasks. Examples of such lower-level visual information may include (but are not limited to) detecting, describing, identifying, and / or enumerating objects present in a scene, tracking or re-identifying objects in a video, or understanding the spatial relationships between multiple objects in a scene. If the visual reasoning problem involves lower-level skills such as object recognition and spatial reasoning, grounds can be determined using surrogate tasks. In some aspects, the grounds for a task may be based on tracking identifiers (IDs), such as location or timestamps.

[0068] In some examples, for lower levels of object recognition skills, a surrogate task may be defined that explicitly enumerates objects in a scene of visual input. Similarly, for spatial reasoning skills, the rationale may be designed so that a surrogate task can be defined that explicitly enumerates each of the objects to the left, right, in front of, and behind a particular target object in a scene of visual input. Thus, the LRR model 500 may enable understanding the spatial relationships of a target object to other objects in the scene.

[0069] Furthermore, for visual reasoning problems involving tracking or re-identification, surrogate tasks can be defined in the evidence to predict location or identify one or more target objects across video frames. By including such lower-level visual tasks in the evidence, solutions to visual reasoning tasks can remain usefully within the context window of LLM 506, so that the lower-level visual tasks can be “remembered” by LLM 506 and utilized to “infer” and solve subsequent tasks. Thus, LRR model 500 can be trained with surrogate tasks to break down complex visual reasoning tasks into simpler steps and to generate evidence 524 about spatial and / or spatiotemporal relationships present in the visual information related to the visual reasoning tasks.

[0070] The LRR model 500 can be trained by maximizing the log-likelihood of the next text input 522 (e.g., a token), given an interleaved sequence of previous visual inputs and text, as given by the following equation:

number

number

number

[0071] Figure 6 is a flowchart illustrating a processor-implemented method 600 for determining the basis of a visual inference task using an artificial neural network (ANN), according to various aspects of the present disclosure. Referring to Figures 1 to 6, the processor-implemented method 600 may be carried out by one or more processors such as a CPU (e.g., 102, 422), a GPU (e.g., 104, 426), a DSP (e.g., 106, 424), and / or an NPU (e.g., 108, 428).

[0072] In block 602, the processor receives interleaved sequences of image and text information through a first artificial neural network (ANN), where the visual information includes one or more lower-level surrogate tasks. The first ANN may comprise a convolutional neural network, such as the DCN 350 shown in Figure 3. In some embodiments, the text information may include tokenized text data.

[0073] In block 604, the processor extracts grid features of the images in the interleaved sequence of images and text information by a first ANN in order to generate a representation of the interleaved sequence of images and text information based on grid features. The grid features may include spatiotemporal information of the images in the interleaved sequence of images and text information.

[0074] In block 606, the processor maps grid features to text regions using a second ANN. The second ANN may comprise, for example, a language model. For example, as illustrated with reference to Figure 5, the LRR model 500 utilizes cross-attention layers 516a, 516b at higher levels {k,...,m} of the hierarchical representation space of LLM 506 to extract grid-level visual features by CNN 502.

number

number

number

number

number

[0075] In block 608, the processor extracts visual information from an interleaved sequence of image and text information using a second ANN based on grid features in the text region. The second ANN may use cross-attention to generate a hidden representation of the query that induces the extraction of visual information from the interleaved sequence of image and text information.

[0076] In block 610, the processor determines the basis by a second ANN based on the visual information. The visual information includes one or more lower-level surrogate tasks. In some embodiments, the visual information may further include a constitutive aspect of the scene shown in an interleaved sequence of images, or a higher-level task inferred from one or more lower-level surrogate tasks. For example, the constitutive aspect may include, for example, an object recognition task, an object re-identification task, an object tracking task, or an object relation task. In some embodiments, the basis may include, for example, a tracking identifier (ID), such as location or a timestamp. In various embodiments, the basis may be used, for example, to determine a solution or answer to a prompt and / or to perform a visual inference task.

[0077] Implementation examples are provided in the following numbered clauses. 1. At least one memory, At least one processor coupled to at least one memory, It is equipped with at least one processor, An interleaved sequence of image and text information is received by a first artificial neural network (ANN), To generate a representation of an interleaved sequence of image and text information based on grid features, the grid features of the image in the interleaved sequence of image and text information are extracted by a first ANN, The second ANN maps grid features to text regions. Based on grid features in the text region, the visual information of the interleaved sequence of image and text information is extracted by a second ANN. A device configured to determine a basis by a second ANN based on visual information, which includes an interleaved sequence of image and text information, and which comprises one or more lower-level surrogate tasks. 2. The apparatus of Clause 1, wherein the visual information further includes a constitutive aspect of a scene shown in an interleaved sequence of images, or a higher-level task inferred from one or more lower-level surrogate tasks, the constitutive aspect including one or more of the following: an object recognition task, an object re-identification task, an object tracking task, or an object relation task. 3. The apparatus of Clause 1 or 2, wherein at least one processor is further configured by a second ANN to perform evidence-based visual reasoning tasks. 4. A device according to any of the clauses 1 to 3, wherein the grid features include spatiotemporal information of images in an interleaved sequence of image and text information. 5. Any apparatus of Clauses 1 to 4, wherein at least one processor is further configured to generate a hidden representation of a query by a second ANN that uses cross-attention to induce the extraction of visual information from an interleaved sequence of image and text information. 6. Any device according to clauses 1 to 5, wherein the first ANN comprises a convolutional neural network or a vision transformer-based model, and the second ANN comprises a language model. 7. Any apparatus of Clauses 1 to 6, wherein at least one processor is further configured to determine the basis by a second ANN based on the maximum log-likelihood of the next token of tokenized text data based on at least one previous token of the tokenized text data in an interleaved sequence of image and text information. 8. Any device from clauses 1 to 7, in which the decision of whether or not to generate evidence is made probabilistically during training. 9. A processor-implemented method carried out by at least one processor, The interleaved sequence of image and text information is received by a first artificial neural network (ANN), To generate a representation of an interleaved sequence of image and text information based on grid features, the grid features of the image in the interleaved sequence of image and text information are extracted by a first ANN, The second ANN maps grid features to text regions, Based on grid features in the text region, visual information of an interleaved sequence of image and text information is extracted by a second ANN, Visual information of an interleaved sequence of image and text information, including one or more lower-level surrogate tasks, to determine the basis by a second ANN based on the visual information, Methods that include... 10. A method by which the processor of Clause 9 implements the visual information, further comprising a constructive aspect of a scene shown in an interleaved sequence of images, or a higher-level task inferred from one or more lower-level surrogate tasks, wherein the constructive aspect comprises one or more of the following tasks: an object recognition task, an object re-identification task, an object tracking task, or an object relation task. 11. A method implemented in the processor of Clause 9 or 10, further comprising performing an evidence-based visual reasoning task by a second ANN. 12. A method by which a grid feature is implemented in a processor of any of the clauses 9 to 11, including spatiotemporal information of images in an interleaved sequence of image and text information. 13. A method implemented in any of the processors of Clauses 9 to 12, further comprising generating a hidden representation of a query by a second ANN that uses cross-attention to induce the extraction of visual information from an interleaved sequence of image and text information. 14. A method implemented on a processor according to any of the clauses 9 to 13, wherein the first ANN comprises a convolutional neural network or a vision transformer-based model, and the second ANN comprises a language model. 15. A method implemented in any of the processors of Clauses 9 to 14, further comprising determining the basis by a second ANN based on the maximum log-likelihood of the next token of tokenized text data based on at least one previous token of the tokenized text data in an interleaved sequence of image and text information. 16. A method implemented in any of the processors of clauses 9 to 15, in which the decision of whether or not to generate grounds is made probabilistically during training. 17. A non-temporary computer-readable medium on which program code is recorded, wherein the program code is executed by a processor, Program code for receiving interleaved sequences of image and text information using a first artificial neural network (ANN), A program code for extracting the grid features of the image in the interleaved sequence of image and text information by a first ANN, in order to generate a representation of the interleaved sequence of image and text information based on grid features, The second ANN provides program code for mapping grid features to text regions, Program code for extracting visual information from an interleaved sequence of image and text information using a second ANN, based on grid features in the text region. Visual information of an interleaved sequence of image and text information, comprising one or more lower-level surrogate tasks, and program code for determining the basis by a second ANN based on the visual information, A non-temporary computer-readable medium comprising [a specific feature]. 18. Non-temporary computer-readable media of Clause 17, wherein the visual information further includes a constitutive aspect of a scene presented in an interleaved sequence of images, or a higher-level task inferred from one or more lower-level surrogate tasks, the constitutive aspect including one or more of the following: object recognition tasks, object re-identification tasks, object tracking tasks, or object relation tasks. 19. A non-temporary computer-readable medium of Clause 17 or 18, comprising program code for performing a reasoned visual reasoning task by a second ANN. 20. A non-temporary computer-readable medium of any of the clauses 17 to 19, in which the grid features include spatiotemporal information of images in an interleaved sequence of image and text information. 21. A non-temporary computer-readable medium of any of the clauses 17 to 20, comprising program code for generating a hidden representation of a query by a second ANN that uses cross-attention to induce the extraction of visual information from an interleaved sequence of image and text information. 22. A non-temporary computer-readable medium of any of the terms 17 to 21, wherein the first ANN comprises a convolutional neural network or a vision transformer-based model, and the second ANN comprises a language model. 23. A non-temporary computer-readable medium of any of the clauses 17 to 22, comprising program code for determining the basis by a second ANN based on the maximum log-likelihood of the next token of tokenized text data based on at least one previous token of tokenized text data in an interleaved sequence of image and text information. twenty four. A means for receiving an interleaved sequence of image and text information by a first artificial neural network (ANN), In order to generate a representation of an interleaved sequence of image and text information based on grid features, means for extracting the grid features of the image of the interleaved sequence of image and text information by a first ANN, The second ANN provides a means for mapping grid features to text regions, A means for extracting visual information of an interleaved sequence of image and text information by a second ANN based on grid features in the text region, Visual information of an interleaved sequence of image and text information, comprising one or more lower-level surrogate tasks, means for determining the basis by a second ANN based on the visual information, A device equipped with the following features. 25. The apparatus of Clause 24, wherein the visual information further includes a constitutive aspect of a scene presented in an interleaved sequence of images, or a higher-level task inferred from one or more lower-level surrogate tasks, the constitutive aspect including one or more of the following: an object recognition task, an object re-identification task, an object tracking task, or an object relation task. 26. The apparatus of Clause 24 or 25, further comprising means for performing evidence-based visual reasoning tasks by a second ANN. 27. An apparatus according to any of the clauses 24 to 26, wherein the grid features include spatiotemporal information of images in an interleaved sequence of image and text information. 28. Any apparatus of clauses 24 to 27, further comprising means for generating a hidden representation of a query by a second ANN that uses cross-attention to induce the extraction of visual information from an interleaved sequence of image and text information. 29. Any apparatus according to clauses 24 to 28, wherein the first ANN comprises a convolutional neural network or a vision transformer-based model, and the second ANN comprises a language model. 30. Any apparatus of clauses 24 to 29, further comprising means for determining the basis by a second ANN based on the maximum log-likelihood of the next token of tokenized text data based on at least one previous token of tokenized text data in an interleaved sequence of image and text information.

[0078] The various operations of the methods described above may be carried out by any suitable means capable of performing the corresponding functions. These means may include, but are not limited to, various hardware and / or software components and / or modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations are shown in the figures, those operations may have corresponding relative means-plus-function components with similar numbering.

[0079] When used, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, computing, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. Additionally, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and furthermore, “determining” may include resolving, selecting, choosing, and establishing.

[0080] When used, the phrase "at least one of" in an enumeration of items refers to any combination of those items that includes a single member. For example, "at least one of a, b, or c" is intended to include a, b, c, ab, ac, bc, and abc.

[0081] The various exemplary logic blocks, modules, and circuits described in this disclosure may be implemented or carried out using general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices (PLDs), discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0082] Steps of the methods or algorithms described in this disclosure may be embodied directly in hardware, in software modules executed by a processor, or in a combination of the two. Software modules may reside in any form of storage medium known in the art. Some examples of storage mediums that may be used include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, and CD-ROMs. Software modules may comprise a single instruction or multiple instructions and may be distributed across several different code segments, between different programs, and across multiple storage mediums. Storage mediums may be coupled to a processor so that the processor can read information from and write information to the storage medium. Alternatively, storage mediums may be integrated with the processor.

[0083] The disclosed methods include one or more steps or actions for achieving the described method. The steps and / or actions of those methods may be interchanged with one another without departing from the claims. In other words, unless a specific order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, may be modified without departing from the claims.

[0084] The described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, the exemplary hardware configuration may include a processing system within the device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnection buses and bridges, depending on the specific application of the processing system and the overall design constraints. The bus may connect various circuits, including processors, machine-readable media, and bus interfaces. The bus interface may, among other things, be used to connect a network adapter to the processing system via the bus. The network adapter may be used to implement signal processing functions. In some embodiments, a user interface (e.g., a keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also connect various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, but these circuits are well known in the art and therefore will not be described further.

[0085] A processor may be responsible for managing the bus and general processing, including the execution of software stored on machine-readable media. A processor may be implemented using one or more general-purpose and / or dedicated processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits capable of executing software. Software is broadly interpreted as meaning instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or by other names. Machine-readable media may include, for example, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage media, or any combination thereof. Machine-readable media may be embodied in computer program products. Computer program products may include packaging materials.

[0086] In hardware implementations, machine-readable media may be part of a processing system separate from the processor. However, as will be readily apparent to those skilled in the art, machine-readable media or any part thereof may be outside the processing system. For example, machine-readable media may include transmission lines, data-modulated carriers, and / or computer products separate from the device, all of which may be accessed by the processor through a bus interface. Alternatively, or additionally, machine-readable media or any part thereof may be integrated into the processor, as may be the case with caches and / or general-purpose register files. The various components discussed may be described as having a specific location, such as local components, but they may also be configured in various ways, such as several components configured as part of a distributed computing system.

[0087] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functions and external memory providing at least a portion of a machine-readable medium, all connected together with other auxiliary circuits via an external bus architecture. Alternatively, the processing system may comprise one or more neuromorphological processors for implementing the described neuron model and neural system model. Another alternative is that the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, a bus interface, a user interface, auxiliary circuits, and at least a portion of a machine-readable medium integrated on a single chip, or using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how to best implement the functions described for the processing system, depending on the specific application and the overall design constraints imposed on the entire system.

[0088] A machine-readable medium may comprise several software modules. These software modules, when executed by a processor, contain instructions that cause the processing system to perform various functions. Software modules may include transmit modules and receive modules. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, a software module may be loaded from a hard drive into RAM when a trigger event occurs. During the execution of a software module, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a general-purpose register file for execution by the processor. When referring to the functions of a software module below, it will be understood that such functions are implemented by the processor when instructions from that software module are executed. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of processors, computers, machines, or other systems implementing such aspects.

[0089] When implemented in software, a function may be stored on or transmitted via a computer-readable medium as one or more instructions or codes. The computer-readable medium includes both computer storage and communication media, including any medium that facilitates the transfer of computer programs from one location to another. The storage medium may be any available medium accessible by a computer. Such computer-readable media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible by a computer. Additionally, any connection is appropriately referred to as computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. When used, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray® disc, where disk typically reproduces data magnetically and disc optically using a laser. Thus, in some embodiments, computer-readable medium may include non-temporary computer-readable medium (e.g., tangible medium). In addition, in other embodiments, the computer-readable medium may include a temporary computer-readable medium (e.g., a signal). The above combinations should also be included within the scope of the computer-readable medium.

[0090] Accordingly, some embodiments may comprise a computer program product that performs the described operation. For example, such a computer program product may comprise a computer-readable medium storing (and / or encoding) instructions, the instructions being executable by one or more processors to perform the described operation. In some embodiments, the computer program product may include packaging material.

[0091] Furthermore, it should be understood that modules and / or other suitable means for implementing the described methods and techniques may, as appropriate, be downloaded and / or otherwise obtained by user terminals and / or base stations. For example, such devices may be coupled to a server to facilitate the transfer of means for implementing the described methods. Alternatively, the various described methods may be provided via storage means so that user terminals and / or base stations can obtain the various methods by coupling or providing storage means (e.g., physical storage media such as RAM, ROM, compact disks (CDs), or floppy disks) to the device. Moreover, any other suitable techniques for providing the described methods and techniques to the device may be utilized.

[0092] It should be understood that the claims are not limited to the exact configurations and components illustrated above. Various modifications, changes, and variations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the claims.

Claims

1. At least one memory, At least one processor coupled to the at least one memory, The at least one processor is An interleaved sequence of image and text information is received by a first artificial neural network (ANN), In order to generate a representation of the interleaved sequence of the image and the text information based on grid features, the grid features of the image in the interleaved sequence of the image and the text information are extracted by the first ANN, The second ANN maps the grid features to the text region, Based on the grid features in the text region, the second ANN extracts the visual information of the interleaved sequence of the image and the text information. The visual information of the interleaved sequence of images and the text information, comprising one or more lower-level surrogate tasks, on which the second ANN determines the basis, It is structured in such a way. Device.

2. The apparatus according to claim 1, wherein the visual information further includes a constitutive aspect of a scene shown in the interleaved sequence of the image, or a higher-level task inferred from one or more lower-level surrogate tasks, the constitutive aspect includes one or more of an object recognition task, an object re-identification task, an object tracking task, or an object relation task.

3. The apparatus according to claim 1, wherein the at least one processor is further configured to perform a visual inference task based on the grounds by the second ANN.

4. The apparatus according to claim 1, wherein the grid features include spatiotemporal information of the image of the interleaved sequence of the image and the text information.

5. The apparatus according to claim 1, wherein the at least one processor is further configured to use cross-attention to generate a hidden representation of a query by the second ANN that induces the extraction of the visual information of the interleaved sequence of the image and the text information.

6. The apparatus according to claim 1, wherein the first ANN comprises a convolutional neural network or a vision transformer-based model, and the second ANN comprises a language model.

7. The apparatus according to claim 1, wherein the at least one processor is further configured to determine the basis by the second ANN based on the maximum log-likelihood of the next token of the tokenized text data based on at least one previous token of the tokenized text data in the interleaved sequence of the image and the text information.

8. The apparatus according to claim 1, wherein the decision of whether or not to generate the aforementioned basis is made probabilistically during training.

9. A processor-implemented method that is carried out by at least one processor, The interleaved sequence of image and text information is received by a first artificial neural network (ANN), In order to generate a representation of the interleaved sequence of the image and the text information based on grid features, the grid features of the image in the interleaved sequence of the image and the text information are extracted by the first ANN, The second ANN maps the grid features to the text area, Based on the grid features in the text region, the second ANN extracts the visual information of the interleaved sequence of the image and the text information. The visual information of the interleaved sequence of images and the text information, comprising one or more lower-level surrogate tasks, wherein the second ANN determines the basis based on the visual information, Methods that include...

10. The method according to claim 9, wherein the visual information further includes a constitutive aspect of a scene shown in the interleaved sequence of the images, or a higher-level task inferred from one or more lower-level surrogate tasks, the constitutive aspect includes one or more of an object recognition task, an object re-identification task, an object tracking task, or an object relation task.

11. The method according to claim 9, further comprising performing a visual reasoning task based on the grounds using the second ANN.

12. The method according to claim 9, wherein the grid feature includes spatiotemporal information of the image of the interleaved sequence of the image and the text information.

13. The method according to claim 9, further comprising using cross-attention to generate a hidden representation of a query by the second ANN that induces the extraction of the visual information of the interleaved sequence of the image and the text information.

14. The method according to claim 9, wherein the first ANN comprises a convolutional neural network or a vision transformer-based model, and the second ANN comprises a language model.

15. The method of claim 9, further comprising determining the basis by the second ANN based on the maximum log-likelihood of the next token of the tokenized text data based on at least one previous token of the tokenized text data in the interleaved sequence of the image and the text information.

16. The method according to claim 9, wherein the decision of whether or not to generate the aforementioned basis is made probabilistically during training.

17. A non-temporary computer-readable medium on which program code is recorded, wherein the program code is executed by a processor, and Program code for receiving an interleaved sequence of image and text information using a first artificial neural network (ANN), A program code for extracting the grid features of the image in the interleaved sequence of the image and text information by the first ANN in order to generate a representation of the interleaved sequence of the image and text information based on grid features, The second ANN provides program code for mapping the grid features to a text area, A program code for extracting visual information of the interleaved sequence of the image and the text information by the second ANN based on the grid features in the text region, Visual information of the interleaved sequence of images and text information, comprising one or more lower-level surrogate tasks, and program code for determining a basis by the second ANN based on the visual information, A non-temporary computer-readable medium comprising [a specific feature].

18. The non-temporary computer-readable medium according to claim 17, wherein the visual information further includes a constitutive aspect of a scene shown in the interleaved sequence of the image, or a higher-level task inferred from one or more lower-level surrogate tasks, the constitutive aspect includes one or more of an object recognition task, an object re-identification task, an object tracking task, or an object relation task.

19. The non-temporary computer-readable medium according to claim 17, wherein the program code comprises program code for performing a visual reasoning task based on the grounds by the second ANN.

20. The non-temporary computer-readable medium according to claim 17, wherein the grid features include spatiotemporal information of the image of the interleaved sequence of the image and the text information.

21. The non-temporary computer-readable medium according to claim 17, wherein the program code comprises program code for the second ANN to generate a hidden representation of a query that uses cross-attention to induce the extraction of the visual information of the interleaved sequence of the image and the text information.

22. The non-temporal computer-readable medium according to claim 17, wherein the first ANN comprises a convolutional neural network or a vision transformer-based model, and the second ANN comprises a language model.

23. The non-temporary computer-readable medium according to claim 17, wherein the program code comprises program code for determining the basis by the second ANN based on the maximum log-likelihood of the next token of the tokenized text data based on at least one previous token of the tokenized text data in the interleaved sequence of the image and the text information.

24. A means for receiving an interleaved sequence of image and text information by a first artificial neural network (ANN), In order to generate a representation of the interleaved sequence of the image and the text information based on grid features, means for extracting the grid features of the image of the interleaved sequence of the image and the text information by the first ANN, A second ANN provides means for mapping the grid features to text regions, Means for extracting visual information of the interleaved sequence of the image and the text information by the second ANN based on the grid features in the text region, The visual information of the interleaved sequence of images and the text information, comprising one or more lower-level surrogate tasks, means for determining a basis by the second ANN based on the visual information, A device equipped with the following features.

25. The apparatus according to claim 24, wherein the visual information further includes a constitutive aspect of a scene shown in the interleaved sequence of the images, or a higher-level task inferred from one or more lower-level surrogate tasks, the constitutive aspect includes one or more of an object recognition task, an object re-identification task, an object tracking task, or an object relation task.

26. The apparatus according to claim 24, further comprising means for performing a visual reasoning task based on the grounds, using the second ANN.

27. The apparatus according to claim 24, wherein the grid features include spatiotemporal information of the image of the interleaved sequence of the image and the text information.

28. The apparatus according to claim 24, further comprising means for the second ANN to generate a hidden representation of a query that uses cross-attention to induce the extraction of the visual information of the interleaved sequence of the image and the text information.

29. The apparatus according to claim 24, wherein the first ANN comprises a convolutional neural network or a vision transformer-based model, and the second ANN comprises a language model.

30. The apparatus according to claim 24, further comprising means for determining the basis by the second ANN based on the maximum log-likelihood of the next token of the tokenized text data based on at least one previous token of the tokenized text data in the interleaved sequence of the image and the text information.

Images