Information processing device, information processing method, and program

Abstract

This reduces the computational cost when performing specific image processing using a neural network model. [Solution] An information processing device that performs predetermined processing using a neural network model, comprising an inference unit that divides an input image into a plurality of patches and performs the predetermined processing by inputting the plurality of patches into the model, wherein the model has a horizontal attention mechanism that calculates features for each row of the plurality of patches constituting the image, and a vertical attention mechanism that calculates features for each column of the plurality of patches constituting the image.

Description

[Technical Field]

[0001] This invention relates to a fundamental model for performing image-related processing, such as image recognition. [Background technology]

[0002] In image recognition, foundational models are widely used. These foundational models are the parts that perform feature extraction to solve specific tasks (such as image classification, object detection, and human pose estimation) using images as input.

[0003] Various models have been proposed as foundational models, including CNN-based models (e.g., ResNet) and Transformer-based models (e.g., Vision Transformer (ViT) in Non-Patent Document 1). In recent years, Transformer-based models have attracted attention due to their high accuracy.

[0004] When solving a desired task using a foundational model, it is common practice to use the foundational model as the backbone, connect task-specific modules (heads) to it, and then tune (fine-tune) the parameters of the entire model based on the provided training data. Furthermore, foundational models are often pre-trained on large datasets (ground truth data is not always necessary) (e.g., Masked Auto Encoder in Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR. [Non-Patent Document 2] He, K.; Chen, X.; Xie, S.; Li, Y.; Doll´ar, P.; and Girshick, R. 2022. Masked Autoencoders Are Scalable Vision Learners. In CVPR. [Non-Patent Document 3] Sun, Y.; Dong, L.; Huang, S.; Ma, S.; Xia, Y.; Xue, J.;Wang, J.; and Wei, F. 2023b. Retentive Network: A Successor to Transformer for Large Language Models. arXiv preprint. [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] Transformer-based models such as ViT have a Self-Attention module. Also, when applying a Transformer-based model to an image, the image is divided into a plurality of patches and used as input to the model. However, in the Self-Attention module, since the computational cost is proportional to the square of the number of input tokens (the number of patches in the case of an image), there is a problem that as the number of patches in the image increases, the computational cost becomes extremely large.

[0007] The present invention has been made in view of the above points, and an object thereof is to provide a technique capable of reducing the computational cost when performing predetermined processing on an image using a neural network model.

Means for Solving the Problems

[0008] According to the disclosed technique, there is provided an information processing apparatus that performs predetermined processing using a neural network model, comprising an inference unit that divides an input image into a plurality of patches and performs the predetermined processing by inputting the plurality of patches into the model. The model has a horizontal attention mechanism that calculates features for each row of the plurality of patches constituting the image, and a vertical attention mechanism that calculates features for each column of the plurality of patches constituting the image. An information processing apparatus is provided.

Advantages of the Invention

[0009] According to the disclosed technique, a technique capable of reducing the computational cost when performing predetermined processing on an image using a neural network model is provided.

Brief Description of the Drawings

[0010] [Figure 1] It is a diagram showing the structure of ViT. [Figure 2] It is a diagram showing an example of pre-training (MAE). [Figure 3] It is a diagram showing the calculation formula of Self-Attention. [Figure 4] It is a diagram showing a configuration example of the information processing apparatus 100 (learning apparatus). [Figure 5] It is a flowchart for explaining the operation of the information processing apparatus 100. [Figure 6] It is a diagram showing a configuration example of the information processing apparatus 200 (inference apparatus). [Figure 7] It is a flowchart for explaining the operation of the information processing apparatus 200. [Figure 8] This figure shows an example configuration of Model 300. [Figure 9] This is a diagram showing the structure of RetNet (Non-Patent Document 3). [Figure 10] This diagram illustrates an example of how the Attention module can be applied. [Figure 11] This figure shows an example configuration in which Block A and Block B are connected. [Figure 12] This diagram shows the configuration of the MSR module. [Figure 13] This is a diagram showing the configuration of ViRet. [Figure 14] This is a diagram illustrating pre-training method 1. [Figure 15] This figure shows a comparison between MAE and pre-training method 1. [Figure 16] This is a diagram illustrating pre-learning method 2-1. [Figure 17] This is a diagram illustrating pre-learning method 2-2. [Figure 18] This figure shows an example of the device's hardware configuration. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the present invention (this embodiment) will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below.

[0012] The following will first describe the conventional technology and its problems in more detail, and then describe the technology according to this embodiment in detail.

[0013] (Regarding conventional technology) As mentioned above, Transformer-based models have recently attracted attention as foundational models due to their high accuracy. The left side of Figure 1 shows the structure of ViT (Non-Patent Literature 1), one of the Transformer-based models. Figure 1 shows an example configuration in which an MLP Head is attached to a Transformer Encoder to perform image classification. As shown in Figure 1, ViT first divides the image into multiple patches. Patches correspond to tokens in language processing. Each patch is converted into a vector by flattening, embedded, and input to the Transformer Encoder. Features are output from the Transformer Encoder, and the class, which is the classification result, is output through the MLP Head.

[0014] The right side of Figure 1 shows the configuration of the Transformer Encoder. As shown in the figure, the Transformer Encoder has a configuration in which multiple blocks (Transformer blocks) consisting of Multi-Head Attention and MLP, etc., are connected (L blocks in Figure 1).

[0015] Figure 2 shows an example of a ViT pre-training method, the method disclosed in Non-Patent Document 2 (Masked Autoencoder). In the example shown in Figure 2, the original image is divided into multiple patches, and any number of patches are masked. The unmasked patches are used as input to the Encoder. In the example in Figure 2, the output from the Encoder is input to the Decoder, and the Encoder and Decoder are trained so that the Decoder can reconstruct the original image. Both the Encoder and Decoder are, for example, ViT.

[0016] (Regarding the issues) In Transformer-based methods like ViT, the underlying model is constructed by connecting multiple Transformer blocks.

[0017] Inside each Transformer block, there is a Self-Attention module processed by the equation in Figure 3. Since the equation in Figure 3 assumes language processing, "tokens" are used. The Self-Attention module can also be called an "attention mechanism."

[0018] As shown in Figure 3, Q (Query), K (Key), and V (Value) are rows obtained by multiplying the embedded input tokens by a linear matrix, and are calculated as "number of input tokens × number of feature dimensions (d k This is a queue of "QK". T Each token has a value that indicates the relationship between tokens (specifically, the value of the dot product between the token vectors). This value is called the attention score. The row sum is transformed using softmax so that it equals 1, and by multiplying by V, a weighted sum is obtained for each token using the softmax-treated value (weight).

[0019] The Self-Attention module can be applied to an image by replacing the above tokens with patches. The above calculation incurs a computational cost equal to the square of the number of input tokens. In an image, the number of tokens corresponds to the number of patches. Assuming the patch size is constant regardless of the image size, the computational cost of Self-Attention for image B, which is twice the width and height of image A, is (2 × 2) the computational cost of Self-Attention for image A. 2 = 16 times.

[0020] As a model that can solve the above problems, the RetentiveNetwork (referred to as RetNet), disclosed in Non-Patent Document 3, has been proposed. However, the data assumed by this model is linguistic data (a one-dimensional sequence) and is not necessarily optimized for images (two-dimensional data).

[0021] (Summary of the embodiment) In this embodiment, a new foundational model for image data is provided by making the improvements described later to RetNet. Furthermore, in this embodiment, two methods described later will be explained for pre-training this new foundational model.

[0022] In the module that constitutes the base model of this embodiment (the module corresponding to Self-Attention), if the number of vertical patches in the input image is h and the number of horizontal patches in the input image is w, then max(h 2 ,w 2 This can be done at the computational cost of (h*w). This computational cost is the same as the computational cost when applying conventional Self-Attention to an image. 2 It is significantly smaller in comparison. Therefore, by using the technology according to this embodiment, it becomes possible to speed up the overall processing and apply it to higher-resolution images within a realistic processing time.

[0023] Furthermore, by using the base model according to this embodiment as a backbone, generating a model with connected head modules that solve a desired task, and tuning the model using a given dataset, it is possible to construct a model that can solve a desired task with high accuracy while retaining the same benefits as the embodiment described above.

[0024] (Example of device configuration) The following describes an example of the device configuration in this embodiment. In this embodiment, separate devices are used for the learning phase and the inference phase, but the same device may be used for both phases. Also, the term "learning" below is assumed to refer to "pre-training," but is not limited to "pre-training."

[0025] <Equipment configuration for the learning phase> Figure 4 shows an example configuration of the information processing device 100 that performs learning. The information processing device 100 that performs learning may also be called a learning device.

[0026] As shown in Figure 4, the information processing device 100 comprises an input unit 110, a learning unit 120, and an output unit 130. The learning unit 120 holds a model 300 to be learned. The model 300 is a model having a base model according to this embodiment. For example, the model 300 may be a model in which a head module for solving a desired task is connected to the base model. Alternatively, the model 300 may be the base model itself.

[0027] The operation of the information processing device 100 will be explained according to the steps in the flowchart in Figure 5. In S101, training data is input from the input unit 110. The training data is, for example, a collection of images.

[0028] In S102, the learning unit 120 inputs the input data to the model 300 and trains the model 300 by updating its parameters so that the error between the output from the model 300 and the ground truth data is minimized. Note that when using pre-training method 1 described later, "ground truth data" refers to the original image before masking (or the original image of the masked patch). In other words, the ground truth data here does not refer to any data specially prepared in addition to the image (such as annotated images).

[0029] When training is complete, in S103, the output unit 130 outputs the trained model 300 (specifically, model parameters, hyperparameters, etc.). The trained model 300 is used in the inference phase.

[0030] <Instrument configuration for the inference phase> Figure 6 shows an example configuration of the information processing device 200 that performs inference. The information processing device 200 that performs inference may also be called an inference device.

[0031] As shown in Figure 6, the information processing device 200 comprises an input unit 210, an inference unit 220, and an output unit 230. The inference unit 220 holds a trained model 300. Alternatively, the inference unit 220 can be considered as the model 300 itself. The trained model 300 may be the model 300 itself trained by the aforementioned information processing device 100, or it may be a base model (the trained model 300) to which a header for a desired task has been added and which has been separately fine-tuned.

[0032] The operation of the information processing device 200 will be explained according to the steps in the flowchart in Figure 7. In S201, input data (e.g., an image) is input from the input unit 210. In S202, the inference unit 220 inputs the input data to the model 300 and obtains output data from the model 300. The output data is passed to the output unit 230. In S203, the output unit 230 outputs the output data.

[0033] For example, if model 300 is a model that performs the task of classifying images, then images are input to model 300, and the classification results are output from model 300.

[0034] (Example of model configuration) Figure 8 shows an example configuration of Model 300 in this embodiment. As shown in Figure 8, Model 300 has Input Embeddings, multiple "Block A and Block B" components, and a Task Head.

[0035] Block A consists of norm (normalization layer), Horizontal MSR (horizontal MSR), Vertical MSR (vertical MSR), norm (normalization layer), and FFN (feedforward layer) in this order. Note that FFN may also be called MLP (Multi-layer perceptron).

[0036] Block B consists of norm (normalization layer), Vertical MSR (vertical MSR), Horizontal MSR (horizontal MSR), norm (normalization layer), and FFN (feedforward layer) in this order.

[0037] MSR stands for Multi-Scale Retention, and it is a layer introduced in the aforementioned RetNet, performing processing equivalent to Self-Attention. Similar to the Attention mechanism, it performs operations between queries and keys in Retention, enabling learning that captures the relationships between tokens. For reference, the structure of RetNet is shown in Figure 9.

[0038] However, the MSR in this embodiment differs from the MSR in RetNet. Details will be described later.

[0039] Both the Vertical MSR and Horizontal MSR shown in Figure 8 are capable of capturing relationships between patches, similar to how attention mechanisms used in language processing capture relationships between tokens. Therefore, the Vertical MSR and Horizontal MSR can be called attention modules (or attention mechanisms), respectively.

[0040] Furthermore, Horizontal MSR may be called the lateral attention mechanism, and Vertical MSR may be called the vertical attention mechanism.

[0041] In the example in Figure 8, it is assumed that blocks A and B are connected alternately, such as block A-block B-block A-block B-...., but the configuration is not limited to this. For example, it may be a configuration in which multiple blocks A and multiple blocks B are connected, such as block A-block A-block A-block B-block B-block B.... Also, the number of blocks to be connected is arbitrary and can be any number.

[0042] Furthermore, as will be described later, in this embodiment, position coding is performed in both the lateral attention mechanism and the vertical attention mechanism by adding a value representing the patch's position to the patch (or the vector representing the patch) using 2D RoPE.

[0043] (Processing Summary) Referring to Figure 10, we will explain an example of applying the Attention module (attention mechanism) using Block A (configured in a horizontal-to-vertical order) as an example. Also, in order to simplify the explanation, we will omit the explanation of the embedding and flattening processes that are performed before input to the block.

[0044] As shown in Figure 10, in this example, the input image is divided into 4 rows x 4 columns. In other words, the input image is divided into 16 patches.

[0045] The lateral attention mechanism performs attention processing on the first row (specifically, the vectors of four patches), the second row, the third row, and the fourth row. The attention processing for each row is basically the same as the attention processing shown in Figure 3. However, more specifically, the attention mechanism of this embodiment is based on the MSR in RetNet disclosed in Non-Patent Document 3, and its details will be described later.

[0046] Next, the vertical attention mechanism performs attention processing on the features corresponding to the first column of the image (specifically, the vectors for the four patches), the features corresponding to the second column, the features corresponding to the third column, and the features corresponding to the fourth column of the patched image that has been processed by the horizontal attention mechanism. As described above, the attention processing for each column is basically the same as the attention processing shown in Figure 3. The processed features are input to the MLP, and the features are output from the MLP.

[0047] Figure 10 shows block A (configuration in the order of horizontal to vertical), but even in block B (configuration in the order of vertical to horizontal), the same processing as in the configuration of Figure 10 is performed, only the order of vertical and horizontal is changed.

[0048] Figure 11 shows an example configuration in which Block A and Block B are connected. As mentioned above, Block A and Block B may be connected alternately, or N Block A units (where N is a non-negative integer) may be connected followed by M Block B units (where M is a non-negative integer), or N Block B units (where N is a non-negative integer) may be connected followed by M Block A units (where M is a non-negative integer).

[0049] (Model details) The model according to this embodiment will be described in more detail below. Here, the model according to this embodiment will be referred to as ViRet. ViRet is an abbreviation for Vision Retentive Network.

[0050] Since ViRet is based on RetNet, we will first explain the architecture of RetNet. Compared to RetNet, ViRet according to this embodiment is scalable and extrapolable for image data.

[0051] <RetNetについて> Like Transformer and its successor models, RetNet consists of L identical blocks, each containing an MSR module and an FFN module. In RetNet, the input is one-dimensional text data.

[0052] The input embedding is X∈R T×d Let's assume that T is the length of the token and d is the dimension of the hidden layer.

[0053] Each block outputs a vector of the same size by sequentially applying the MSR() and FFN() functions to its input. In each block, RMSNorm is applied before MSR() and FFN(), followed by residual connections.

[0054] <MSRについて> Fig. 12 shows a configuration example of the MSR module. Each MSR includes N retention head modules. The n-th (1 ≤ n ≤ N) retention head applies the following function to the input embedding X to generate a vector Y head =(d n =d / N)-dimensional vector Y T×d-head ∈R n First, three linear projections are applied to X as shown below to generate matrices for each of the query Q n , key K n , and value V

[0055]

Equation

[0056]

Equation

[0057]

Equation

[0058]

Equation

[0059]

number

[0060]

number

[0061]

number

[0062] In each MSR, the outputs of all retention heads are first concatenated along the feature dimension, followed by the application of RMSNorm, Swish gate, and additional projections. Formally, MSR(X) is defined as follows:

[0063]

number

[0064]

number

[0065] <Feedforward Network (FFN)> For a given input X, each FFN sequentially performs projection, Swish gating, and another projection. This process is defined by the following equation.

[0066]

Number

[0067] <Consideration of RetNet> The parallel-form RetNet as described above has the same quadratic computational complexity as the Transformer, and its scalability is limited when applied to high-resolution image data. Also, regarding extrapolation ability, RetNet is not optimal for image data because the positional encoding (i.e., RoPE in Equation (2)) 1D ) and the decay matrix (i.e., D in Equation (6)) consider the 1D distance between tokens. From these points, in this embodiment, ViRet is realized by making several modifications to improve the scalability and extrapolation ability of RetNet.

[0068] <About ViRet> When an input image I is given, the corresponding 2D embedding X 2D ∈R H×W×d can be easily obtained using an appropriate patch embedding module. Here, H = H I / p, W = W I / p. H I is the height (vertical length) of the image, W I is the width of the image, and p is the patch size. Flatten X 2D into a 1D vector (i.e., X 1D ∈R HW×dWhile it is possible to directly apply RetNet by doing so, this method is not optimal for processing 2D image data. Below, we will explain how ViRet was realized by modifying RetNet. Here, an example of the ViRet configuration is shown in Figure 13.

[0069] (a) Removal of causal mask and introduction of Softmax X 2D Since all embeddings within are given at once, causal masking to prevent eavesdropping is no longer necessary, and equation (6) can be modified as follows:

[0070]

number

[0071]

number

[0072]

number

[0073] (c) Axis decomposition of retention Directly applying the retention operation (equivalent to the attention operation) (i.e., directly applying equation (3)) requires quadratic computational complexity with respect to the token length (i.e., HW). To address this problem, the retention operation is decomposed horizontally and vertically and applied sequentially. Specifically, taking into account (a) above, equations (3) to (5) are replaced as follows.

[0074]

number

[0075]

number

[0076]

number

[0077]

number

[0078] (Pre-training method for the base model 1) Pre-training method 1 is similar to the MAE method described above. That is, patches within the image are randomly masked, and the base model is pre-trained so that it can reconstruct the image of the masked patches.

[0079] However, in the base model (Model 300) according to this embodiment, attention calculation is performed only in the horizontal and vertical directions of the image, so MAE cannot be directly applied to pre-training.

[0080] Therefore, in pre-training method 1, the learning unit 120 of the information processing device 100 performs pre-training of model 300 in the procedure shown in Figure 14. Here, model 300 is the "base model + decoder" shown in Figure 14. The structure of the base model is as already described, for example, the structure in Figure 8 with the "Task Head" removed. The decoder may be a ViT, or it may have the same structure as the base model in this embodiment, or it may have any other structure.

[0081] As shown in Figure 14, the learning unit 120 first divides the input image into patches and independently selects elements in predetermined proportions in both the horizontal and vertical directions. In the example in Figure 14, the first and fourth rows are selected horizontally, and the second and fourth columns are selected vertically.

[0082] The learning unit 120 does not mask the patch corresponding to the "selected row number and selected column number," but masks all other patches. For example, the patch in the first row and second column is not masked.

[0083] The learning unit 120 inputs unmasked patches into the base model. In the example in Figure 14, the patches in the first row and second column, the first row and fourth column, the fourth row and second column, and the fourth row and fourth column are input into the base model. Features are output from the base model and input into the decoder. The decoder outputs an unmasked image.

[0084] The learning unit 120 tunes the parameters of the model 300 (= base model + decoder) so that the error (loss) between the output image and the input image (the original, unmasked image) is minimized. Note that the error calculation may be performed only on the image of the masked patch portion.

[0085] Figure 15 shows a comparison between the conventional MAE pre-training method (left side of Figure 15) and the pre-training method 1 according to this embodiment (right side of Figure 15).

[0086] (Pre-training method for the base model 2) Next, we will explain the pre-training method 2 executed by the learning unit 120. Here, we will explain pre-training method 2-1 and pre-training method 2-2.

[0087] <Pre-learning method 2-1> Pre-training method 2-1 will be explained with reference to Figure 16. In pre-training method 2-1, the learning unit 120 holds model 300, which is the base model according to this embodiment, and an existing base model such as ViT (referred to as base model A).

[0088] In pre-training method 2-1, the learning unit 120 inputs images to both model 300 and base model A. Feature maps are output from both model 300 and base model A.

[0089] The learning unit 120 tunes the parameters of model 300 to reproduce the feature map output by base model A. That is, the learning unit 120 obtains feature maps from both model 300 and base model A, and tunes the parameters of model 300 to minimize the error between them.

[0090] <Pre-learning method 2-2> Pre-training method 2-2 will be explained with reference to Figure 17. Pre-training method 2-2 uses multiple existing foundational models. The number of existing foundational models is not limited to a specific number, but here we will explain using the case of using two existing foundational models as an example.

[0091] In pre-training method 2-2, the learning unit 120 holds Model 300, which is the base model according to this embodiment, an existing base model such as ViT (referred to as base model A), and an existing base model different from base model A (referred to as base model B).

[0092] In pre-training method 2-2, the learning unit 120 inputs images to model 300, base model A, and base model B. Feature maps are output from each of model 300, base model A, and base model B.

[0093] The learning unit 120 obtains feature maps from model 300, base model A, and base model B. The learning unit 120 tunes the parameters of model 300 so that the sum of the error between the feature map from model 300 and the feature map from base model A and the error between the feature map from model 300 and the feature map from base model B is minimized.

[0094] (Example hardware configuration) Any of the devices (information processing devices 100, 200) described in this embodiment can be realized, for example, by having a computer execute a program. This computer may be a physical computer or a virtual machine on the cloud. Note that the model is not included in the program. However, the model may be included in the program.

[0095] In other words, the device can be realized by using hardware resources such as the CPU and memory built into a computer to execute a program corresponding to the processing performed by the device. The program can be recorded on a computer-readable recording medium (such as portable memory), saved, and distributed. It can also be provided via a network, such as the Internet or email.

[0096] Figure 18 shows an example of the hardware configuration of the computer described above. The computer in Figure 18 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., all of which are interconnected by bus B. The computer may also be equipped with a GPU.

[0097] The program that enables processing on the computer is provided, for example, on a recording medium 1001 such as a CD-ROM or memory card. When the recording medium 1001 containing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001; it may also be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files and data.

[0098] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when a program startup command is received. The CPU 1004 implements the functions related to the memory device 1003 according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) etc. generated by a program. The input device 1007 consists of a keyboard and mouse, buttons, or a touch panel etc., and is used to input various operation commands. The output device 1008 outputs the calculation results.

[0099] (Summary of the embodiments, effects, etc.) As described above, the technology described in this embodiment makes it possible to reduce the computational cost when performing predetermined image processing using a neural network model.

[0100] The following additional information is disclosed regarding the embodiments described above.

[0101] <Note> (Additional note 1) An information processing device that performs predetermined processing using a neural network model, The inference unit divides the input image into multiple patches and performs the predetermined processing by inputting the multiple patches into the model. The aforementioned model, The image comprises a horizontal attention mechanism that calculates features for each row of multiple patches constituting the image, and a vertical attention mechanism that calculates features for each column of multiple patches constituting the image. Information processing device. (Additional note 2) The model alternately includes blocks that first process the lateral attention mechanism and then the vertical attention mechanism, and blocks that first process the vertical attention mechanism and then the lateral attention mechanism. The information processing device described in Appendix 1. (Additional note 3) The aforementioned model performs position coding for patches using 2D RoPE in both the lateral attention mechanism and the vertical attention mechanism. The information processing device described in Appendix 1. (Additional note 4) The aforementioned model is a model trained using patches corresponding to selected rows and selected columns in multiple patches obtained from an image as input. The information processing device described in Appendix 1. (Additional note 5) The aforementioned model is a model that has been trained to minimize the error between the feature map output by the aforementioned model for an input image and the feature map output by another model for the same input image. The information processing device described in Appendix 1. (Additional note 6) An information processing device for training a neural network model, The system includes a learning unit that selects rows and columns from multiple patches obtained from an input image, inputs the patches corresponding to the selected rows and columns into the model, and trains the model so that the error between the image obtained from the features output from the model and the input image is minimized. The aforementioned model, It has a horizontal attention mechanism that calculates features for each row of multiple patches, and a vertical attention mechanism that calculates features for each column of multiple patches. Information processing device. (Additional note 7) An information processing device for training a neural network model, The system includes a learning unit that trains the model so as to minimize the error between the feature map output from the model for an input image and the feature map output from another model for the same input image. The aforementioned model, It has a horizontal attention mechanism that calculates features for each row of multiple patches, and a vertical attention mechanism that calculates features for each column of multiple patches. Information processing device. (Additional note 8) An information processing method performed by an information processing device that performs predetermined processing using a neural network model, The inference step includes dividing an input image into multiple patches and inputting the multiple patches into the model to perform the predetermined processing, The aforementioned model, The image comprises a horizontal attention mechanism that calculates features for each row of multiple patches constituting the image, and a vertical attention mechanism that calculates features for each column of multiple patches constituting the image. Information processing methods. (Additional note 9) A non-temporary storage medium that stores a program for causing a computer to function as a component of an information processing device described in any one of the appendices 1 through 7.

[0102] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims. [Explanation of Symbols]

[0103] 100, 200 Information Processing Devices 110, 210 Input Section 120 Learning Department 130, 230 Output section 220 Reasoning Department 300 Models 1000 drive unit 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing device that performs predetermined processing using a neural network model, The inference unit divides the input image into multiple patches and performs the predetermined processing by inputting the multiple patches into the model. The aforementioned model, The image comprises a horizontal attention mechanism that calculates features for each row of multiple patches constituting the image, and a vertical attention mechanism that calculates features for each column of multiple patches constituting the image. Information processing device.

2. The model alternately includes blocks that first process the lateral attention mechanism and then the vertical attention mechanism, and blocks that first process the vertical attention mechanism and then the lateral attention mechanism. The information processing apparatus according to claim 1.

3. The aforementioned model performs position coding for patches using 2D RoPE in both the lateral attention mechanism and the vertical attention mechanism. The information processing apparatus according to claim 1.

4. The aforementioned model is a model trained using patches corresponding to selected rows and selected columns in multiple patches obtained from an image as input. The information processing apparatus according to claim 1.

5. The aforementioned model is a model that has been trained to minimize the error between the feature map output by the aforementioned model for an input image and the feature map output by another model for the same input image. The information processing apparatus according to claim 1.

6. An information processing device for training a neural network model, The system includes a learning unit that selects rows and columns from multiple patches obtained from an input image, inputs the patches corresponding to the selected rows and columns into the model, and trains the model so that the error between the image obtained from the features output from the model and the input image is minimized. The aforementioned model, It has a horizontal attention mechanism that calculates features for each row of multiple patches, and a vertical attention mechanism that calculates features for each column of multiple patches. Information processing device.

7. An information processing device for training a neural network model, The system includes a learning unit that trains the model so as to minimize the error between the feature map output from the model for an input image and the feature map output from another model for the same input image. The aforementioned model, It has a horizontal attention mechanism that calculates features for each row of multiple patches, and a vertical attention mechanism that calculates features for each column of multiple patches. Information processing device.

8. An information processing method performed by an information processing device that performs predetermined processing using a neural network model, The inference step includes dividing an input image into multiple patches and inputting the multiple patches into the model to perform the predetermined processing, The aforementioned model, The image comprises a horizontal attention mechanism that calculates features for each row of multiple patches constituting the image, and a vertical attention mechanism that calculates features for each column of multiple patches constituting the image. Information processing methods.

9. A program for causing a computer to function as a component of an information processing device described in any one of claims 1 to 7.

Images