Information processing device, information processing method, and recording medium

Abstract

An information processing device (10) is provided with: a generation means (110) for generating a patch token and sentence-start token corresponding to an input image; an expansion means (120) for expanding the sentence-start token to a sentence-start token block having a size corresponding to the number of a plurality of patch token blocks into which the patch tokens have been divided by region according to a prescribed grid pattern; and a computation means (130) for subjecting the sentence-start token block and the patch token blocks to computation on the basis of a self-attention mechanism for each element group of which the position within each block are shared in-common. By using this information processing device, it is possible to appropriately subject an input image to processing based on the self-attention mechanism.

Description

[Technical field]

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium. [Background technology]

[0002] As this type of device, there is known a device that divides an image into a grid and executes various processes. For example, Patent Document 1 discloses a method of obtaining a quantized gradient direction feature from a luminance gradient in each lattice obtained by dividing an image into a grid. Patent Document 2 discloses a method of dividing an image into an N×N grid and extracting a D-dimensional feature vector from each cell in the grid.

[0003] Also, devices that use a self-attention mechanism in image processing devices are known. For example, Patent Document 3 discloses that a layer of a device that recognizes images has a self-attention structure. Patent Document 4 discloses that a feature vector related to an image is corrected using a query, a key, and a value. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-201498 A [Patent Document 2] JP 2017-091525 A [Patent Document 3] Patent Publication No. 2021-093144 [Patent Document 4] International Publication No. 2021 / 095212 Summary of the Invention [Problem to be solved by the invention]

[0005] This disclosure is intended to improve upon the techniques disclosed in the prior art documents. [Means for solving the problem]

[0006] One aspect of the information processing device disclosed herein comprises a generating means for generating a sentence start token and a patch token corresponding to an input image, an expanding means for expanding the sentence start token into a sentence start token block having a size corresponding to the number of multiple patch token blocks obtained by dividing the patch token into areas according to a predetermined grid pattern, and a computing means for computing the sentence start token block and the patch token block based on a self-attention mechanism for each group of elements that have a common position within each block.

[0007] One aspect of the information processing method disclosed herein is an information processing method executed by at least one computer, which generates a sentence start token and a patch token corresponding to an input image, expands the sentence start token to a sentence start token block whose size corresponds to the number of multiple patch token blocks obtained by dividing the patch token into regions according to a predetermined grid pattern, and performs calculations on the sentence start token block and the patch token block based on a self-attention mechanism for each group of elements that have a common position within each block.

[0008] One aspect of the recording medium of this disclosure is a computer program recorded on at least one computer to execute an information processing method, which generates a sentence start token and a patch token corresponding to an input image, expands the sentence start token to a sentence start token block whose size corresponds to the number of multiple patch token blocks obtained by dividing the patch token into regions according to a predetermined grid pattern, and performs calculations on the sentence start token block and the patch token block based on a self-attention mechanism for each group of elements that have a common position within each block. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a hardware configuration of an information processing device according to a first embodiment. [Diagram 2] 1 is a block diagram showing an overall configuration of an information processing device according to a first embodiment. [Diagram 3] 2 is a block diagram showing the configuration of a self-attention mechanism unit in the information processing device according to the first embodiment. FIG. [Figure 4] 1 is a block diagram showing a functional configuration of an information processing device according to a first embodiment. [Diagram 5] 5 is a flowchart showing a flow of feature embedding processing in the information processing device according to the first embodiment. [Figure 6] 4A to 4C are conceptual diagrams illustrating an example of a process for expanding a sentence start token and a process for modifying a patch token in the information processing device according to the first embodiment. [Figure 7] FIG. 2 is a conceptual diagram showing an example of processing based on a self-attention mechanism for each element group in the information processing device according to the first embodiment. [Figure 8] FIG. 11 is a block diagram showing a functional configuration of an information processing device according to a second embodiment. [Figure 9] 13 is a flowchart showing the flow of feature conversion processing in the information processing device according to the second embodiment. [Figure 10] FIG. 11 is a conceptual diagram showing an example of a restoration process in the information processing device according to the second embodiment. [Figure 11] 13 is a conceptual diagram showing an example of a restoration process using an average value in the information processing device according to the third embodiment. FIG. [Figure 12] FIG. 13 is a conceptual diagram showing an example of a restoration process using a maximum value in the information processing device according to the fourth embodiment. [Figure 13] FIG. 13 is a block diagram showing a functional configuration of an information processing device according to a fifth embodiment. [Figure 14] 13 is a flowchart showing the flow of feature embedding processing in an information processing device according to a fifth embodiment. [Figure 15] 13A and 13B are conceptual diagrams showing intra-block transformation processing of patch tokens in an information processing device according to a fifth embodiment. [Figure 16] FIG. 13 is a block diagram showing a functional configuration of an information processing device according to a sixth embodiment. [Figure 17]13 is a flowchart showing the flow of feature conversion processing in an information processing device according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of an information processing device, an information processing method, and a recording medium will be described with reference to the drawings.

[0011] First Embodiment An information processing device according to a first embodiment will be described with reference to FIGS.

[0012] (Hardware configuration) First, the hardware configuration of the information processing device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the information processing device according to the first embodiment.

[0013] 1, an information processing device 10 according to the first embodiment includes a processor 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The information processing device 10 may further include an input device 15 and an output device 16. The above-mentioned processor 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are connected via a data bus 17.

[0014] The processor 11 reads a computer program. For example, the processor 11 is configured to read a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the processor 11 may read a computer program stored in a computer-readable recording medium using a recording medium reading device (not shown). The processor 11 may acquire (i.e., read) a computer program from a device (not shown) disposed outside the information processing device 10 via a network interface. The processor 11 controls the RAM 12, the storage device 14, the input device 15, and the output device 16 by executing the computer program that the processor 11 reads. In particular, in this embodiment, when the processor 11 executes the computer program that the processor 11 reads, a functional block that executes processing based on a self-attention mechanism with an image as an input is realized in the processor 11. That is, the processor 11 may function as a controller that executes each control in the information processing device 10.

[0015] The processor 11 may be configured as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), a demand-side platform (DSP), or an application specific integrated circuit (ASIC). The processor 11 may be configured as one of these, or may be configured to use a plurality of these in parallel.

[0016] The RAM 12 temporarily stores computer programs executed by the processor 11. The RAM 12 temporarily stores data that is temporarily used by the processor 11 when the processor 11 is executing the computer programs. The RAM 12 may be, for example, a dynamic random access memory (D-RAM) or a static random access memory (SRAM). Also, instead of the RAM 12, other types of volatile memory may be used.

[0017] The ROM 13 stores computer programs executed by the processor 11. The ROM 13 may also store other fixed data. The ROM 13 may be, for example, a P-ROM (Programmable Read Only Memory) or an EPROM (Erasable Read Only Memory). Also, instead of the ROM 13, other types of non-volatile memory may be used.

[0018] The storage device 14 stores data that is to be stored long-term by the information processing device 10. The storage device 14 may operate as a temporary storage device for the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0019] The input device 15 is a device that receives an input instruction from a user of the information processing device 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel. The input device 15 may be configured as a mobile terminal such as a smartphone or a tablet. The input device 105 may be, for example, a device that includes a microphone and is capable of voice input.

[0020] The output device 16 is a device that outputs information related to the information processing device 10 to the outside. For example, the output device 16 may be a display device (for example, a display) that can display information related to the information processing device 10. The output device 16 may also be a speaker or the like that can output information related to the information processing device 10 as audio. The output device 16 may be configured as a mobile terminal such as a smartphone or a tablet. The output device 106 may also be a device that outputs information in a format other than an image. For example, the output device 106 may be a speaker that outputs information related to the information processing device 10 as audio.

[0021] 1, some of the hardware may be provided in a device other than the information processing device 10. For example, the information processing device 10 may be configured to include only the above-mentioned processor 11, RAM 12, and ROM 13, and the other components (i.e., the storage device 14, the input device 15, and the output device 16) may be provided in, for example, an external device connected to the information processing device 10. Also, the information processing device 10 may have some of its calculation functions realized by an external device (for example, an external server, a cloud, etc.).

[0022] (Overall composition) Next, the overall configuration of the information processing device 10 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the overall configuration of the information processing device according to the first embodiment.

[0023] As shown in FIG. 2, the information processing device 10 according to the first embodiment may be configured to include a batch embedding processing unit 55 and a plurality of conversion blocks 50. Each of the plurality of conversion blocks 50 may include a self-attention mechanism unit 20 and a feature conversion unit 30. The information processing device 10 may be configured, for example, as a neural network that constructs a network using the plurality of conversion blocks 50. The feature conversion unit 30 according to the first embodiment is configured to receive an image as input and output some feature amount related to the image.

[0024] The batch embedding processor 55 is configured to be able to execute batch embedding processing on the input. The batch embedding processing here may be processing that compresses local regions of an input image into tokens into feature vectors based on a convolutional layer.

[0025] The self-attention mechanism unit 20 is configured to generate new features by dividing the input features into three features, namely, a query, a key, and a value, and performing a predetermined calculation process. The specific configuration and operation of the self-attention mechanism unit 20 will be described in detail below.

[0026] The feature conversion unit 30 is configured to be able to extract features (feature map) from an input image. The feature conversion unit 30 may be configured as a unit that performs feature extraction, for example, by using a local kernel convolution layer. Note that a specific feature extraction method by the feature conversion unit 30 can be appropriately adopted from existing technologies, and therefore a detailed description thereof will be omitted here.

[0027] (Self-Attention Mechanism Unit) Next, the configuration and operation of the above-mentioned self-attention mechanism unit 20 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the self-attention mechanism unit in the information processing device according to the first embodiment.

[0028] As shown in FIG. 3, the self-attention mechanism unit 20 includes three feature embedding processors 31, 32, and 33, a correlation calculation processor 34, a counting processor 35, a residual processor 36, and a feature conversion processor 37.

[0029] The feature embedding processing unit 31 is configured to be able to extract a "query" from the input feature map. The feature embedding processing unit 32 is configured to be able to extract a "key" from the input feature map. The feature embedding processing unit 33 is configured to be able to extract a "value" from the input feature map. Each of the feature embedding processing units 31, 32, and 33 may extract features using, for example, a convolutional layer or a fully connected layer used in a convolutional neural network. The query generated by the feature embedding processing unit 31 and the key generated by the feature embedding processing unit 32 are configured to be output to the correlation calculation unit 34. In addition, the value generated by the feature embedding processing unit 33 is configured to be output to the aggregation processing unit 35.

[0030] The correlation calculation unit 34 is configured to be able to calculate a feature map showing the correlation between the query generated by the feature embedding processing unit 31 and the key generated by the feature embedding processing unit 32. In this embodiment, in particular, a predetermined grid pattern is used to be able to refer to the entire space of the input feature map. This grid pattern will be described in detail later. The correlation calculation unit 34 may obtain the correlation by, for example, performing a shape transformation of a tensor and then calculating a matrix product. The correlation calculation unit 34 may also obtain the correlation by combining the two embedded features after performing a shape transformation of a tensor on the embedded features of the query and the key. The correlation calculation unit 34 may obtain a feature map showing the final correlation by performing a convolution and a rectified linear unit (ReLU) calculation on the matrix product or the combined features calculated as described above. The correlation calculation unit 34 may further be provided with a convolution layer for convolution. Furthermore, the correlation calculation unit 122 may normalize the feature map indicating the correlation using a sigmoid function, a softmax function, etc. The feature map indicating the correlation calculated by the correlation calculation unit 34 is configured to be output to the aggregation processing unit 35.

[0031] The aggregation processing unit 35 is configured to be able to reflect the feature map indicating the correlation calculated by the correlation calculation unit 34 as a weight in the value generated by the feature embedding processing unit 33. Such processing may be executed, for example, by calculating a matrix product of the feature map of the correlation (weight) and the value. The feature map reflecting the correlation is configured to be output to the residual processing unit 36.

[0032] The residual processing unit 36 ​​is configured to be able to execute residual processing on the feature map generated by the counting processing unit 35. This residual processing may be a process of adding the feature map generated by the counting processing unit 35 and the feature map input to the self-attention mechanism unit 20. This is to prevent the feature map from not being generated as the calculation result of the self-attention mechanism unit 20 even if the correlation is not calculated. For example, when 0 is calculated as the correlation (weight), the value is multiplied by 0, so that the feature value becomes 0 (disappears) in the feature map output by the counting unit 123. To prevent this, the residual processing unit 36 ​​executes the above-mentioned residual processing. The feature map generated by the residual processing unit 36 ​​is configured to be output to the feature conversion processing unit 37.

[0033] The feature conversion processing unit 37 is configured to be capable of executing a process for converting the feature map generated by the residual processing unit 36 ​​into an appropriate state (hereinafter, appropriately referred to as a "feature conversion process"). The specific processing contents of the feature conversion process will be described in detail in other embodiments described later.

[0034] (Functional configuration) Next, the functional configuration of the information processing device 10 according to the first embodiment (particularly the configuration for realizing the functions of the above-mentioned feature embedding processing units 31, 32, and 33) will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configuration of the information processing device according to the first embodiment.

[0035] 4, the information processing device 10 according to the first embodiment includes, as components for realizing its functions, a generating unit 110, an extension unit 120, and a calculation unit 130. Each of the generating unit 110, the extension unit 120, and the calculation unit 130 may be a processing block realized by, for example, the above-mentioned processor 11 (see FIG. 1).

[0036] The generation unit 110 is configured to be able to generate a sentence start token and a patch token corresponding to an input image. The sentence start token is an attribute token for assisting in understanding the structure of the input token, and has other attributes such as a phrase and an end of the sentence. The patch token is a token obtained by vectorizing pixels in a local area of ​​the input image. The sentence start token and the patch token may be token vectors handled by ViT (Vision Transromer). The sentence start token may be generated based on random numbers. In this case, the elements of the vector may be optimized by learning. The sentence start token generated by the generation unit 110 is configured to be output to the extension unit 120. On the other hand, the patch token is configured to be output to the calculation unit 130.

[0037] The expansion unit 120 is configured to be able to expand the size of the sentence start token generated by the generation unit 110. Specifically, the expansion unit 120 is configured to be able to expand the sentence start token to a size corresponding to the number of multiple patch token blocks included in the patch token. The patch token block is a block obtained by dividing the patch token into areas (equally divided) according to a predetermined grid pattern. The sentence start token is typically composed of only one element, whereas the patch token block is composed of multiple elements. The expansion unit 120 expands the sentence start token to a sentence start token block of the same size as the number of blocks of the patch token block, for example, by copying and pasting one element of the sentence start token. The sentence start token block expanded by the expansion unit 120 is configured to be output to the calculation unit 120.

[0038] The calculation unit 130 is configured to be able to execute calculation processing based on the self-attention mechanism (i.e., various processes described in FIG. 3) for the sentence start token block expanded by the expansion unit 120 and the patch token (multiple patch token blocks) acquired from the generation unit 120. Here, the calculation unit 130 according to this embodiment is particularly configured to be able to execute calculation processing based on the self-attention mechanism for each element group having a common position in each block for the sentence start token block and the patch token block. For example, the elements located at the top left of each block are integrated into one element group, and calculation processing based on the self-attention mechanism is executed. Similarly, the elements located at the top right of each block are integrated into one element group, and calculation processing based on the self-attention mechanism is executed. This processing will be described in detail later with a specific example.

[0039] (Operation flow) Next, a flow of operation of the information processing device 10 according to the first embodiment (particularly, the feature embedding process by the above-mentioned feature embedding processing units 31, 32, and 33) will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of the feature embedding process in the information processing device according to the first embodiment.

[0040] 5, when the information processing device 10 according to the first embodiment starts a feature embedding process (i.e., the process by the feature embedding processors 31, 32, and 33 shown in FIG. 3), a linear transformation process is first performed on the input feature amount (step S101). The linear transformation may be performed in a convolution layer or a fully connected layer.

[0041] Next, the generation unit 110 generates a sentence start token and a patch token corresponding to the input image, and divides the patch token into a plurality of regions (step S102). The sentence start token may be generated from a random number as described above. The expansion unit 120 expands the size of the sentence start token to create a sentence start token block (step S103). Meanwhile, a tensor conversion process is performed on the patch token (step S104). The tensor conversion process here is a process of converting elements that are at the same position within a block into one tensor.

[0042] Next, the calculation unit 130 tensor-transforms and integrates the sentence start token block and the tensor-transformed patch token (step S105). Specifically, the elements of the sentence start token block and the tensor-transformed patch token are integrated and converted into a one-dimensional tensor. Each tensor contains elements that are in the same position within each block.

[0043] (Specific operation example) Next, a specific operation example of the information processing device 10 according to the first embodiment (particularly, an operation example of the above-mentioned generating unit 110, expanding unit 120, and computing unit 130) will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a conceptual diagram showing an example of a sentence start token expanding process and a patch token transforming process in the information processing device according to the first embodiment. Fig. 7 is a conceptual diagram showing an example of a process based on a self-attention mechanism for each element group in the information processing device according to the first embodiment.

[0044] As shown in Fig. 6, the generation unit 110 generates a sentence start token and a patch token corresponding to an input image. Here, the sentence start token consists of only one element, and the patch token consists of 12 x 12 elements. The elements of the patch token may correspond to the number of pixels of the image.

[0045] The patch tokens can be divided into a number of patch token blocks by a predetermined grid pattern. Here, a patch token block is composed of 3 x 3 elements. In the example shown in the figure, for the sake of convenience, the color (shade) is shown to vary depending on the position within the block.

[0046] The start of sentence token is expanded to the same size as the number of blocks in the patch token block (here 4 × 4). Meanwhile, the patch token is tensor-transformed for each element that has a common position in the block (i.e., elements shown in the same color are grouped together).

[0047] As shown in Figure 7, the sentence start token and patch token are integrated into a one-dimensional vector. Specifically, a tensor is generated in which the sentence start token is at the top, followed by each element of the patch token. This tensor is a collection of elements that share a common position within a block. Therefore, the patch tokens included in each tensor are elements that are displayed in the same color.

[0048] In this embodiment, a calculation process based on the self-attention mechanism is executed for each of the above-mentioned tensors (i.e., groups of elements that have the same position in the block). In the case of a general self-attention mechanism, the amount of calculation required is squared with respect to the number of input elements, but by performing calculation processing for each group of elements as in this embodiment, the amount of calculation is reduced to the number of elements × K 2 × C, where K is the kernel size and C is the channel. The result of the self-attention mechanism is a feature map corresponding to the sentence start token and the patch token.

[0049] (Technical effect) Next, technical effects obtained by the information processing device 10 according to the first embodiment will be described.

[0050] As described in FIG. 1 to FIG. 7, in the information processing device 10 according to the first embodiment, a sentence start token and a patch token corresponding to an input image are generated. Then, for the expanded sentence start token block and a plurality of patch token blocks included in the patch token, a calculation process based on the self-attention mechanism is executed for each element group having a common position in the block. In this way, the amount of calculation in the self-attention mechanism can be reduced. This effect is remarkable when an image is input (i.e., when there are many elements). Note that since there is a sentence start token that has only one element, it is difficult to integrate it with the patch token as it is (integration is difficult unless each block is the same size), but in this embodiment, an expansion process is performed on the sentence start token, so appropriate integration can be performed and calculation processing can be executed for each element group.

[0051] The information processing device 10 according to the present embodiment can be applied to tasks that handle high-dimensional feature vectors, for example. For example, the information processing device 10 can be applied to object detection, object tracking, semantic segmentation, etc. The information processing device 10 can also be used for image pattern recognition.

[0052] <Second embodiment> The information processing device 10 according to the second embodiment will be described with reference to Fig. 8 to Fig. 10. The second embodiment differs from the first embodiment in only a portion of the configuration, and other parts may be the same as those of the first embodiment. Therefore, hereinafter, the parts that differ from the first embodiment already described will be described in detail, and the description of other overlapping parts will be omitted as appropriate.

[0053] (Functional configuration) First, the functional configuration of the information processing device 10 according to the second embodiment (particularly the configuration for realizing the functions of the feature embedding processing units 31, 32, and 33 and the feature conversion processing unit 37) will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the functional configuration of the information processing device according to the second embodiment. In Fig. 8, the same elements as those shown in Fig. 4 are denoted by the same reference numerals.

[0054] As shown in Fig. 8, the information processing device 10 according to the second embodiment includes, as components for realizing its functions, a generation unit 110, an extension unit 120, a calculation unit 130, and a restoration unit 140. That is, the information processing device 10 according to the second embodiment is configured to further include a restoration unit 140 in addition to the configuration of the first embodiment described above (see Fig. 4). Note that the restoration unit 140 may be a processing block realized by, for example, the above-mentioned processor 11 (see Fig. 1).

[0055] The restoration unit 140 is configured to be able to restore the feature corresponding to the start token block, among the features obtained as a result of the calculation based on the self-attention mechanism, to the size of the start token before expansion. For example, the restoration unit 140 may be configured to be able to perform downsizing by pooling processing.

[0056] (Operation flow) Next, a flow of operation of the information processing device 10 according to the second embodiment (particularly, the feature conversion processing by the feature conversion processing unit 37 described above) will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of the feature conversion processing in the information processing device according to the second embodiment.

[0057] 9, when the information processing device 10 according to the first embodiment starts a feature conversion process, the generating unit 110 first generates a sentence start token and a patch token corresponding to the input image, and divides the patch token into a plurality of regions (step S201). This process may be the same as the feature division in the feature embedding process described above (i.e., step S101 in FIG. 5).

[0058] The restoration unit 140 performs restoration processing on the divided start tokens to restore them to the size before expansion by the expansion unit 120 (step S202). Then, the calculation unit 130 tensor transforms and integrates the restored start tokens and patch tokens (step S203). That is, the start tokens and patch tokens are converted into one feature map.

[0059] (Specific operation example) Next, a specific operation example of the information processing device 10 according to the second embodiment (particularly, an operation example of the above-mentioned restoration unit 140) will be described with reference to Fig. 10. Fig. 10 is a conceptual diagram showing an example of restoration processing in the information processing device according to the second embodiment.

[0060] As shown in FIG. 10, the feature quantity corresponding to the sentence head token obtained as a result of the calculation based on the self-attention mechanism has a size (here, 4×4) expanded by the expansion unit 120. The restoration unit 140 executes a restoration process on this to convert it to the size before expansion (here, 1×1). Note that the specific method of the restoration process is not particularly limited. A specific example of the restoration process will be described in detail in another embodiment described later.

[0061] (Technical effect) Next, technical effects obtained by the information processing device 10 according to the second embodiment will be described.

[0062] As described in FIG. 8 to FIG. 10, in the information processing device 10 according to the second embodiment, the sentence start token obtained as a result of the calculation based on the self-attention mechanism (i.e., the sentence start token of the expanded size) is restored to the size before expansion. In this way, the size of the sentence start token that was temporarily changed for the calculation based on the self-attention mechanism can be restored to the original size. In other words, it can be restored to a size appropriate for a sentence start token.

[0063] <Third embodiment> The information processing device 10 according to the third embodiment will be described with reference to Fig. 11. The third embodiment describes a more specific example of the second embodiment described above, and the device configuration and overall operation may be the same as those of the first and second embodiments. Therefore, in the following, only the parts that differ from the embodiments already described will be described in detail, and the description of other overlapping parts will be omitted as appropriate.

[0064] (Example of restoration process) First, the restoration process in the information processing device 10 according to the third embodiment will be described with reference to Fig. 11. Fig. 11 is a conceptual diagram showing an example of the restoration process using an average value in the information processing device according to the third embodiment.

[0065] 11, in the information processing device 10 according to the third embodiment, the restoration unit 140 executes restoration processing by calculating the average value of the feature corresponding to the start token. More specifically, the restoration unit 140 calculates the average value of each element included in the feature corresponding to the start token, and generates a start token consisting of one element having the average value.

[0066] (Technical effect) Next, technical effects obtained by the information processing device 10 according to the third embodiment will be described.

[0067] 11, in the information processing device 10 according to the third embodiment, the restoration process is performed using the average value. In this way, it is possible to obtain an integrated token taking into account all patch tokens, so that it is possible to easily and accurately perform the restoration process of the sentence start token.

[0068] <Fourth embodiment> The information processing device 10 according to the fourth embodiment will be described with reference to Fig. 12. Note that the fourth embodiment describes a more specific example of the second embodiment, similar to the above-mentioned third embodiment, and the device configuration and overall operation may be the same as those of the first and second embodiments. Therefore, in the following, the parts that differ from the embodiments already described will be described in detail, and the explanation of other overlapping parts will be omitted as appropriate.

[0069] (Example of restoration process) First, the restoration process in the information processing device 10 according to the fourth embodiment will be described with reference to Fig. 12. Fig. 12 is a conceptual diagram showing an example of the restoration process using a maximum value in the information processing device according to the fourth embodiment.

[0070] 12, the information processing device 10 according to the fourth embodiment executes the restoration process by the restoration unit 140 calculating the maximum value of the feature corresponding to the start token. More specifically, the restoration unit 140 calculates the maximum value of each element included in the feature corresponding to the start token, and generates a start token consisting of one element having the maximum value.

[0071] (Technical effect) Next, technical effects obtained by the information processing device 10 according to the fourth embodiment will be described.

[0072] 12, in the information processing device 10 according to the fourth embodiment, the restoration process is performed using the maximum value. In this way, a representative patch token can be selected based on the maximum value to obtain the final sentence start token, so that the restoration process of the sentence start token can be easily and accurately performed.

[0073] <Fifth embodiment> The information processing device 10 according to the fifth embodiment will be described with reference to Fig. 13 to Fig. 15. The fifth embodiment differs from the first to fourth embodiments in part of its configuration and operation, and other parts may be the same as the first to fourth embodiments. Therefore, hereinafter, parts that differ from the embodiments already described will be described in detail, and other overlapping parts will be omitted as appropriate.

[0074] (Functional configuration) First, the functional configuration of the information processing device 10 according to the fifth embodiment (particularly, the configuration for realizing the functions of the feature embedding processing units 31, 32, and 33) will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the functional configuration of the information processing device according to the fifth embodiment. In Fig. 13, the same elements as those shown in Fig. 4 are denoted by the same reference numerals.

[0075] As shown in Fig. 13, the information processing device 10 according to the fifth embodiment includes, as components for realizing its functions, a generation unit 110, an extension unit 120, a calculation unit 130, and a transformation unit 150. That is, the information processing device 10 according to the fifth embodiment is configured to further include a transformation unit 150 in addition to the configuration of the first embodiment described above (see Fig. 4). Note that the transformation unit 150 may be a processing block realized by, for example, the above-mentioned processor 11 (see Fig. 1).

[0076] The transformation unit 150 is configured to be able to execute a process of transforming the tensor of the patch token. More specifically, the transformation unit 150 is configured to be able to execute a process of transforming the tensor so that a 1×1 convolution layer (fully connected layer) can be applied within each patch token block. This process is executed to refer to elements of a local region in a feature map. That is, since arithmetic processing is performed for each element group according to the position within the block, this process is executed to suppress the influence of elements being referenced in a sparse pattern. Hereinafter, the process executed by the transformation unit 150 is appropriately referred to as "intra-block transformation process".

[0077] (Operation flow) Next, the flow of operation of the information processing device 10 according to the fifth embodiment (particularly, the feature embedding process by the feature embedding processing units 31, 32, and 33) will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the flow of the feature embedding process in the information processing device according to the fifth embodiment. Note that in Fig. 14, the same processes as those shown in Fig. 5 are denoted by the same reference numerals.

[0078] 14, when the information processing device 10 according to the fifth embodiment starts a feature embedding process, a linear transformation process is first performed on the input feature amount (step S101). Then, the generation unit 110 generates a sentence head token and a patch token corresponding to the input image, and divides the patch token into a plurality of regions (step S102).

[0079] For the start token, the expansion unit 120 expands the size to make it a start token block (step S103). On the other hand, for the patch token, a tensor transformation process is executed (step S104). Then, in this embodiment, the transformation unit 140 further executes an intra-block transformation process (step S501).

[0080] After that, the calculation unit 130 performs tensor transformation and integration on the sentence-head token block and the patch token on which the intra-block transformation process has been executed (step S105).

[0081] The above-mentioned intra-block transformation process may be performed on at least one of the query, key, and value. For example, the intra-block transformation process may be performed on only one of the query, key, and value. Alternatively, the intra-block transformation process may be performed on two of the query, key, and value. Alternatively, the intra-block transformation process may be performed on all three of the query, key, and value.

[0082] (Specific operation example) Next, a specific operation example (particularly, an operation example of the above-mentioned intra-block transformation process) of the information processing device 10 according to the fifth embodiment will be described with reference to Fig. 15. Fig. 15 is a conceptual diagram showing intra-block transformation process of a patch token in the information processing device according to the fifth embodiment.

[0083] Consider the case where a patch token of HxWxC is transformed as shown in Figure 15. In this case, in the intra-block transformation process, tensor transformation is performed so that elements that are in the same position within each block (i.e., elements shown in the same color) are grouped together. As a result, the vertical direction of the figure is the number of channels (C), and the horizontal direction is the number of elements to be referenced (16 in this case). And the depth direction is the block size (3x3 in this case).

[0084] Here, the elements contained in one patch token block (in other words, elements in a local region) are lined up in the upper left part after the deformation. In this way, when the intra-block deformation process is performed, a 1 × 1 convolution layer (fully connected layer) can be applied within the block region.

[0085] (Technical effect) Next, technical effects obtained by the information processing device 10 according to the fifth embodiment will be described.

[0086] As described in Fig. 13 to Fig. 15, in the information processing device 10 according to the fifth embodiment, in the feature embedding process, the patch token is transformed so that a 1x1 convolution layer (fully connected layer) can be applied within the region of the block. In this way, it is possible to eliminate the lack of information about the local region caused by the division using the grid pattern. Therefore, for example, it is possible to suppress the deterioration of processing accuracy caused by the lack of information about the local region. Sixth Embodiment An information processing device 10 according to the sixth embodiment will be described with reference to Fig. 16 and Fig. 17. The sixth embodiment differs from the fifth embodiment in part of its configuration and operation, and other parts may be the same as the first to fifth embodiments. Therefore, hereinafter, parts that differ from the embodiments already described will be described in detail, and other overlapping parts will be omitted as appropriate.

[0087] (Functional configuration) First, the functional configuration of the information processing device 10 according to the sixth embodiment (particularly the configuration for realizing the functions of the feature embedding processing units 31, 32, and 33 and the feature conversion processing unit 37) will be described with reference to Fig. 16. Fig. 16 is a block diagram showing the functional configuration of the information processing device according to the sixth embodiment. In Fig. 16, the same elements as those shown in Fig. 8 are denoted by the same reference numerals.

[0088] As shown in Fig. 16, the information processing device 10 according to the sixth embodiment includes, as components for realizing the functions thereof, a generating unit 110, an expanding unit 120, a computing unit 130, a restoring unit 140, and a transforming unit 155. That is, the information processing device 10 according to the sixth embodiment is configured to further include a transforming unit 155 in addition to the configuration of the second embodiment described above (see Fig. 8). Note that the transforming unit 155 according to the sixth embodiment may have the same function as the transforming unit 150 according to the fifth embodiment described above, and may be, for example, a processing block realized by the above-mentioned processor 11 (see Fig. 1).

[0089] (Operation flow) Next, the flow of operation of the information processing device 10 according to the sixth embodiment (particularly, the feature conversion processing by the feature conversion processing unit 37) will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the flow of the feature conversion processing in the information processing device according to the sixth embodiment. Note that in Fig. 17, the same processes as those shown in Fig. 9 are denoted by the same reference numerals.

[0090] As shown in FIG. 17, when feature conversion processing by the information processing device 10 according to the sixth embodiment is started, the generation unit 110 first generates a sentence start token and a patch token corresponding to the input image, and divides the patch token into multiple regions (step S201).

[0091] Next, the restoration unit 140 performs restoration processing on the divided sentence head tokens to restore their sizes (step S202). Then, particularly in this embodiment, the transformation unit 140 executes intra-block transformation processing on one of the patch tokens (step S601). This processing may be the same as the intra-block transformation processing in the fifth embodiment described above (i.e., step S501 in FIG. 14).

[0092] Thereafter, the calculation unit 130 performs tensor transformation and integration on the restored sentence head token and the patch token that has been subjected to the intra-block transformation process (step S203).

[0093] (Technical effect) Next, technical effects obtained by the information processing device 10 according to the sixth embodiment will be described.

[0094] As described in FIG. 17, in the information processing device 10 according to the sixth embodiment, in the feature conversion process, the patch token is transformed so that a 1×1 convolution layer (fully connected layer) can be applied within the region of the block. In this way, it is possible to eliminate the lack of information about the local region caused by the division using the grid pattern. Therefore, for example, it is possible to suppress the decrease in processing accuracy caused by the lack of information about the local region.

[0095] In addition, the fifth and sixth embodiments described above may be realized in combination. That is, the intra-block transformation process may be performed on both the query, key, and value, and the calculation result of the self-attention mechanism.

[0096] The scope of each embodiment also includes a processing method in which a program that operates the configuration of each embodiment to realize the functions of the above-mentioned embodiments is recorded on a recording medium, the program recorded on the recording medium is read as code, and the program is executed on a computer. That is, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, not only the recording medium on which the above-mentioned program is recorded, but also the program itself is included in each embodiment.

[0097] Examples of recording media that can be used include floppy disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs. In addition, the scope of each embodiment is not limited to programs recorded on the recording media that execute processes by themselves, but also includes programs that operate on an OS in cooperation with other software and functions of an extension board to execute processes. Furthermore, the program itself may be stored in a server, and part or all of the program may be downloaded from the server to a user terminal.

[0098] <Additional Notes> The above-described embodiment may be further described as follows, but is not limited to the following.

[0099] (Appendix 1) The information processing device described in Appendix 1 is an information processing device comprising: a generation means for generating a sentence start token and a patch token corresponding to an input image; an expansion means for expanding the sentence start token to a sentence start token block having a size corresponding to the number of multiple patch token blocks obtained by dividing the patch token into areas according to a predetermined grid pattern; and a calculation means for calculating the sentence start token block and the patch token block based on a self-attention mechanism for each group of elements that have a common position within each block.

[0100] (Appendix 2) The information processing device described in Supplementary Note 2 is the information processing device described in Supplementary Note 1, further comprising a restoration means for restoring a feature corresponding to the start token block among the features obtained as a result of the calculation by the calculation means to the size of the start token before expansion.

[0101] (Appendix 3) The information processing device described in Appendix 3 is the information processing device described in Appendix 2, wherein the restoration means restores a feature corresponding to the start token block to the size of the start token before expansion by calculating the average value of elements contained in the feature.

[0102] (Appendix 4) The information processing device described in Appendix 4 is the information processing device described in Appendix 2, wherein the restoration means restores a feature corresponding to the start token block to the size of the start token before expansion by calculating the maximum value of the elements contained in the feature.

[0103] (Appendix 5) The information processing device described in Supplementary Note 5 is the information processing device described in any one of Supplementary Notes 1 to 4, further comprising a transformation means for transforming the patch tokens into tensors to which a 1×1 convolutional layer can be applied within each block.

[0104] (Appendix 6) The information processing device described in Supplementary Note 6 is the information processing device described in Supplementary Note 5, wherein the transformation means transforms tensors for at least one of a query, a key, a value, and a feature obtained as a result of the calculation of the self-attention mechanism.

[0105] (Appendix 7) The information processing method described in Supplementary Note 7 is an information processing method executed by at least one computer, which generates a sentence start token and a patch token corresponding to an input image, expands the sentence start token to a sentence start token block having a size corresponding to the number of multiple patch token blocks obtained by dividing the patch token into regions according to a predetermined grid pattern, and performs calculations on the sentence start token block and the patch token block based on a self-attention mechanism for each group of elements that have a common position within each block.

[0106] (Appendix 8) The recording medium described in Appendix 8 is a recording medium having recorded thereon a computer program for causing at least one computer to execute an information processing method, which includes generating a sentence start token and a patch token corresponding to an input image, expanding the sentence start token to a sentence start token block having a size corresponding to the number of multiple patch token blocks obtained by dividing the patch token into regions according to a predetermined grid pattern, and performing calculations on the sentence start token block and the patch token block based on a self-attention mechanism for each group of elements that have a common position within each block.

[0107] (Appendix 9) The computer program described in Appendix 9 is a computer program that causes at least one computer to execute an information processing method, which generates a sentence start token and a patch token corresponding to an input image, expands the sentence start token to a sentence start token block having a size corresponding to the number of multiple patch token blocks obtained by dividing the patch token into regions according to a predetermined grid pattern, and performs calculations on the sentence start token block and the patch token block based on a self-attention mechanism for each group of elements that have a common position within each block.

[0108] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and recording media incorporating such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0109] 10. Information processing device 11 Processors 20 Self-Attention Mechanism Unit 30 Feature Transformation Unit 31 Feature embedding processing section (query) 32 Feature embedding processing section (key) 33 Feature embedding processing section (value) 34 Correlation calculation section 35 Counting Processing Unit 36 Residual Processing Section 37 Feature conversion processing section 50 Conversion Blocks 55 Batch embedding processor 110 Generation part 120 Extension 130 Arithmetic section 140 Restoration Department 150,155 Deformation part

Claims

1. A generating means for generating a prefix token and a patch token corresponding to an input image; an expansion means for expanding the sentence start token to a sentence start token block having a size corresponding to the number of a plurality of patch token blocks obtained by dividing the patch token into regions according to a predetermined grid pattern; A calculation means for calculating, for each group of elements having a common position within each block, the sentence start token block and the patch token block based on a self-attention mechanism; An information processing device comprising:

2. a restoring unit that restores a feature corresponding to the start token block, among the feature values ​​obtained as a result of the calculation by the calculating unit, to a size of the start token before expansion, The information processing device according to claim 1 .

3. the restoration means restores the feature corresponding to the start token block to the size of the start token before expansion by calculating an average value of elements included in the feature. The information processing device according to claim 2 .

4. the restoration means restores the feature corresponding to the start token block to the size of the start token before expansion by calculating a maximum value of elements included in the feature. The information processing device according to claim 2 .

5. The patch tokens are transformed into tensors that can be subjected to a 1×1 convolutional layer in each block. The information processing device according to claim 1 .

6. The transformation means transforms a tensor for at least one of a query, a key, a value, and a feature obtained as a result of the operation of the self-attention mechanism, The information processing device according to claim 5 .

7. 1. A method of information processing implemented by at least one computer, comprising: Generate a prefix token and a patch token corresponding to the input image; expanding the beginning of the sentence token into an beginning of the sentence token block having a size corresponding to the number of a plurality of patch token blocks obtained by dividing the patch token into regions according to a predetermined grid pattern; For the sentence start token block and the patch token block, a calculation is performed based on a self-attention mechanism for each element group having a common position within each block. Information processing methods.

8. At least one computer Generate a prefix token and a patch token corresponding to the input image; expanding the beginning of the sentence token into an beginning of the sentence token block having a size corresponding to the number of a plurality of patch token blocks obtained by dividing the patch token into regions according to a predetermined grid pattern; For the sentence start token block and the patch token block, a calculation is performed based on a self-attention mechanism for each element group having a common position within each block. A computer program for executing an information processing method.

Images