Image processing apparatus, image processing method, and storage medium

Abstract

To speed up image processing by reducing tokens to be calculated in ViT (VisionTransformer).SOLUTION: Extracting, from a first input feature amount input at a first time point, a changed portion that has changed from a second output feature amount output at a second time point prior to the first time point; And a sparse MSA (Multi-HeadSelf-Attention) processing unit including an extraction unit that generates a change matrix based on a change portion of a first matrix indicating a first input feature amount, a Linear processing unit that calculates a Linear matrix which is a matrix product of the change matrix and a weight matrix, and a replacement unit that replaces a portion corresponding to a change portion of a second matrix indicating a second output feature amount with the Linear matrix calculated by the Linear processing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program that perform image processing at high speed. [Background technology]

[0002] One well-known technology for speeding up image processing is the Vision Transformer (ViT), which applies the Transformer used in natural language processing to image processing.However, unlike conventional Convolutional Neural Networks (CNNs), ViT has a complex structure, which requires a very large amount of calculation, making it difficult to perform high-resolution tasks at high speed.

[0003] As a related technique, Patent Document 1 discloses an image processing device that improves the performance of a feature extractor using ViT. The image processing device in Patent Document 1 first divides an input image into a sequence of partial images, which are multiple partial images, and converts the partial images into tokens with fixed-dimensional vectors, thereby converting the divided sequence of partial images into a token sequence. Next, the image processing device in Patent Document 1 adds class tokens, which have vectors of the same dimension as two or more tokens, to the token sequence, and updates the token sequence with the added class tokens based on the associations between the tokens to obtain a final encoded representation sequence. Next, the image processing device in Patent Document 1 obtains encoded representations corresponding to class tokens from the encoded representation sequence as a class token encoded representation sequence, and integrates the class token encoded representation sequence to obtain a feature vector of the input image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-042973 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the image processing device of Patent Document 1 does not reduce the amount of calculation of ViT by targeting only tokens that have movement as the calculation target.

[0006] An example of an objective of the present disclosure is to speed up image processing in ViT by reducing the number of tokens to be calculated. [Means for solving the problem]

[0007] In order to achieve the above object, an image processing device according to one aspect of the present disclosure includes: an extraction unit that extracts a change portion of a first input feature input at a first time point that has changed from a second output feature output at a second time point that is earlier than the first time point, and generates a change matrix based on the change portion of a first matrix indicating the first input feature; a sparse MSA (Multi-Head Self-Attention) processing unit including: a linear processing unit that calculates a linear matrix which is a matrix product of the change matrix and a weight matrix; and a replacement unit that replaces a part of a second matrix indicating the second output feature amount, which corresponds to the change part, with the linear matrix; The present invention is characterized by having the following.

[0008] In order to achieve the above object, an image processing method according to one aspect of the present disclosure includes: The image processing device extracting a change portion of a first input feature input at a first time point that has changed from a second output feature output at a second time point that is earlier than the first time point, and generating a change matrix based on the change portion of a first matrix indicating the first input feature; calculating a linear matrix which is a matrix product of the change matrix and a weight matrix, and replacing a portion of a second matrix indicating the second output feature quantity, which corresponds to the change portion, with the linear matrix; and performing sparse MSA (Multi-Head Self-Attention) processing. It is characterized by:

[0009] Furthermore, in order to achieve the above object, a program according to one aspect of the present disclosure comprises: On the computer, extracting a change portion of a first input feature input at a first time point that has changed from a second output feature output at a second time point that is earlier than the first time point, and generating a change matrix based on the change portion of a first matrix indicating the first input feature; calculating a linear matrix which is a matrix product of the change matrix and a weight matrix, and replacing a portion of a second matrix indicating the second output feature quantity, which corresponds to the change portion, with the linear matrix; and executing sparse MSA (Multi-Head Self-Attention) processing. It is characterized by: [Effects of the Invention]

[0010] As described above, according to the present disclosure, the number of tokens to be calculated in ViT can be reduced, thereby speeding up image processing. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of an image processing device. [Figure 2] FIG. 2 is a diagram illustrating an example of a system including an image processing device. [Figure 3] FIG. 3 is a diagram for explaining an MSA processing unit using normal attention and linear attention. [Figure 4] FIG. 4 is a diagram for explaining (1) the token reduction method of MSA in normal attention. [Figure 5] FIG. 5 is a diagram illustrating an example of the linear processing unit and the replacement unit. [Figure 6] FIG. 6 is a diagram for explaining the token reduction method of MSA using SFDA in (2) normal attention. [Figure 7]FIG. 7 is a diagram illustrating an example of the SFDA processing unit. [Figure 8] FIG. 8 is a diagram for explaining the token reduction method of MSA in (3) Linear Attention. [Figure 9] FIG. 9 is a diagram for explaining a token reduction method of MSA using SFDLA in (4) Linear Attention. [Figure 10] FIG. 10 is a diagram illustrating an example of the SFDLA processing unit. [Figure 11] FIG. 11 is a diagram for explaining the operation of the image processing device. [Figure 12] FIG. 12 is a diagram illustrating an example of a computer that realizes the image processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described with reference to the drawings. In the drawings described below, elements having the same or corresponding functions are denoted by the same reference numerals, and repeated description thereof may be omitted.

[0013] (Embodiment) The configuration of an image processing device in an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an example of an image processing device.

[0014] [Device configuration] The image processing device shown in FIG. 1 is a device that speeds up image processing by reducing the number of tokens (patches) to be calculated in ViT. An image processing device using ViT is, for example, a device that performs object detection, pose estimation, etc. Specifically, the image processing device is a device that performs behavior recognition and analysis using a surveillance camera. Also, as shown in FIG. 1, the image processing device 10 has an extraction unit (extraction means) 11 and a sparse MSA (Multi-Head Self-Attention) processing unit (sparse MSA processing means) 12.

[0015] The extraction unit 11 extracts a change portion of a first input feature input at a first time point that has changed from a second output feature output at a second time point prior to the first time point, and generates a change matrix based on the change portion of a first matrix indicating the first input feature.

[0016] The sparse MSA processing unit 12 includes a linear processing unit (linear processing means) that calculates a linear matrix, which is the matrix product of a change matrix and a weight matrix, and a substitution unit (substitution processing) that replaces the part of the second matrix indicating the second output feature corresponding to the change part with a linear matrix.

[0017] In this manner, in the embodiment, by adopting sparse MSA processing in ViT, it is possible to reduce the number of tokens (patches) to be calculated, thereby speeding up image processing.

[0018] [System Configuration] Next, the configuration of the image processing device 10 in the embodiment will be described in more detail with reference to Fig. 2. Fig. 2 is a diagram showing an example of a system having an image processing device.

[0019] As shown in FIG. 2, the system 100 in this embodiment includes an image processing device 10 and a storage device 20.

[0020] The image processing device 10 is, for example, a CPU (Central Processing Unit), a programmable device such as an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a circuit equipped with one or more of these, a server computer, a personal computer, a mobile terminal, or other device.

[0021] The storage device 20 is a database, a server computer, a circuit having a memory, etc. The storage device 20 stores, for example, information such as at least an input image 21, an output image 22, and parameters 23. Note that, although the storage device 20 is provided outside the image processing device 10 in the example of Fig. 2, it may also be provided inside the image processing device 10.

[0022] Input images 21 are a plurality of images captured in time series by an imaging device. Output images 22 are a plurality of images generated using sparse MSA processing. Parameters 23 are various parameters used in the sparse MSA processing.

[0023] ●Explain image processing devices in detail. The image processing device 10 includes a generating unit 13, an extracting unit 11, a sparse MSA processing unit 12, and an output unit 14.

[0024] The generation unit 13 first acquires images captured in time series by an imaging device (not shown). The images may be acquired directly from the imaging device, or may acquire input images 21 from the storage device 20.

[0025] Next, the generation unit 13 divides the acquired image into a preset number m of images, where m represents the number of patch dimensions, the number of tokens, the number of patches, or the like.

[0026] Next, the generation unit 13 generates a feature vector having d preset feature amounts for each of the m divided images (patches). That is, the generation unit 13 generates a first matrix X (n×m×d) indicating the input feature amounts. Here, d represents the number of feature dimensions, channel size, number of channels, etc., and n represents the batch size. Note that hereinafter, (n×m×d) indicates that there are n matrices with m rows and d columns.

[0027] For details of the generation unit 13, please refer to reference [1]. [1]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, [online], Published: 13 Jan 2021, Last Modified: 17 Sept 2023, International Conference on Learning Representations ICLR 2021, [Searched on September 27, 2023], Internet<URL:https: / / openreview.net / forum?id=YicbFdNTTy>

[0028] The extraction unit 11 first extracts a change portion of a first input feature (Xt(n×m×d)) input at a first time point t that has changed from a second output feature (Yt-1(n×m×d)) output at a second time point t-1 prior to the first time point t.

[0029] Next, the extraction unit 11 generates a change matrix Xc(m'xd) based on the change portion of the first matrix Xt(nxmxd) indicating the first input feature, where m' represents the number of patch dimensions, the number of tokens, the number of patches, etc. of the change portion.

[0030] The sparse MSA processor 12 calculates a linear matrix L(m'×d) which is the matrix product of the change matrix Xc(m'×d) and the weight matrix W(d×d), and replaces the part corresponding to the change part of the second matrix Yt-1(n×m×d) indicating the second output feature with the linear matrix L(m'×d). Details of the sparse MSA processor 12 will be described later.

[0031] The output unit 14 stores the first output feature (Yt(n×m×d)) generated by the sparse MSA processing unit 12 in an output image 22 of the storage device 20. The output unit 14 may also output the output image 22 to an output device (not shown).

[0032] The output device outputs the output image 22 converted into an outputtable format. The output device is, for example, an image display device using a liquid crystal, an organic EL (Electro Luminescence), or a CRT (Cathode Ray Tube). Furthermore, the image display device may also include an audio output device such as a speaker.

[0033] ●The sparse MSA processing section is explained in detail. First, we will explain a conventional MSA processing unit. Figure 3 is a diagram for explaining an MSA processing unit using normal attention and linear attention. A in Figure 3 shows an MSA processing unit for normal attention, and B in Figure 3 shows an MSA processing unit for linear attention. In the MSA processing unit for normal attention, the computational complexity of matrix multiplication in the linear processing unit is n·m·d 2 (Hereafter, "·" denotes multiplication), and the computational cost of matrix multiplication after attention is n·m 2 In addition, the MSA processing of Linear Attention is all matrix multiplication, and the computational complexity is n m d 2 The deactivation function processing unit is, for example, a process such as ReLU (Rectified Linear Unit) in a neural network.

[0034] Next, the sparse MSA processor is a process that reduces the computational load of the conventional MSA processor using normal attention and linear attention. For example, the sparse MSA processors shown in (1), (2), (3), and (4) are possible. (1) A token reduction method in normal attention, (2) A token reduction method using SFDA (Sparse Frame Difference Attention) in normal attention, (3) A token reduction method in linear attention, and (4) A token reduction method using SFDLA (Sparse Frame Difference Linear Attention) in linear attention are possible. Specifically, the sparse MSA processors shown in (1), (2), (3), and (4) reduce the computational load by reducing the tokens to be calculated.

[0035] (1) Token reduction method in standard attention Fig. 4 is a diagram for explaining the token reduction method of MSA in (1) normal attention. In the example of Fig. 4, the sparse MSA processing unit 12 has a linear processing unit (first linear processing unit) 12a, a linear processing unit (second linear processing unit) 12b, a linear processing unit (third linear processing unit) 12c, a permutation unit (first permutation unit) 12d, a permutation unit (second permutation unit) 12e, a permutation unit (third permutation unit) 12f, an attention unit (first attention unit) 12g, a softmax processing unit 12h, and an attention unit (second attention unit) 12i.

[0036] The linear processing unit 12a calculates a linear matrix L1(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W1(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the linear matrix L1(m'×d) to the substitution unit 12d.

[0037] The linear processing unit 12b calculates a linear matrix L2(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W2(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the linear matrix L2(m'×d) to the substitution unit 12e.

[0038] The linear processing unit 12c calculates a linear matrix L3(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W3(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the linear matrix L3(m'×d) to the substitution unit 12f.

[0039] 5 is a diagram illustrating an example of the linear processing unit and the replacement unit. Specifically, as shown in FIG. 5, the linear processing unit calculates the matrix product of the change matrix Xc(m′×d) and the weight matrix W(d×d) to calculate the linear matrix L(m′×d).

[0040] The substitution unit 12d is provided at a stage subsequent to the linear processing unit 12a. The substitution unit 12d also substitutes (updates) a portion corresponding to a changed portion of the second matrix Yt-1(n×m×d) indicating the second output feature with the linear matrix L1(m′×d) calculated by the linear processing unit 12a to calculate a substitution matrix O1(n×m×d), and outputs the resulting matrix to the attention unit 12g.

[0041] The substitution unit 12e is provided at a stage subsequent to the linear processing unit 12b. The substitution unit 12e also substitutes (updates) a portion corresponding to a changed portion of the second matrix Yt-1(n×m×d) indicating the second output feature with the linear matrix L2(m′×d) calculated by the linear processing unit 12b to calculate a substitution matrix O2(n×m×d), and outputs the result to the attention unit 12g.

[0042] The substitution unit 12f is provided at a stage subsequent to the linear processing unit 12c. The substitution unit 12f also substitutes (updates) a portion corresponding to a changed portion of the second matrix Yt-1(n×m×d) indicating the second output feature with the linear matrix L3(m′×d) calculated by the linear processing 12c to calculate a permutation matrix O3(n×m×d), and outputs the permutation matrix O3(n×m×d) to the attention unit 12i.

[0043] Specifically, as shown in FIG. 4, the substitution unit uses matrix sum to substitute the part (shaded part) corresponding to the changed part of the second matrix Yt-1(n×m×d)) with the linear matrix L(m′×d) (shaded part: updated part).

[0044] The attention unit 12g calculates the matrix product of the permutation matrix O1 (n×m×d) calculated by the permutation unit 12d and the permutation matrix O2 (n×m×d) calculated by the permutation unit 12e to generate a matrix S (n×m×m) (third matrix) and outputs it to the softmax processing unit 12h.

[0045] The softmax processing unit 12h normalizes the matrix S(n×m×m) output by the attention unit 12g using a softmax function to calculate a normalized matrix Smax(n×m×m), and outputs it to the attention unit 12i.

[0046] The attention unit 12i calculates the matrix product of the permutation matrix O3 (n×m×d) calculated by the permutation unit 12f and the normalization matrix Smax (n×m×m), and outputs a matrix Yt (n×m×d) indicating a new second output feature.

[0047] In this way, by adopting the sparse MSA processing of (1), the extraction unit 11 extracts the changed parts of the tokens, the linear processing units 12a, 12b, and 12c perform linear processing using only the compressed change matrix (two-dimensional matrix) corresponding to the changed parts, and the replacement units 12d, 12e, and 12f replace the changed parts, which reduces the amount of calculation compared to conventional MSA processing, and as a result, image processing can be speeded up.

[0048] In addition, since the calculation results of the previous token are reused except for important tokens that have changed, there is little deterioration in accuracy. Therefore, while many methods recover accuracy by re-learning, our method does not require costly re-learning.

[0049] (2) Token reduction method using SFDA in regular attention In the method (2), instead of the replacement unit 12d, replacement unit 12e, and attention unit 12g in FIG. 3, an SFDA processing unit 12h is provided after the linear processing unit 12a and the linear processing unit 12b.

[0050] Fig. 6 is a diagram for explaining (2) a token reduction method of MSA using SFDA in normal attention. In the example of Fig. 6, the sparse MSA processing unit 12 has a linear processing unit (first linear processing unit) 12a, a linear processing unit (second linear processing unit) 12b, a linear processing unit (third linear processing unit) 12c, an SFDA (Sparse Frame Difference Attention) processing unit 12k, a substitution unit (third substitution unit) 12f, a softmax processing unit 12h, and an attention unit (second attention unit) 12i.

[0051] The linear processing unit 12a calculates a linear matrix LQ(m'×d) (Q: query) using the change matrix Xc(m'×d) and a weighting matrix W1(d×d) previously stored in the parameter 23 of the storage device 20.

[0052] The linear processing unit 12b calculates a linear matrix LK(m'×d) (K: key) using the change matrix Xc(m'×d) and a weighting matrix W2(d×d) previously stored in the parameter 23 of the storage device 20.

[0053] The linear processing unit 12c calculates a linear matrix LV(m'×d) (V: value) using the change matrix Xc(m'×d) and a weighting matrix W3(d×d) stored in advance in the parameter 23 of the storage device 20.

[0054] The SFDA processing unit 12k first calculates a K-change matrix using the linear matrix LQ(m'×d) calculated by the linear processing unit 12a and a K-matrix (a first matrix Xt(n×m×d) indicating the first input feature amount). The SFDA processing unit 12k also calculates a Q-change matrix using the linear matrix LK(m'×d) calculated by the linear processing unit 12b and a Q-matrix (a first matrix Xt(n×m×d) indicating the first input feature amount).

[0055] Next, the SFDA processing unit 12k replaces (updates) the part corresponding to the change part of the second matrix Yt-1(n×m×d) with a K change matrix and a Q change matrix to generate a matrix SFDA(n×m×d) (third matrix), and outputs it to the softmax processing unit 12h.

[0056] Fig. 7 is a diagram for explaining an example of an SFDA processing unit. Specifically, as shown in Fig. 7, the SFDA processing unit 12k calculates a K change matrix using a linear matrix LQt(m'xd) and a K matrix Kt(mxd) (first matrix Xt(nxmxd)) at a first time point t. In addition, the SFDA processing unit 12k also calculates a Q change matrix using a linear matrix LKt(m'xd) and a Q matrix Qt(mxd) (first matrix Xt(nxmxd)) at the first time point t.

[0057] Next, the SFDA processing unit 12k replaces (updates) the part corresponding to the change part of the second matrix Yt-1 (n×m×d) at the second time point t-1 using the transposed K change matrix and the Q change matrix to generate a matrix SFDAt (n×m×m) (third matrix).

[0058] The softmax processing unit 12h normalizes the matrix SFDA (n×m×m) output by the SFDA processing unit 12k using a softmax function to calculate a normalized matrix Smax (n×m×m), and outputs it to the attention unit 12i.

[0059] The substitution unit 12f is provided at a stage subsequent to the linear processing unit 12c. The substitution unit 12f also substitutes (updates) a portion corresponding to a changed portion of the second matrix Yt-1(n×m×d) indicating the second output feature with the linear matrix LV(m′×d) calculated by the linear processing unit 12c to calculate a permutation matrix OV(n×m×d), and outputs the permutation matrix OV(n×m×d) to the attention unit 12i.

[0060] The attention unit 12i calculates the matrix product of the permutation matrix OV (n×m×d) calculated by the permutation unit 12f and the normalization matrix Smax (n×m×m), and outputs a matrix Yt (n×m×d) indicating a new second output feature.

[0061] In this way, by adopting the sparse MSA processing of (2), the extraction unit 11 extracts the changed parts of the tokens, the linear processing units 12a, 12b, and 12c perform linear processing using only the compressed change matrix (two-dimensional matrix) corresponding to the changed parts, and the SFDA processing unit 12k and the replacement unit 12f replace the changed parts, so the amount of calculation can be reduced compared to conventional MSA processing, and as a result, image processing can be speeded up.

[0062] In addition, since the calculation results of the previous token are reused for tokens other than those that have changed, there is little degradation in accuracy. Therefore, while many methods recover accuracy by re-training, this method does not require costly re-training. Compared to method (1), the amount of attention calculation can be significantly reduced, enabling even faster image processing.

[0063] (3) Token reduction method in Linear Attention Fig. 8 is a diagram for explaining the token reduction method of MSA in (3) Linear Attention. In the example of Fig. 8, the sparse MSA processing unit 12 includes a linear processing unit (first linear processing unit) 12a, a linear processing unit (second linear processing unit) 12b, a linear processing unit (third linear processing unit) 12c, a deactivation function processing unit 12l, a deactivation function processing unit 12m, a substitution unit (fourth substitution unit) 12n, a substitution unit (fifth substitution unit) 12o, an attention unit (third attention unit) 12p, an attention unit (fourth attention unit) 12q, and a substitution unit (sixth substitution unit) 12r.

[0064] The linear processing unit 12a calculates a linear matrix L1(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W1(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the calculated matrix L1(m'×d) to the deactivation function processing unit (first deactivation function processing unit) 12l.

[0065] The linear processing unit 12b calculates a linear matrix L2(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W2(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the calculated matrix to the deactivation function processing unit (second deactivation function processing unit) 12m.

[0066] The linear processing unit 12c calculates a linear matrix L3(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W3(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the linear matrix L3(m'×d) to the substitution unit 12o.

[0067] The deactivation function processing unit 12l is provided at the subsequent stage of the linear processing unit 12a. The deactivation function processing unit 12l also performs deactivation function processing on the linear matrix L1(m'×d) calculated by the linear processing unit 12a to calculate a matrix ReLU1(m'×d), and outputs the matrix ReLU1(m'×d) to the attention unit 12q.

[0068] The deactivation function processing unit 12m is provided at a stage subsequent to the linear processing unit 12b. The deactivation function processing unit 12m performs deactivation function processing on the linear matrix L2(m'×d) calculated by the linear processing unit 12b to calculate a matrix ReLU2(m'×d), and outputs the matrix ReLU2(m'×d) to the substitution unit 12n.

[0069] The permutation unit 12n is provided at a stage subsequent to the deactivation function processing unit 12m. The permutation unit 12n also calculates (updates) a permutation matrix O4(n×m×d) by replacing (updating) a portion corresponding to a changed portion of the second matrix Yt-1(n×m×d) indicating the second output feature with the matrix ReLU2(m′×d) calculated by the deactivation function processing unit 12m, and outputs the permutation matrix O4(n×m×d) to the attention unit 12p.

[0070] The permutation unit 12o is provided at a stage subsequent to the linear processing unit 12c. The permutation unit 12o also calculates (updates) a permutation matrix O5(n×m×d) by replacing (updating) a portion corresponding to a changed portion of the second matrix Yt-1(n×m×d) indicating the second output feature with the linear matrix L3(m′×d) calculated by the linear processing unit 12c, and outputs the permutation matrix O5(n×m×d) to the attention unit 12p.

[0071] The attention unit 12p calculates the matrix product of the permutation matrix O4 (n×m×d) calculated by the permutation unit 12n and the permutation matrix O5 (n×m×d) calculated by the permutation unit 12o, and outputs a matrix A1 (n×d×d) (fourth matrix).

[0072] The attention unit 12q calculates the matrix product of the matrix ReLU1(m'×d) calculated by the deactivation function processing unit 12l and the matrix A1(n×d×d) calculated by the attention unit 12p, and outputs the matrix A2(m'×d) (fifth matrix) to the substitution unit 12r.

[0073] The replacement unit 12r replaces the part corresponding to the changed part of the second matrix Yt-1(n×m×d) indicating the second output feature with the matrix A2(m′×d) calculated by the attention unit 12q, and outputs the matrix Yt(n×m×d) indicating the new second output feature.

[0074] In this way, by adopting the sparse MSA processing of (3), the extraction unit 11 extracts the changed parts of the tokens, the linear processing units 12a, 12b, and 12c perform linear processing using only the compressed change matrix (two-dimensional matrix) corresponding to the changed parts, and the replacement units 12n, 12o, and 12r replace the changed parts, so the amount of calculation can be reduced compared to the conventional MSA processing using Linear Attention. As a result, image processing can be speeded up.

[0075] (4) Token reduction method using SFDLA in Linear Attention In the method (4), instead of the replacement unit 12n, replacement unit 12o, and attention unit 12p in FIG. 7, an SFDLA processing unit 12s is provided after the deactivation function processing unit 12m and the linear processing unit 12c.

[0076] 9 is a diagram for explaining the token reduction method of MSA using SFDLA in (4) Linear Attention. In the example of FIG. 9, the sparse MSA processing unit 12 includes a linear processing unit (first linear processing unit) 12a, a linear processing unit (second linear processing unit) 12b, a linear processing unit (third linear processing unit) 12c, a deactivation function processing unit 12l, a deactivation function processing unit 12m, an SFDLA processing unit 12s, an attention unit (fifth attention unit) 12t, and a substitution unit (seventh substitution unit) 12u.

[0077] The linear processing unit 12a calculates a linear matrix L1(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W1(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the calculated matrix L1(m'×d) to the deactivation function processing unit (first deactivation function processing unit) 12l.

[0078] The linear processing unit 12b calculates a linear matrix L2(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W2(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the calculated matrix to the deactivation function processing unit (second deactivation function processing unit) 12m.

[0079] The linear processing unit 12c calculates a linear matrix L3(m'×d) using the change matrix Xc(m'×d) and a weighting matrix W3(d×d) previously stored in the parameter 23 of the storage device 20, and outputs the linear matrix L3(m'×d) to the SFDLA processing unit 12s.

[0080] The deactivation function processing unit 12l is provided at the subsequent stage of the linear processing unit 12a. The deactivation function processing unit 12l also performs deactivation function processing on the linear matrix L1(m'×d) calculated by the linear processing unit 12a to calculate a matrix ReLU1(m'×d), and outputs the matrix ReLU1(m'×d) to the attention unit 12t.

[0081] The deactivation function processing unit 12m is provided after the linear processing unit 12b. The deactivation function processing unit 12m performs deactivation function processing on the linear matrix L2(m'×d) calculated by the linear processing unit 12b to calculate a matrix ReLU2(m'×d), and outputs the matrix ReLU2(m'×d) to the SFDLA processing unit 12s.

[0082] In the first frame, the SFDLA processing unit 12s calculates matrix A, which is the matrix product of the part of the matrix ReLU2(m'×d) (Q1 matrix) calculated by the deactivation function processing unit 12m that will never be updated from the second frame onwards, and the part of the linear matrix L3(m'×d) (K1 matrix) calculated by the linear processing unit 12c that will never be updated from the second frame onwards.

[0083] In addition, in the first frame, the SFDLA processing unit 12s calculates a B1 matrix, which is the matrix product of a part of the matrix ReLU2(m'×d) calculated by the deactivation function processing unit 12m that is updated once from the second frame onwards and a part of the linear matrix L3(m'×d) calculated by the linear processing unit 12c that is updated once from the second frame onwards.

[0084] The non-updated portion refers to the x and y coordinates that were not extracted by the extraction unit 11. The extracted positions are determined to be important regions, and are updated by multiplication performed by the SFDLA processing unit 12s, but the previous multiplication results are reused for the other portions and are not updated.

[0085] Next, the SFDLA processing unit 12s calculates the matrix sum of the A matrix and the B1 matrix, and outputs the C1 matrix (SFDLA(d×d)) of the first frame to the attention unit 12t.

[0086] Furthermore, in the second frame and thereafter, the SFDLA processing unit 12s reflects the update token in the matrix ReLU2(m'×d) (Qi-1 matrix) calculated by the deactivation function processing unit 12m and the linear matrix L3(m'×d) (Ki-1 matrix) calculated by the linear processing unit 12c, and calculates the matrix ReLU2(m'×d) (Qi matrix) and the linear matrix L3(m'×d) (Ki matrix) calculated by the linear processing unit 12c in the third frame, where i is an integer equal to or greater than 2.

[0087] The updated token refers to a token that has been extracted by the extraction unit 11 and determined to be important.

[0088] Next, the SFDLA processing unit 12s calculates the Bi matrix, which is the matrix product of the part of the matrix ReLU2(m'×d) (Qi matrix) calculated by the deactivation function processing unit 12m that is updated once from the second frame onwards and the part of the linear matrix L3(m'×d) (Ki matrix) calculated by the linear processing unit 12c that is updated once from the second frame onwards.

[0089] Next, the SFDLA processing unit 12s calculates the matrix sum of the A matrix and the Bi matrix, and outputs the Ci matrix (SFDLA(d×d)) for the second frame and onward to the attention unit 12t.

[0090] FIG. 10 is a diagram illustrating an example of the SFDLA processing unit. In the SFDLA processing unit 12s, simply saving the calculation results of the previous frame (frame: image) requires a large amount of memory. In other words, d·d·m worth of memory is required. Therefore, the calculation method for Frame #1 and Frame #2 onwards is changed to reduce the required memory. Note that the batch size is set to a value greater than 1.

[0091] The calculation of Frame #1 will be explained. First, in Frame #1 of Figure 10, the matrix product of matrix Q1 (ReLU2(m' x d) of Frame #1) and matrix K1 (Linear matrix L3(m' x d) of Frame #1) is calculated for tokens that have not been updated since Frame #2 (blank area in Figure 10: other than the shaded area), and this is used as matrix A (K change matrix).

[0092] Next, the matrix product of the matrix Q1 and the matrix K1 for the tokens (shaded area) that are updated once after Frame #2 is calculated to calculate the matrix B1.

[0093] Next, matrix C1 is calculated by calculating the matrix sum of matrix A and matrix B1 (mathematically equivalent to a normal matrix multiplication).

[0094] The calculation of Frame #i from Frame #2 onwards will be explained. First, from Frame #2 onwards in Figure 10, the update token is reflected in matrix Qi-1 (ReLU2(m'×d) of Frame #n-1) and matrix Ki-1 (Linear matrix L3(m'×d) of Frame #i-1), and matrices Qi and Ki are calculated.

[0095] Next, the matrix product of the tokens (shaded area) that are updated once after Frame #2 is calculated using the matrices Qi and Ki to calculate the matrix Bi.

[0096] Next, matrix Ci is calculated by calculating the matrix sum of matrix A and matrix Bi (mathematically equivalent to a normal matrix multiplication).

[0097] In this way, by using SFDLA processing, the maximum memory usage can be reduced to 2·d·m.

[0098] The attention unit 12t calculates the matrix product of the matrix ReLU1(m'×d) calculated by the deactivation function processing unit 12l and the matrix SFDLA(d×d) generated by the SFDLA processing unit 12s, and outputs the matrix A3(m'×d) (seventh matrix) to the substitution unit 12u.

[0099] The substitution unit 12u substitutes the part corresponding to the changed part of the second matrix Yt-1(n×m×d) indicating the second output feature with the matrix A3(m′×d) (seventh matrix) calculated by the attention unit 12t, and outputs the matrix Yt(n×m×d) indicating the new second output feature.

[0100] In this way, by adopting the sparse MSA processing of (4), the extraction unit 11 extracts the changed parts of the tokens, the linear processing units 12a, 12b, and 12c perform linear processing using only the compressed change matrix (two-dimensional matrix) corresponding to the changed parts, and the SFDLA processing unit 12s and the replacement unit 12u replace the changed parts, so the amount of calculation can be reduced compared to the conventional MSA processing using linear attention, and as a result, image processing can be speeded up.

[0101] [Device operation] Next, the operation of the image processing device in the embodiment will be described with reference to FIG. 11. FIG. 11 is a diagram for explaining the operation of the image processing device. In the following description, the diagram will be referenced as appropriate. In addition, in the embodiment, an image processing method is implemented by operating the image processing device. Therefore, the description of the image processing method in the embodiment will be replaced by the description of the operation of the image processing device below.

[0102] As shown in FIG. 11, first, the generating unit 13 divides an image into m images, and generates a feature vector having d feature amounts for each of the m divided images (step A1).

[0103] Specifically, in step A1, the generation unit 13 first acquires images captured in time series by an imaging device (not shown). Next, in step A1, the generation unit 13 divides the acquired images into a preset number of m images. Next, in step A1, the generation unit 13 generates a feature vector having a preset number of d feature amounts for each of the m divided images (patches). That is, the generation unit 13 generates a first matrix X (n×m×d) indicating the input feature amounts.

[0104] Next, the extraction unit 11 extracts a change portion of the first input feature input at the first time point that has changed from the second output feature output at the second time point, and generates a change matrix based on the change portion (step A2).

[0105] Specifically, in step A2, the extraction unit 11 extracts the changed portion of the first input feature (Xt(n×m×d)) input at the first time point t that has changed from the second output feature (Yt-1(n×m×d)) output at the second time point t-1 prior to the first time point t.

[0106] Next, in step A2, the extraction unit 11 generates a change matrix Xc(m'xd) based on the change portion of the first matrix Xt(nxmxd) indicating the first input feature amount.

[0107] The sparse MSA processor 12 calculates a linear matrix L(m'×d) which is the matrix product of the change matrix Xc(m'×d) and the weight matrix W(d×d), and replaces the part corresponding to the change part of the second matrix Yt-1(n×m×d) indicating the second output feature with the linear matrix L(m'×d) (step A3). For details of the sparse MSA processor 12, please refer to the above (1) to (4).

[0108] The output unit 14 stores the first output feature (Yt(n×m×d)) generated by the sparse MSA processing unit 12 in the output image 22 of the storage device 20 and / or outputs the output image 22 to an output device (not shown) (step A4).

[0109] [Effects of the embodiment] As described above, according to the embodiment, by adopting the sparse MSA processes (1) to (4) in ViT, it is possible to reduce the number of tokens (patches) to be calculated, thereby speeding up image processing.

[0110] [program] The program in the embodiment may be a program that causes a computer to execute steps A1 to A4 shown in Fig. 11. By installing and executing this program on a computer, the image processing device and image processing method in the embodiment can be realized. In this case, the processor of the computer functions as a generation unit 13, an extraction unit 11, a sparse MSA processing unit 12, and an output unit 14 to perform processing.

[0111] The program in the embodiment may be executed by a computer system constructed by a plurality of computers, in which case, for example, each computer may function as one of the generating unit 13, the extracting unit 11, the sparse MSA processing unit 12, and the output unit 14.

[0112] [Physical configuration] A computer that realizes the image processing device by executing a program in the embodiment will now be described with reference to Fig. 12. Fig. 12 is a diagram for explaining an example of a computer that realizes the image processing device in the embodiment.

[0113] 12, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU or an FPGA in addition to or instead of the CPU 111.

[0114] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0115] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0116] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0117] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0118] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0119] Note that the image processing device 10 in the embodiment can be realized not by a computer on which a program is installed, but by hardware corresponding to each unit, such as an electronic circuit. Furthermore, the image processing device 10 may be partially realized by a program and the remaining unit by hardware. In the embodiment, the computer is not limited to the computer shown in FIG. 12.

[0120] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Industrial Applicability]

[0121] According to the above description, the ViT can speed up image processing by reducing the number of tokens to be calculated, and is useful in fields where the ViT is required. [Explanation of symbols]

[0122] 10 Image processing device 11 Extraction part 12 Sparse MSA processing section 13 Generation part 14 Output section 20 Storage device 100 systems 110 Computer 111 CPU 112 main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display Device 120 Recording Media 121 Bus

Claims

1. an extraction means for extracting a change portion of a first input feature input at a first time point that has changed from a second output feature output at a second time point that is earlier than the first time point, and for generating a change matrix based on the change portion of a first matrix indicating the first input feature; a sparse MSA (Multi-Head Self-Attention) processing means including: a linear processing unit that calculates a linear matrix which is a matrix product of the change matrix and a weight matrix; and a replacement unit that replaces a portion of a second matrix indicating the second output feature amount, which corresponds to the change portion, with the linear matrix; An image processing device having:

2. The sparse MSA processing means a first substitution unit provided after the first linear processing unit, a second substitution unit provided after the second linear processing unit, and a third substitution unit provided after the third linear processing unit; a first attention unit that calculates a matrix product of the matrix calculated by the first permutation unit and the matrix calculated by the second permutation unit, and outputs a third matrix; a softmax processing unit that normalizes the third matrix and outputs a normalized matrix; a second attention unit that calculates a matrix product of the matrix calculated by the third permutation unit and the normalized matrix, and outputs a matrix indicating a new second output feature amount; The image processing device according to claim 1 .

3. The sparse MSA processing means a SFDA (Sparse Frame Difference Attention) processing unit is provided at a subsequent stage of the first linear processing unit and the second linear processing unit, instead of the first substitution unit, the second substitution unit, and the first attention unit; The SFDA processing section Calculating a K change matrix which is a matrix product of the linear matrix calculated by the first linear processing unit and the first matrix, and a Q change matrix which is a matrix product of the linear matrix calculated by the second linear processing unit and the first matrix; generating the third matrix by replacing the portion of the second matrix corresponding to the changed portion with the K-change matrix and the Q-change matrix; The image processing device according to claim 2 .

4. The sparse MSA processing means a first deactivation function processing unit provided downstream of the first linear processing unit, which performs deactivation function processing on a first linear matrix calculated by the first linear processing unit; and a second deactivation function processing unit provided downstream of the second linear processing unit, which performs deactivation function processing on a second linear matrix calculated by the second linear processing unit. a fourth permutation unit provided after the second deactivation function processing unit, and a fifth permutation unit provided after the third linear processing unit; a third attention unit that calculates a matrix product of the matrix calculated by the fourth permutation unit and the matrix calculated by the fifth permutation unit, and outputs a fourth matrix; a fourth attention unit that calculates a matrix product of the fourth matrix calculated by the third attention unit and a matrix calculated by the first deactivation function processing unit, and outputs a fifth matrix; a sixth substitution unit provided at a subsequent stage of the fourth attention unit, which substitutes a portion of the second matrix corresponding to the changed portion with the fifth matrix and outputs a matrix indicating a new second output feature; The image processing device according to claim 1 .

5. The sparse MSA processing means a SFDLA (Sparse Frame Difference Linear Attention) processing unit is provided at a subsequent stage of the second deactivation function processing unit and the third linear processing unit, instead of the fourth substitution unit, the fifth substitution unit, and the third attention unit; The SFDLA processing unit includes: In a first frame, an A matrix is ​​calculated which is a matrix product of a part of the Q1 matrix calculated by the second deactivation function processing unit that is not updated even once after the second frame and a part of the linear matrix calculated by the third linear processing unit that is not updated even once after the second frame; In the first frame, a B1 matrix is ​​calculated which is a matrix product of a part of the Q1 matrix that is updated once in the second frame or later and a part of the linear matrix calculated by the third linear processing unit that is updated once in the second frame or later; calculating the fifth matrix in the first frame, which is a matrix sum of the A matrix and the B1 matrix; In a second frame or later after the first frame, an update token is reflected in the Q matrix calculated by the second deactivation function processing unit and the linear matrix calculated by the third linear processing unit, and a Q matrix and a K matrix are calculated in a third frame after the second frame; Also calculate a Bi matrix which is a matrix product of a part of the Qi matrix which is updated once after the second frame and a part of the Ki matrix which is updated once after the second frame; calculating the fifth matrix for the second frame and subsequent frames, which is the matrix sum of the A matrix and the Bi matrix; The image processing device according to claim 4 .

6. The image processing device extracting a change portion of a first input feature input at a first time point that has changed from a second output feature output at a second time point that is earlier than the first time point, and generating a change matrix based on the change portion of a first matrix indicating the first input feature; a linear matrix that is a matrix product of the change matrix and a weight matrix is ​​calculated, and a portion of a second matrix indicating the second output feature that corresponds to the change portion is replaced with the linear matrix, thereby performing sparse MSA (Multi-Head Self-Attention) processing; Image processing methods.

7. On the computer, extracting a change portion of a first input feature input at a first time point that has changed from a second output feature output at a second time point that is earlier than the first time point, and generating a change matrix based on the change portion of a first matrix indicating the first input feature; Calculating a linear matrix which is a matrix product of the change matrix and a weight matrix, and replacing a portion of a second matrix indicating the second output feature quantity, which corresponds to the change portion, with the linear matrix, by executing sparse MSA (Multi-Head Self-Attention) processing. program.

Images