Image recognition device and image recognition method

Abstract

This invention provides an image recognition device and image recognition method that enable efficient learning while simplifying the structure of machine learning models that have an attention mechanism. [Solution] The method includes inputting an image, dividing it into multiple patches, generating an attention map using the self-attention mechanism of a vision transformer, modifying the attention map using the self-attention mechanism, and outputting the identification result. An attention map is generated by normalizing the output values ​​of each patch of the image to 0-1 using a first activation function, a second activation function which is the inverse function of the first activation function is used to reverse the processing by the first activation function on the modified attention map, and a third activation function is used to normalize the overall output value of the attention map after processing by the second activation function to 1.

Description

[Technical Field]

[0001] This disclosure relates to an image recognition device and an image recognition method. [Background technology]

[0002] Non-patent document 1 discloses an AttentionBranchTransformer (hereinafter also referred to as "ABT"), which incorporates an Attention Branch into a Vision Transformer (hereinafter also referred to as "ViT"). The ABT is constructed by dividing the base network into a FeatureExtractor and a Perception Branch, and by providing an Attention Branch in the layer after the Feature Extractor. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Masashi Suzuki, "A Proposal for an AttentionBranchTransformer that Acquires Robustness Against Noise and Visual Explainability," [online], Doctoral Dissertation, Chubu University, 2023, [Retrieved November 28, 2024], Internet,<http: / / mprg.jp / data / FLABResearchArchive / Bachelor / B22 / Abstract / suzukim.pdf> [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] In the above ABT, an Attention Branch is provided for the attention mechanism. Thus, in the above ABT, an Attention Branch must be provided separately from the Perception Branch to construct the attention mechanism, which can complicate the structure of the machine learning model. Therefore, it is conceivable to use the self-Attention function of the Transformer that is originally present in ViT to execute the attention mechanism. In this configuration, there is no need to provide an Attention Branch in the machine learning model, so the structure of the machine learning model can be simplified.

[0005] However, in self-Attention, an activation function such as the softmax function is used to convert multiple input values ​​(each component of a vector) into an output value that is "a value between 0 and 1, with a sum of 1". In this configuration, if the attention area is modified in response to user actions on the attention map, and the sum becomes greater than 1, the activation function cannot complete the normalization process to make the sum "1". Therefore, efficient learning cannot be performed in the machine learning model.

[0006] This disclosure aims to provide an image recognition device and an image recognition method that enable efficient learning while simplifying the structure of machine learning models having an attention mechanism. [Means for solving the problem]

[0007] An image recognition device relating to one aspect of this disclosure is an image recognition device that performs image processing using a vision transformer as a machine learning model, comprising: an input unit that inputs an image and divides it into multiple patches; a generation unit that generates an attention map representing the gaze region of the input image using the self-attention mechanism of the vision transformer; a modification unit that performs modifications to the attention map according to user modification operations or a predetermined program using the self-attention mechanism; and an output unit that outputs the generated identification result when an image recognition result is generated by the vision transformer based on the modified attention map and the image, wherein the generation unit generates an attention map in which the output value of each patch of the image is normalized to 0 to 1 using a first activation function; the modification unit modifies the attention map normalized by the generation unit, performs a process that reverses the processing by the first activation function on the modified attention map using a second activation function which is the inverse function of the first activation function, and normalizes the overall output value of the attention map after processing by the second activation function to 1 using a third activation function.

[0008] An image recognition method relating to one aspect of this disclosure is an image recognition method executed by an image recognition device that performs image processing using a vision transformer as a machine learning model, and includes an input step of inputting an image and dividing it into multiple patches; a generation step of generating an attention map representing the gaze region of the input image using the self-attention mechanism of the vision transformer; a modification step of performing modifications on the attention map according to user modification operations or a predetermined program using the self-attention mechanism; and an output step of outputting the generated identification result when the vision transformer has generated an image recognition result based on the modified attention map and the image, wherein the generation step generates an attention map in which the output values ​​of each patch of the image are normalized to 0 to 1 using a first activation function; the modification step modifies the attention map normalized in the generation step, performs a process to reverse the processing by the first activation function on the modified attention map using a second activation function which is the inverse function of the first activation function, and normalizes the overall output value of the attention map after processing by the second activation function to 1 using a third activation function. [Effects of the Invention]

[0009] According to this disclosure, efficient learning is possible while simplifying the structure of machine learning models that have an attention mechanism. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a diagram showing the configuration of an image recognition device according to the first embodiment. [Figure 2] Figure 2 shows the configuration of the machine learning model. [Figure 3] Figure 3 shows the configuration of the Transformer Encoder. [Figure 4] Figure 4 shows the configuration of Multi-Head Self Attention. [Figure 5]FIG. 5 is a flowchart of the processing executed by the processing unit. [Figure 6] FIG. 6 is a flowchart of the model training process. [Figure 7] FIG. 7 is a diagram showing an example of an attention map. [Figure 8] FIG. 8 is a diagram showing an example of the Sigmoid value of a patch. [Figure 9] FIG. 9 is a diagram showing the configuration of the Multi-Head SelfAttention of the image identification device according to the second embodiment. [Figure 10] FIG. 10 is a flowchart of the model training process of the image identification device shown in FIG. 9.

MODE FOR CARRYING OUT THE INVENTION

[0011] [Description of Embodiments of the Present Disclosure] First, the contents of the embodiments of the present disclosure will be listed and described. At least a part of the embodiments described below may be arbitrarily combined.

[0012] (1) The image identification device according to one aspect of the present disclosure is an image identification device that performs image processing using a vision transformer as a machine learning model, and includes an input unit that inputs an image and divides it into a plurality of patches, a generation unit that generates an attention map representing a fixation region of the input image by a self-attention mechanism of the vision transformer, a correction unit that corrects the attention map according to a user's correction operation or a predetermined program by the self-attention mechanism, and an output unit that outputs the generated identification result when the identification result of the image is generated based on the corrected attention map and the image by the vision transformer. The generation unit generates an attention map in which the output value for each patch of the image is normalized to 0 to 1 using a first activation function. The correction unit corrects the attention map normalized by the generation unit, and performs a process of restoring the process by the first activation function in the corrected attention map using a second activation function that is the inverse function of the first activation function, and normalizes using a third activation function so that the overall output value of the attention map after the process by the second activation function becomes 1.

[0013] In the image identification device according to one aspect of the present disclosure, the generation unit generates an attention map by a self-attention mechanism of the vision transformer. The generation unit generates an attention map in which the output value for each patch of the image is normalized to 0 to 1 using a first activation function. Thereby, an attention map suitable for visualization by the user (an attention map that is easy for the user to recognize) can be generated. Therefore, in the image identification device, there is no need to separately construct an Attention Branch to generate the attention map. Therefore, in the image identification device, it is possible to simplify the structure of the machine learning model having an attention mechanism.

[0014] Furthermore, in the image recognition device, the correction unit uses a second activation function, which is the inverse function of the first activation function, to reverse the processing performed by the first activation function in the corrected attention map. Thus, even when a configuration for creating an attention map using the first activation function is added to the vision transformer in the image recognition device, the value input to the third activation function remains virtually unchanged because a second activation function is provided. Therefore, the image recognition device can complete the process of making the sum of the overall output values ​​of the attention map "1" using the third activation function (e.g., the softmax function). Consequently, the image recognition device can efficiently train its machine learning model.

[0015] (2) In the image recognition device described in (1) above, the correction unit may correct a part of the attention map in accordance with a correction operation by the user or a predetermined program. In this configuration, the manner in which the attention map is corrected can be set according to the specifications desired by the user.

[0016] (3) In the image recognition device described in (1) above, the correction unit may correct the entire attention map in accordance with the user's correction operation or a predetermined program. In this configuration, the manner of correcting the attention map can be set according to the specifications desired by the user.

[0017] (4) In any one of the image recognition devices described in (1) to (3) above, the generation unit generates multiple attention maps, compresses the generated multiple attention maps into one to generate a single attention map, and performs normalization processing on the single attention map using a first activation function. The modification unit modifies the single attention map normalized by the generation unit, uses a second activation function to reverse the processing by the first activation function on the modified single attention map, expands the single attention map into multiple attention maps, and uses a third activation function to normalize the total output value of each of the multiple attention maps to 1. In this configuration, the vision transformer performs the compression of multiple attention maps into a single attention map and the expansion of the single attention map into multiple attention maps. In this configuration, the image recognition device reverses the processing by the first activation function on the modified single attention map, expands the single attention map into multiple attention maps, and uses a third activation function to normalize the total output value of each of the multiple attention maps to 1. This allows the image recognition device, in the vision transformer, to learn by expanding a single correction instructed by the user into multiple attention maps.

[0018] (5) An image recognition method relating to one aspect of the present disclosure is an image recognition method executed by an image recognition device that performs image processing using a vision transformer as a machine learning model, and includes an input step of inputting an image and dividing it into multiple patches; a generation step of generating an attention map representing the gaze region of the input image using the self-attention mechanism of the vision transformer; a modification step of performing modifications on the attention map according to user modification operations or a predetermined program using the self-attention mechanism; and an output step of outputting the generated identification result when the vision transformer has generated an image recognition result based on the modified attention map and the image, wherein the generation step generates an attention map in which the output values ​​of each patch of the image are normalized to 0 to 1 using a first activation function; the modification step modifies the attention map normalized in the generation step, performs a process to reverse the processing by the first activation function on the modified attention map using a second activation function which is the inverse function of the first activation function, and normalizes the overall output value of the attention map after processing by the second activation function to 1 using a third activation function.

[0019] In one aspect of the image recognition method described herein, the generation step generates an attention map using the self-attention mechanism of the vision transformer. In the generation step, an attention map is generated by normalizing the output values ​​of each patch in the image to between 0 and 1 using a first activation function. This makes it possible to generate an attention map suitable for user visualization. Therefore, in the image recognition method, there is no need to separately construct an Attention Branch to generate the attention map. Consequently, the image recognition method can simplify the structure of the machine learning model having an attention mechanism.

[0020] Furthermore, in the image recognition method, the correction step uses a second activation function, which is the inverse function of the first activation function, to reverse the processing performed by the first activation function on the attention map that has been corrected. Thus, even when a configuration that creates an attention map using the first activation function is added to the vision transformer in the image recognition method, the value input to the third activation function does not change substantially because a second activation function is provided. Therefore, in the image recognition method, the process of making the sum of the total output values ​​of the attention map "1" can be completed in the third activation function (for example, the softmax function). Consequently, the image recognition method allows for efficient learning in machine learning models.

[0021] [Details of the embodiments of this disclosure] Specific examples of embodiments of this disclosure will be described below with reference to the drawings. This disclosure is not limited to these examples, and all modifications within the meaning and scope of the claims are intended to be included. In the description of the drawings, identical elements are denoted by the same reference numerals, and redundant descriptions are omitted.

[0022] <First Embodiment> Figure 1 is a configuration diagram of the image recognition device 1 according to the first embodiment. As shown in Figure 1, the image recognition device 1 comprises an operating device 2, a display device 3, a memory 4, and a processing unit 5.

[0023] The operating device 2 is a device that receives user input and outputs a signal corresponding to the received input to the processing unit 5. The operating device 2 may be, for example, a mouse, keyboard, touch panel, etc. The display device 3 is a device that displays images to the user.

[0024] Memory 4 includes RAM, a rewritable volatile storage medium; ROM, a non-rewritable non-volatile storage medium; and flash memory, a rewritable non-volatile storage medium. RAM, ROM, and flash memory are non-transitional tangible storage media. Flash memory is pre-recorded with data from a trained machine learning model M.

[0025] The processing unit 5 executes a program (not shown) stored in ROM or flash memory, and uses RAM as a working area during execution to perform various processes described later.

[0026] Next, we will explain the machine learning model M. Figure 2 shows the configuration of the machine learning model M. The machine learning model M is a Vision Transformer (ViT) (hereinafter also referred to as "ViT"). The machine learning model M is a model that identifies images. By identifying images, the machine learning model M can predict the presence or absence of defects (abnormalities) in the inspection of items (e.g., harnesses) (e.g., the presence or absence of tape on a harness).

[0027] As shown in Figure 2, the machine learning model M is constructed including a feature extraction unit (input unit) 10, an attention unit (generation unit, correction unit) 12, and a discrimination unit 14.

[0028] The feature extraction unit 10 may function as a Feature Extractor in ViT. The feature extraction unit 10 may have the functions of Linear Projection of Flattened Patches, Position Embedding, and Transformer Encoder. The Transformer Encoder is constructed including Multi-Head Self Attention and FFN (Feed Forward Network).

[0029] The feature extraction unit 10 takes an image G as input and divides the image G into patches (regions). The image G can be a 2D or 3D image. The feature extraction unit 10 performs a process to flatten each patch into one dimension (e.g., linear projection) and converts each patch into a token, which is a sequence of vectors. This output is also called patch embedding.

[0030] The feature extraction unit 10 adds a class token to the beginning of each token. The class token is used as input for class identification after training. The feature extraction unit 10 adds a numerical value of position information to the token and the class token (position embedding). The feature extraction unit 10 outputs the token value as an output value to the attention unit 12 and the identification unit 14.

[0031] The attention unit 12 may function as the Perception Branch in ViT. The attention unit 12 generates an attention map AM representing the gaze region of the input image G based on the output values ​​output from the feature extraction unit 10. The attention unit 12 generates an attention map AM by normalizing the output values ​​of each patch of image G to between 0 and 1 using the first activation function (Sigmoid38, described later).

[0032] The attention unit 12 modifies the normalized attention map AM in response to user modification operations. The attention unit 12 modifies a portion of the attention map AM in response to user modification operations. The attention unit 12 modifies the entire attention map AM in response to user modification operations. The target of modification (partial or whole) of the attention map AM in the attention unit 12 can be set by the user.

[0033] The attention unit 12 is the second activation function, which is the inverse function of the first activation function (Sigmoid, described later). -1The attention map AM, which has undergone modification, is processed using 40). The attention unit 12 uses a third activation function (Softmax 42, described later) to normalize the attention map after processing by the second activation function so that the overall output value of the attention map becomes 1.

[0034] The identification unit 14 may function as a Perception Branch. The identification unit 14 identifies image G based on the modified attention map AM and image G. The identification unit 14 extracts class tokens from the features and sets the number of dimensions to the number of classes to be identified. The identification unit 14 outputs the probability of each class. The identification unit 14 outputs the class with the highest probability as the identification result of ViT.

[0035] As shown in Figure 3, the attention unit 12 and the identification unit 14 may be configured to include a Transformer Encoder 20. The Transformer Encoder 20 is constructed to include a Multi-Head SelfAttention 22 and an FFN (Feed Forward Network) 24. The Transformer Encoder 20 alternately repeats the processing of the Multi-Head SelfAttention 22 and the FFN 24 a predetermined number of times. The FFN 24 may include a Layer Norm and an MLP (Multi-Layer Perceptron).

[0036] Figure 4 shows the configuration of the Multi-Head Self Attention 22. As shown in Figure 4, the Multi-Head Self Attention 22 consists of Layer Norm 30, Linear 32, Linear 34, Linear 36, Sigmoid (first activation function) 38, and Sigmoid -1 It is constructed including (second activation function) 40 and Softmax (third activation function) 42.

[0037] Multi-Head Self Attention22 has multiple heads, and each head performs attention. Multi-Head Self Attention22 divides the input patch features horizontally (in the number of dimensions) according to the number of heads. The divided patch features are distributed to each head, and self attention is calculated. Multi-Head Self Attention22 performs a scaling operation by dividing the dot product by the square root of the number of dimensions.

[0038] Multi-Head Self Attention22 calculates the inner product of feature Q, which is the output of Linear32, and feature K, which is the transposed output of Linear34. After the inner product calculation, Multi-Head Self Attention22 performs Sigmoid38 and Sigmoid -1 Softmax42 is applied to normalize the output values ​​of 40 to [0,1]. In Multi-Head Self Attention22, these processed values ​​are multiplied by the feature quantity V, which is the output of Linear36, to obtain the output.

[0039] Layer Norm30 receives the values ​​output from the feature extraction unit 10 as input. Layer Norm30 transforms the input values ​​so that their mean is "0" and their variance is "1". Layer Norm30 outputs the transformed values ​​to Linear32, Linear34, and Linear36.

[0040] Linear32, Linear34, and Linear36 each have different weights. The output value of Linear32 is Q (Query). The output value of Linear34 is K (Key). The output value of Linear36 is V (Value).

[0041] Sigmoid38 is a function that converts an input value into a numerical value in the range of 0 to 1 and outputs it. Sigmoid38 takes the dot product of the output value of Linear32 (Q) and the output value obtained by transposing the output of Linear34 (K) as input. Based on the input value, Sigmoid38 generates an attention map AM. Sigmoid38 generates an attention map AM that normalizes the output values ​​of each patch of image G to 0 to 1. Sigmoid38 outputs the generated attention map AM.

[0042] Sigmoid -1 40 is the inverse function of Sigmoid 38. -1 40 processes the attention map AM output from Sigmoid38. -1 40 applies a process to the attention map AM, which has been modified by the user, to revert the processing performed by Sigmoid38, and outputs it to Softmax42.

[0043] Softmax42 is a function that transforms each input value so that the sum of the outputs equals 1. Softmax42 is a Sigmoid function. -1 The value output from 40 is converted and output to FFN24.

[0044] The functions of the feature extraction unit 10, the attention unit 12, and the discrimination unit 14 are actually performed by the processing unit 5, which processes according to the structure and parameters of the machine learning model M. The feature extraction unit 10, the attention unit 12, and the discrimination unit 14 are trained by unsupervised learning so that they can perform the functions described above.

[0045] Next, we will explain the image recognition process (image recognition method) of the processing unit 5 using the machine learning model M configured in this way.

[0046] Figure 5 is a flowchart of the process executed by the processing unit 5. When predetermined conditions are met, such as a user performing an execution start operation on the operating device 2, the processing unit 5 starts the process shown in Figure 5, which is defined in a predetermined program stored in memory 4. In this process, the processing unit 5 first reads the machine learning model M from memory 4 (step S01).

[0047] Next, the processing unit 5 acquires image G and functions as an input unit that inputs image G to the machine learning model M (step S02). Image G may be an image selected from among multiple images pre-recorded in memory 4 by an operation on the user's operating device 2, or it may be an image received from another device via a communication network (not shown).

[0048] Next, when image G is input to machine learning model M, as described above, the feature extraction unit 10 in machine learning model M extracts features from image G, and the attention unit 12 generates an attention map AM from the features.

[0049] The processing unit 5 acquires the attention map AM generated in this manner (step S03). That is, the processing unit 5 copies or moves the attention map AM generated in memory 4 by the machine learning model M to another area in memory 4.

[0050] Next, the processing unit 5 functions as a map modification unit that modifies the acquired (i.e., copy destination or move destination) attention map AM based on the user's modification operation on the operating device 2 (step S04). In this way, the attention map AM is modified according to the user's knowledge.

[0051] Next, the processing unit 5 functions as an identification unit 14 based on the attention map AM modified in the previous step S04 (step S05). The identification unit 14 generates an identification result based on the attention map AM.

[0052] The processing unit 5 functions as an output unit that acquires and outputs the identification result generated by the identification unit 14 (step S06). The output destination may be another device via a communication network (not shown), the memory 4, or the display device 3.

[0053] In this way, by modifying the attention map AM using user insights, the identification unit 14 gives higher weight to the areas intended by the user. As a result, image recognition can be performed in accordance with the user's intentions. In other words, the identification results can be adjusted by using an attention map that has been manually modified based on user insights.

[0054] Figure 6 is a flowchart of the model training process performed when the attention map AM is modified. The training process of the machine learning model M when the attention map AM is modified will be explained with reference to Figure 6.

[0055] In the model training process, an image is first input to the machine learning model M (step S11). This image may be the image G mentioned above, or it may be another training image. Based on this, the feature extraction unit 10 extracts the features of the image, and the attention unit 12 generates an attention map AM.

[0056] Next, the processing unit 5 determines whether or not the user has made a modification operation (modification instruction) of the attention map AM (step S12). If a modification instruction has been made, the processing unit 5 regenerates the attention map AM according to the modification instruction from the user (step S13).

[0057] If a modification instruction is received, the processing unit 5 modifies part or all of the attention map AM. When the processing unit 5 modifies part of the attention map AM, for example, it generates an attention map AM in which only the attention areas related to class tokens have been modified.

[0058] When the processing unit 5 modifies the entire attention map AM, it processes it as follows. Figure 7 shows an example of an attention map AM. As shown in Figure 7, for example, in an attention map AM based on an image containing three dogs, let's explain the case where one of the three dogs (for example, a Chihuahua) is designated as the target (area of ​​fixation). In this case, as shown in Figure 8, the processing unit 5 increases the value (Sigmoid38 value) of the patch related to the target (the part shown in black in Figure 8) (for example, 0.8), decreases the values ​​of the patches other than the target (the part shown in gray in Figure 8) (for example, 0.1), and generates the attention map AM. The values ​​of the patches other than the target (values ​​other than the value specified by the user) can be freely set in the machine learning model M.

[0059] Furthermore, the processing unit 5 increases the values ​​of patches related to the target, decreases the values ​​of patches related to dogs other than the target, and does not need to modify the values ​​of patches related to the background.

[0060] Returning to Figure 6, if no correction instructions were given in step S12, or if the regeneration of the attention map AM was completed in step S13, it is determined whether all correction processes have been completed (step S14). If they have been completed, retraining is performed using the training data (step S15). The training data here includes the regenerated attention map AM. Once retraining is complete (step S16), the model training process is completed.

[0061] [effect] As described above, in the image recognition device 1 according to the present embodiment, the attention unit 12 generates an attention map AM by means of the self-attention mechanism of the vision transformer. The attention unit 12 uses Sigmoid38 to generate an attention map AM in which the output value for each patch of the image G is normalized to 0 to 1. Thereby, an attention map AM suitable for visualization by the user can be generated. Therefore, in the image recognition device 1, there is no need to separately construct an Attention Branch in order to generate the attention map AM. Therefore, in the image recognition device 1, it is possible to simplify the structure of the machine learning model having an attention mechanism.

[0062] Also, in the image recognition device 1, the attention unit 12 processes the attention map AM for which a correction operation has been performed, using Sigmoid40 which is the inverse function of Sigmoid38. Thus, in the image recognition device 1, even when a configuration for creating the attention map AM using Sigmoid38 is added to the vision transformer, since Sigmoid40 is provided, the value input to Softmax42 does not substantially change. Therefore, in the image recognition device 1, in Softmax42, it is possible to complete the process of making the sum of the overall output values of the attention map AM equal to "1". Therefore, in the image recognition device 1, efficient learning can be performed in the machine learning model M.

[0063] In the image recognition device 1, by providing Sigmoid38 and Sigmoid40 in the vision transformer, the attention mechanism is executed in the basic configuration of the vision transformer. Therefore, in the image recognition device 1, it can be applied to an existing machine learning model.

[0064] ​​​​​​In the image recognition device 1 according to this embodiment, the attention unit 12 may modify a part of the attention map AM in response to a user's modification operation. Alternatively, the attention unit 12 may modify the entire attention map AM in response to a user's modification operation. In this way, the image recognition device 1, with this configuration, can set the manner of modification of the attention map AM according to the specifications desired by the user.

[0065] <Second Embodiment> Next, a second embodiment will be described. Figure 9 is a diagram showing the configuration of the Multi-Head Self Attention 22A of the image recognition device 1A according to the second embodiment. As shown in Figure 9, the Multi-Head SelfAttention 22A consists of a Layer Norm 30, a Linear 32, a Linear 34, a Linear 36, a Sigmoid 38, and a Sigmoid -1 It is constructed including 40, Softmax 42, a compression section 44, and a decompression section 46.

[0066] The compression unit 44 receives the dot product of the output value (Q) of Linear32 and the output value (K) obtained by transposing the output of Linear34 as input. Based on the input value, the compression unit 44 generates multiple attention maps AM. Specifically, the compression unit 44 generates attention maps AM equal to the number of heads. In this embodiment, for example, the compression unit 44 generates six attention maps AM. The compression unit 44 compresses the generated multiple attention maps AM to generate a single attention map SAM. The compression unit 44 outputs the generated single attention map SAM.

[0067] Sigmoid38 receives the single attention map SAM output from the compression unit 44 as input. Sigmoid38 normalizes the input single attention map SAM. Sigmoid38 outputs the single attention map SAM.

[0068] Sigmoid -140 is the inverse function of Sigmoid 38. -1 40 processes the single attention map SAM output from Sigmoid38. -1 40 applies a process to the single attention map SAM that has been modified by the user, reverting the processing performed by Sigmoid38, and outputs it to the expansion unit 46.

[0069] The expansion unit 46 expands (decomposes) a single attention map SAM that has been modified by the user into multiple attention maps AM. For example, the expansion unit 46 expands a single attention map SAM into six attention maps AM. The expansion unit 46 outputs multiple attention maps AM.

[0070] Softmax42 is a function that transforms each input value so that the sum of the outputs equals 1. Softmax42 transforms each of the multiple attention maps AM output from the expansion unit 46 and outputs them to FFN24.

[0071] Figure 10 is a flowchart of the model training process performed when the attention map AM is modified. The training process of the machine learning model M when the attention map AM is modified will be explained with reference to Figure 10.

[0072] In the model training process, an image is first input to the machine learning model M (step S21). This image may be the image G mentioned above, or it may be another training image. The feature extraction unit 10 then extracts the features of the image, and the attention unit 12 generates a single attention map SAM by compressing multiple attention maps AM.

[0073] Next, the processing unit 5 determines whether or not the user has made a modification operation (modification instruction) for the single attention map SAM (step S22). If a modification instruction has been made, the processing unit 5 regenerates the single attention map SAM according to the modification instruction from the user (step S23).

[0074] If a modification instruction is received, the processing unit 5 modifies part or all of the single attention map SAM. When the processing unit 5 modifies part of the single attention map SAM, for example, it generates a single attention map SAM in which only the attention areas related to class tokens have been modified.

[0075] If no correction instructions are given in step S22, or if the regeneration of the single attention map SAM is completed in step S23, it is determined whether all correction processes are complete (step S24). If they are complete, the single attention map SAM is expanded into multiple attention maps AM (step S25). Then, retraining is performed using training data (step S26). The training data here includes the multiple attention maps AM after regeneration. Once retraining is complete (step S27), the model training process is completed.

[0076] [effect] As described above, in the image recognition device 1A according to this embodiment, the vision transformer compresses multiple attention maps AM into a single attention map SAM, and expands the single attention map SAM into multiple attention maps AM. In this configuration, the image recognition device 1A reverses the processing by Sigmoid38 on the modified single attention map SAM, expands the single attention map SAM into multiple attention maps AM, and normalizes it using Softmax42 so that the total output value of each of the multiple attention maps AM becomes 1. As a result, the image recognition device 1A can learn a single modification instructed by the user by expanding it into multiple attention maps AM in the vision transformer.

[0077] While embodiments of this disclosure have been described above, this disclosure is not necessarily limited to the embodiments described above, and various modifications are possible without departing from its essence.

[0078] In the above embodiment, a configuration using Sigmoid38 as the first activation function was described as an example. However, the first activation function may be any other function. In this case, the second activation function can be the inverse function of the first activation function.

[0079] In the above embodiment, a configuration using Softmax42 as the third activation function was described as an example. However, the third activation function may be any other function.

[0080] In the above embodiment, the modification unit was described as modifying the attention map AM in response to a user's modification operation. However, the modification unit may also modify the attention map AM according to a predetermined program, for example.

[0081] The various embodiments and modifications described above may be combined as appropriate without departing from the spirit of this disclosure. [Explanation of Symbols]

[0082] 1,1A…Image recognition device 2...Operating device 3...Display device 4…Memory 5… Processing Unit 10…Feature extraction unit (input unit) 12... Attention section (generation section, correction section) 14…Identification section 20…Transformer Encoder 22,22A…Multi-Head Self Attention 24…FFN 30…Layer Norm 32…Linear 34…Linear 36…Linear 38…Sigmoid (first activation function) 40…Sigmoid -1 (Second activation function) 42...Softmax (third activation function) 44... Compression section 46... Expansion section AM... Attention Map G...Image K...Features M...Machine learning model Q...Features SAM…Single Attention Map V...Features

Claims

1. An image recognition device that performs image processing using a vision transformer as a machine learning model, An input section that takes an image as input and divides it into multiple patches, The self-attention mechanism of the vision transformer generates an attention map that represents the gaze region of the input image, The self-attention mechanism includes a modification unit that performs user modification operations or modifications according to a predetermined program on the attention map, The Vision Transformer includes an output unit that outputs the generated identification result when the Vision Transformer generates an identification result of the image based on the corrected attention map and the image, The generation unit generates the attention map by normalizing the output values ​​of each patch in the image to between 0 and 1 using a first activation function. The aforementioned modification section is, The attention map normalized by the generation unit is modified, Using a second activation function, which is the inverse function of the first activation function, the modified attention map is processed to reverse the action performed by the first activation function. An image recognition device that uses a third activation function to normalize the overall output value of the attention map after processing by the second activation function so that the total output value becomes 1.

2. The image recognition device according to claim 1, wherein the modification unit modifies a portion of the attention map in accordance with the user's modification operation or a predetermined program.

3. The image recognition device according to claim 1, wherein the modification unit modifies the entire attention map in accordance with the user's modification operation or a predetermined program.

4. The generation unit generates multiple attention maps, compresses the generated multiple attention maps into one to generate a single attention map, and performs the normalization process using the first activation function on the single attention map. The aforementioned modification section is, The single attention map normalized by the generation unit is modified, Using the second activation function, the single attention map that has been modified is processed to reverse the processing performed by the first activation function, and the single attention map is expanded into multiple attention maps. The image recognition device according to claim 1 or 2, wherein the third activation function is used to normalize the total output value of each of the multiple attention maps so that it becomes 1.

5. An image recognition method performed by an image recognition device that processes images using a vision transformer as a machine learning model, An input step that takes an image as input and splits it into multiple patches, The self-attention mechanism of the vision transformer generates an attention map that represents the gaze region of the input image, The self-attention mechanism performs a modification step in which the user performs a modification operation on the attention map or makes modifications according to a predetermined program. The Vision Transformer includes an output step of outputting the generated identification result when the Vision Transformer generates an identification result of the image based on the modified attention map and the image, In the generation step, the attention map is generated by normalizing the output values ​​of each patch in the image to 0 to 1 using a first activation function. In the aforementioned correction step, In the generation step, the normalized attention map is modified, Using a second activation function, which is the inverse function of the first activation function, the modified attention map is processed to reverse the action performed by the first activation function. An image recognition method that uses a third activation function to normalize the overall output value of the attention map after processing by the second activation function so that the total output value becomes 1.

Images