A visual analysis method and device based on Transformer self-attention

Abstract

The application discloses a visual analysis method and device based on a Transformer self-attention. The application can understand the overall distribution and statistical rules of the training layer and the self-attention of the attention head of a deep learning model through a visual analysis chart, and can view the connection of the self-attention in an instance through a data link chart and a matrix chart. The attention visual analysis in the field of computer vision reveals the mutual attention between pixel blocks in a training task, the self-attention distribution between different layers and heads is viewed through global normalization and local normalization, and the process of obtaining a result of a downstream task is obtained. Through the application, researchers can directly observe the value distribution of the attention head in the Transformer model by using statistical analysis charts, and select an interesting attention head. And through the specific single attention head visualization, researchers can analyze the role of the attention head in a specific task, and help researchers to improve and optimize the model.

Description

Technical Field

[0001] This invention relates to the field of computer data visualization, and more specifically to a visual analysis method and apparatus based on Transformer self-attention. Background Technology

[0002] In recent years, the rise of Transformer-based models has brought significant performance improvements to many natural language processing (NLP) and computer vision tasks. In particular, BERT models in NLP and ViT models in computer vision have achieved state-of-the-art results on multiple tasks. In NLP, Transformer-based models pre-trained on large-scale corpora can be effectively fine-tuned for various downstream tasks, such as sentiment analysis, question answering, and text summarization. In computer vision, Transformers are used for tasks such as image classification, object detection, semantic segmentation, and video understanding. Due to their superior performance, Visual Transformer-based models have become a mainstream research direction for handling visual tasks. However, understanding what these models have learned and why they succeed and fail is crucial for researchers to develop better models and for decision-makers to trust these models—a serious challenge.

[0003] The development of interactive visualization and visual analytics technologies has brought researchers new methods to study the working mechanism of models. By analyzing the data generated by model training through various visual analytics charts, users can discover some performance patterns within the data and explore and analyze them. Through the linkage analysis between charts, users can gain a deeper understanding of the internal principles of complex deep learning models.

[0004] In summary, visual analytics is currently a feasible approach to help researchers better understand the working principle of the internal self-attention mechanism of Transformer-based models and why they produce successful and unsuccessful predictions. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a visual analysis method and apparatus based on Transformer self-attention. To help researchers better understand the internal self-attention mechanism of Transformer models, this invention designs various visual analysis methods for text Transformer and image Transformer models, revealing the internal connections of self-attention, statistically summarizing change patterns, and deepening researchers' understanding of the model's interpretability.

[0006] In a first aspect, the present invention provides a visual analytics method based on Transformer self-attention, the method comprising the following steps:

[0007] (1) Model training: Users train their own models;

[0008] (2) Data acquisition: Users use the provided API interface to save the self-attention data and original input data generated by the model during training;

[0009] (3) Log writing: Using the model training data saved by the user, rewrite it into a log data format that the system can parse, and feed it into the visual analysis system to obtain visual analysis results;

[0010] (4) Results analysis: Users view the visual analysis results obtained by the visual analysis system, and through multi-dimensional visual analysis charts and interactive linkage with the charts, they gradually analyze the model results and explore the self-attention mechanism inside the model.

[0011] Furthermore, the visual analysis results obtained in the visual analysis system in step (3) are as follows:

[0012] Visual analytics systems analyze data in two ways: attention-based visual analytics in the field of natural language processing and attention-based visual analytics in the field of computer vision.

[0013] The attention visual analysis in the field of natural language processing includes three major components: statistical information tables, statistical information graphs, and attention visualization. Through visual analysis charts, we can understand the overall distribution and statistical patterns of self-attention in the training layers and attention heads of deep learning models. At the same time, through data link graphs and matrix graphs, we can see the connection between self-attention in specific training samples.

[0014] The attention visual analysis in the field of computer vision is used to represent the mutual attention between pixel blocks in the training task. It examines the attention distribution between different layers and heads through two methods: global normalization and local normalization, and obtains data on how downstream tasks arrive at the corresponding results.

[0015] Furthermore, the attention visual analysis in the natural language processing domain specifically includes:

[0016] (2.1) The overall self-attention status of all training layers and attention heads of the model is displayed in the form of statistical information tables. Users can sort and view the data by different statistical indicators such as maximum value (max), minimum value (min), interquartile range (quar), and variance (variance).

[0017] (2.2) At the same time, the self-attention information is displayed in the form of statistical information charts. The self-attention information of each training layer and each attention head is arranged in the form of a sunburst chart. The starting position of the bar chart encodes the maximum and minimum self-attention features of the attention head, and the color encodes the variance features. When the mouse hovers over the corresponding attention head, the specific information of each attention head can be displayed.

[0018] (2.3) At the same time, the Sunrise Chart supports multiple filtering operations and allows users to choose different filtering criteria. Users can discover attention heads that are of interest or have abnormal self-attention and explore the reasons in the visual analysis chart of self-attention of a single attention head. After the user selects a single attention head, two visual analysis charts are provided to show the self-attention distribution of the input character within each head.

[0019] Furthermore, the self-attention calculation process for a single attention head in step (2.3) is as follows:

[0020] (2.3.1) Encode the input text into a vector form, X = [X1, X2, ..., X... N ],X∈R N×D N is the number of input characters, and D is the vector dimension;

[0021] (2.3.2) Position encoding information, encoded as t = [t1, t2, ..., t N ], t∈R N×D X and t are added together, and the encoded vector X carrying the location information is fed into the model's encoding layer;

[0022] (2.3.3) In the model encoding layer, through linear transformation, the query vector sequence Q, the key vector sequence K, and the value vector sequence V are obtained:

[0023] Q = W Q X, K = W K X, V = W V X;

[0024] Among them, W Q W K W V This is the corresponding linear transformation matrix;

[0025] (2.3.4) In self-attention models, the scaled dot product is usually used as the attention scoring function, and the output self-attention is as follows:

[0026]

[0027] Output self-attention vector Z = [Z1, Z2, ..., Z N ], Z i ∈R 1×N, i∈[1,N], Z i Each term represents the self-attention magnitude of the i-th character with respect to other characters. This value is input into the color mapping function colorProject to obtain the color mapping score matrix C, which is used to draw the data link graph and matrix graph.

[0028] Furthermore, the steps for drawing the data link diagram are as follows:

[0029] (5.1) Input attention data Z and calculate the length Len;

[0030] (5.2) Calculate the size of the data link graph, with a width of TextBoxWidth*2+AttentionWidth and a height of TextBoxHeight*Len;

[0031] (5.3) Using the attention matrix Z, determine the corresponding quantitative relationship between each character and calculate the connection offset;

[0032] (5.4) Using the color mapping fraction matrix C and the corresponding positional relationship, determine the color of the corresponding line;

[0033] (5.5) Draw a data link diagram.

[0034] Furthermore, the steps for drawing the matrix diagram are as follows:

[0035] (6.1) Input attention data Z and calculate the length Len;

[0036] (6.2) Calculate the size of the matrix diagram, with a width of MatrixBox * Len + TextBoxWidth and a height of MatrixBox.

[0037] *Len+TextBoxHeight;

[0038] (6.3) Generate a matrix grid diagram;

[0039] (6.4) Use the color fraction matrix C to color the matrix grid.

[0040] Furthermore, the attention visual analysis in the field of computer vision specifically involves: for attention map data attn_map, the dimensions are (L, num_heads, h, w, h, w), where L is the number of model layers, num_heads is the number of attention heads, h is the height of the attention map, and w is the width of the attention map; h and w correspond to the x and y values ​​of the visual image patch clicked by the user, that is, when the user clicks on an image patch, the algorithm will request to process L×num_heads attention maps;

[0041] Users can choose between global or local normalization based on their needs. First, the user selects the image to be visually analyzed, then chooses the normalization method, and clicks on an image patch. The system requests the attention map data corresponding to the patch's coordinates (x, y) and performs normalization processing. After normalization, the attention map is transformed into a heatmap using the JET algorithm, with areas showing higher values ​​highlighted in red for emphasis. When global normalization is selected, users can slide the scale 'r' to view the attention values ​​more clearly. After multiplying the heatmap by the scale 'r', if any value exceeds 255, it will be automatically set to 255 due to a pre-set threshold. Finally, the heatmap is overlaid on the original image to demonstrate the effect of the attention map on the original image.

[0042] Furthermore, the user can choose between a global normalization method or a local normalization method according to their needs, specifically:

[0043] (8.1) When the user selects the local normalization method, the algorithm will perform normalization operation in each attention map; that is, it will perform min-max normalization in L×num_heads attention maps respectively, so that the value of the image is 0-255; first, we obtain a temporary matrix temp=A-min(A) by processing attention map A, and then we obtain the normalized image A^' by A^'=temp / max(temp)×255;

[0044] (8.2) When the user selects the global normalization method, the algorithm will visualize the entire range of attention maps corresponding to the clicked image patch in the visual analysis image; the magnitude of the attention values ​​of certain layers and certain heads can be observed in this way; during global normalization, for the L×num_heads attention map matrix A corresponding to the clicked image patch, a temporary matrix temp=A-min(A) is obtained after processing, and then the normalized L×num_heads images A^' are obtained after processing by A^'=temp / max(temp)×255.

[0045] A second aspect of the present invention: a visual analysis device based on Transformer self-attention, the device comprising the following modules:

[0046] Model training module: Users train their own models;

[0047] Data acquisition module: Users can use the provided API interface to save the self-attention data and raw input data generated by the model during training;

[0048] Log writing module: It uses the model training data saved by the user, rewrites it into a log data format that the system can parse, and feeds it into the visual analysis system to obtain visual analysis results;

[0049] Results Analysis Module: Users can view the visual analysis results obtained by the visual analysis system, analyze the model results step by step through multi-dimensional visual analysis charts and interactive links with the charts, and explore the self-attention mechanism inside the model.

[0050] A third aspect of the present invention: a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement steps in a visual analysis method based on Transformer self-attention.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] By analyzing the self-attention mechanism through various visual analytics techniques, we can help users understand the flow of self-attention in the encoding layer, analyze the inherent distribution pattern of self-attention, and help users find the real reason why the model produces the training results. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0054] Figure 1 This is an architecture diagram of the visual analysis method based on Transformer self-attention in this invention;

[0055] Figure 2 This is a flowchart of the visual analysis method based on Transformer self-attention in this invention;

[0056] Figure 3 This is a statistical graph showing the overall self-attention of the model in this invention;

[0057] Figure 4 The bar chart for the rising sun diagram of the present invention;

[0058] Figure 5 This is a bar chart of the filtered sunrise pattern of the present invention;

[0059] Figure 6 These are two visual analysis diagrams of the present invention;

[0060] Figure 7 This is a self-attention map of each training layer and each attention head in this invention;

[0061] Figure 8 These are two normalized display self-attention maps of the present invention;

[0062] Figure 9This is the attention distribution map of each attention head under the local normalization mode of the present invention.

[0063] Figure 10 This is a schematic diagram of the device process of the present invention. Detailed Implementation

[0064] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0065] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0066] To provide a more detailed description of the present invention, specific examples and accompanying drawings are provided below.

[0067] The core of this invention is to provide a visual analysis method based on Transformer self-attention. This includes attention visual analysis in the fields of natural language processing and computer vision. The attention visual analysis in the natural language processing field mainly comprises three components: statistical information tables, statistical information graphs, and attention visualization. Based on the TianShu AI open-source platform of Zhejiang Lab, it enables visual analysis of the self-attention mechanism. Through interactive visualization, it helps users understand the internal mechanisms of the model, observe the layer-by-layer flow and changes of self-attention, and improve users' understanding of the model's interpretability.

[0068] To provide a more detailed description of the present invention, specific examples and accompanying drawings are provided below.

[0069] This method includes visual analysis of self-attention in both the natural language processing (NLP) and computer vision domains. The NLP analysis reveals the overall distribution and statistical patterns of self-attention in the encoding layers and attention heads of deep learning models, and allows observation of the relationships between self-attention instances through specific training samples. The computer vision analysis reveals the mutual attention between pixel blocks during training tasks, examines the distribution of self-attention across different encoding layers and attention heads, and analyzes how downstream tasks make result decisions.

[0070] Furthermore, the attention visualization analysis in the natural language processing field first displays the overall self-attention status of all encoding layers and attention heads in a statistical information table format. Users can sort and view the data using different statistical indicators such as maximum (max), minimum (min), quartiles (quar), and variance (variance). Since the table format may not be intuitive enough, a statistical information graph is also provided to display self-attention information. The self-attention values ​​of each encoding layer and attention head are arranged in a sunburst chart format. The starting position of the bar chart encodes the maximum and minimum self-attention values ​​of the attention head, and the color encodes the variance characteristics. Hovering the mouse over the corresponding attention head displays specific information for each attention head. Simultaneously, the sunburst chart supports multiple filtering operations. By selecting different filtering criteria, users can discover attention heads of interest or those with abnormal self-attention, and explore the reasons within the individual attention head self-attention visualization analysis chart.

[0071] refer to Figure 1 , Figure 1 This is an architecture diagram of the visual analytics method based on Transformer self-attention in this invention. The architecture diagram includes two parts:

[0072] The first part is the visual analysis of self-attention in the field of natural language processing, namely text self-attention visualization, which includes statistical information tables, statistical information graphs, line graphs and matrix graphs; the second part is the visual analysis of self-attention in the field of computer vision, namely image self-attention visualization, which includes global normalization visualization and local normalization visualization.

[0073] refer to Figure 2 , Figure 2 This is a flowchart of the visual analysis method based on Transformer self-attention in this invention. The flowchart includes:

[0074] (1) Model training: Users train their own models.

[0075] (2) Data acquisition: Users use the provided API interface to save the self-attention data and original input data generated by the model during training.

[0076] (3) Log writing: Using the model training data saved by the user, rewrite it into a log data format that the system can parse.

[0077] (4) Results Analysis: Users can utilize the front-end visual analysis interface to perform text self-attention visualization and image self-attention visualization for different tasks. Through multi-dimensional visual analysis charts and interactive links between the charts, users can gradually analyze the model results, perform individual attention head effect analysis, and help users improve and optimize the model.

[0078] refer to Figure 3 , Figure 3 This demonstrates the text self-attention analysis based on the Transformer self-attention visual analysis method of this invention; the operation steps are as follows:

[0079] (2.1) Users can understand the overall self-attention statistics of the model through statistical information charts, for reference. Figure 3 The selected attention points are highlighted, and users can click on any row to switch attention points, which will change the overall display accordingly. Users can sort all attention points in ascending or descending order based on the maximum value (max), minimum value (min), interquartile range (quar), and variance (variance).

[0080] (2.2) Users can understand statistical information in a more intuitive way through statistical infographics, for reference. Figure 4 In the sunburst chart, the bars represent a single attention head. The starting position encodes the minimum and maximum values, and the color encodes the variance. Users can hover the mouse over the bars to view the corresponding information, and click on the bars to switch attention heads.

[0081] (2.3) The sunrise chart provides a filtering function. Users can select their preferred focus points based on four filtering criteria. Figure 5 .

[0082] (2.4) After selecting an attention head, two visual analysis charts can be viewed in the attention visualization area to explore the attention of a single attention head. (Reference) Figure 6 Users can change the selected attention head by choosing the Layer and Head. For data linking graphs, users can view the attention distribution of a single character to other characters by sliding the character wheel; for matrix graphs, users can visually view the distribution of all attention. Color indicates the magnitude of the attention value.

[0083] refer to Figure 7 , Figure 7 This demonstrates the image self-attention analysis method based on Transformer self-attention in this invention. The operation steps are as follows:

[0084] refer to Figure 7 Users can select images and training layers in the control panel on the right, while the left side will display the self-attention maps of each training layer and each attention head in the selected area.

[0085] Users can choose between global normalization and local normalization to display the self-attention map. (See reference) Figure 8 In global normalization mode, users can choose to zoom in to adjust the overall attention display.

[0086] In local normalization mode, users will have a more intuitive view of the attention distribution of each attention head, and can refer to... Figure 9 .

[0087] Furthermore, after the user selects a single attention head, the invention provides two visual analysis charts to display the self-attention distribution within each head with respect to the input character. The self-attention calculation process in each attention head is as follows:

[0088] a. Encode the input text into a vector form, X = [X1, X2, ..., X...]. N ],X∈R N×D N is the number of input characters, and D is the vector dimension.

[0089] b. Position encoding information, encoded as t = [t1, t2, ..., t N ], t∈R N×D X and t are added together, and the encoded vector X carrying the location information is fed into the model's encoding layer.

[0090] c. In the model encoding layer, through linear transformation, the query vector sequence Q, the key vector sequence K, and the value vector sequence V are obtained.

[0091] (W Q W K W V (The corresponding linear transformation matrix):

[0092] Q = W Q X, K = W K X, V = W V X

[0093] d. In self-attention models, scaled dot product is typically used as the attention scoring function, and the output self-attention is as follows:

[0094]

[0095] Output self-attention vector Z = [Z1, Z2, ..., Z N ], Z i ∈R 1×N , i∈[1,N], Z i Each term represents the self-attention magnitude of the i-th character with respect to other characters. This value is input into the color mapping function colorProject to obtain the color mapping score matrix C, which is used to draw the data link graph and matrix graph.

[0096] The steps for drawing a data link diagram are as follows:

[0097] a. Input attention data Z, calculate length Len

[0098] b. Calculate the size of the data link graph, with a width of TextBoxWidth * 2 + AttentionWidth and a height of...

[0099] TextBoxHeight*Len

[0100] c. Using the attention matrix Z, determine the corresponding quantity relationship between each character and calculate the connection offset.

[0101] d. Using the color mapping fraction matrix C and the corresponding positional relationships, determine the colors of the corresponding connecting lines.

[0102] e. Draw a data link diagram

[0103] The steps for drawing the matrix diagram are as follows:

[0104] a. Input attention data Z, calculate length Len

[0105] b. Calculate the size of the matrix diagram, with a width of MatrixBox * Len + TextBoxWidth and a height of MatrixBox * Len.

[0106] +TextBoxHeight

[0107] c. Generate a matrix grid diagram

[0108] d. Use the color fraction matrix C to color the matrix grid.

[0109] Furthermore, in the aforementioned visual attention analysis in the field of computer vision, the attention map data attn_map has dimensions (L, num_heads, h, w, h, w), where L is the number of model layers, num_heads is the number of attention heads, h is the height of the attention map, and w is the width of the attention map. h and w correspond to the x and y values ​​of the visual analysis image patch clicked by the user. That is, when a user clicks on an image patch, the algorithm requests the processing of L × num_heads attention maps.

[0110] Users can choose between global and local normalization methods based on their needs. First, the user selects the image to be visually analyzed, and then chooses the normalization method. Clicking on an image patch in the image will prompt the system to request and process the attention map data corresponding to the patch's coordinates (x, y).

[0111] When local normalization is selected, the algorithm performs normalization within each attention map individually. That is, it performs min-max normalization on each of the L×num_heads attention maps, resulting in image values ​​between 0 and 255. First, we process the attention map A to obtain a temporary matrix temp = A - min(A), and then we obtain the normalized image A^' using A^' = temp / max(temp) × 255.

[0112] When global normalization is selected, the algorithm visualizes the entire range of attention maps corresponding to the clicked image patch in the visual analysis image. This allows observation of which layers and heads have larger attention values. During global normalization, for the L×num_heads attention map matrix A corresponding to the clicked image patch, a temporary matrix temp = A-min(A) is obtained after processing. Then, the normalized L×num_heads images A^' are obtained by processing A^' = temp / max(temp)×255.

[0113] After normalization, the attention map is transformed into a heatmap using the JET algorithm, with areas showing higher values ​​highlighted in red for emphasis. During global normalization, we allow users to slide a scaler (r) to more clearly view the attention values. Multiplying the heatmap by r may result in values ​​greater than 255, so we set a threshold of 255 for values ​​exceeding 255. The heatmap is then overlaid on the original image to demonstrate the attention map's effect on the original image.

[0114] A second aspect of the present invention: a visual analysis device based on Transformer self-attention, the device comprising the following modules:

[0115] Model training module: Users train their own models;

[0116] Data acquisition module: Users can use the provided API interface to save the self-attention data and raw input data generated by the model during training;

[0117] Log writing module: It uses the model training data saved by the user, rewrites it into a log data format that the system can parse, and feeds it into the visual analysis system to obtain visual analysis results;

[0118] Results Analysis Module: Users can view the visual analysis results obtained by the visual analysis system, analyze the model results step by step through multi-dimensional visual analysis charts and interactive links with the charts, and explore the self-attention mechanism inside the model.

[0119] A third aspect of the present invention: a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement steps in a visual analysis method based on Transformer self-attention.

[0120] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0121] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0122] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0123] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A visual analytics method based on Transformer self-attention, characterized in that, The method includes the following steps: (1) Model training: Users train their own models; (2) Data acquisition: Users use the provided API interface to save the self-attention data and original input data generated by the model during training; (3) Log writing: Using the model training data saved by the user, the data is rewritten into a log data format that the system can parse, and then fed into the visual analysis system to obtain visual analysis results; the visual analysis results obtained in the visual analysis system are specifically as follows: Visual analytics systems analyze data in two ways: attention-based visual analytics in the field of natural language processing and attention-based visual analytics in the field of computer vision. The attention visual analysis in the field of natural language processing includes three major components: statistical information tables, statistical information graphs, and attention visualization. Through visual analysis charts, we can understand the overall distribution and statistical patterns of self-attention in the training layers and attention heads of deep learning models. At the same time, through data link graphs and matrix graphs, we can see the connection between self-attention in specific training samples. The attention visual analysis in the field of computer vision is used to represent the mutual attention between pixel blocks in the training task. It examines the attention distribution between different layers and heads through two methods: global normalization and local normalization, and obtains data on how downstream tasks obtain the corresponding results. The attention visualization analysis in the natural language processing domain specifically refers to: (3.1) The overall self-attention status of all training layers and attention heads of the model is displayed in the form of statistical information tables. Users can sort and view the data by different statistical indicators such as maximum value (max), minimum value (min), interquartile range (quar), and variance (variance). (3.2) At the same time, the self-attention information is displayed in the form of statistical information charts. The self-attention information of each training layer and each attention head is arranged in the form of a sunburst chart. The starting position of the bar chart encodes the maximum and minimum self-attention features of the attention head, and the color encodes the variance features. When the mouse hovers over the corresponding attention head, the specific information of each attention head can be displayed. (3.3) At the same time, the Sunrise Chart supports multiple filtering operations and allows users to choose different filtering criteria. Users can discover attention heads that are of interest or have abnormal self-attention and explore the reasons in the visual analysis chart of self-attention of a single attention head. After the user selects a single attention head, two visual analysis charts are provided to show the self-attention distribution of each head with respect to the input character. The attention visual analysis in the field of computer vision is specifically as follows: For attention map data attn_map, the dimension is (L, num_heads, h, w, h, w), where L is the number of model layers, num_heads is the number of attention heads, h is the height of the attention map, and w is the width of the attention map; h and w correspond to the x and y values ​​of the visual image patch clicked by the user, that is, when the user clicks on an image patch, the algorithm will request to process L×num_heads attention maps; Users can choose between global or local normalization based on their needs. First, the user selects the image to be visually analyzed, then chooses the normalization method, and clicks on an image patch. The system requests the attention map data corresponding to the patch's coordinates (x, y) and performs normalization. After normalization, the attention map is transformed into a heatmap using the JET algorithm, with areas showing higher values ​​highlighted in red for emphasis. When global normalization is selected, users can slide the scale r to view the attention values ​​more clearly. After multiplying the heatmap by the scale r, if any value is greater than 255, it will be automatically set to 255 due to a pre-set threshold. Finally, the heatmap is overlaid on the original image to demonstrate the effect of the attention map on the original image. (4) Results analysis: Users view the visual analysis results obtained by the visual analysis system, and through multi-dimensional visual analysis charts and interactive linkage with the charts, they gradually analyze the model results and explore the self-attention mechanism inside the model.

2. The visual analysis method based on Transformer self-attention according to claim 1, characterized in that, The self-attention calculation process for a single attention head in step (3.3) is as follows: (3.3.1) Encode the input text into vector form. N is the number of input characters, and D is the vector dimension; (3.3.2) Position encoding information, encoded as X and t are added together, and the encoded vector X carrying the location information is fed into the model's encoding layer; (3.3.3) In the model encoding layer, through linear transformation, the query vector sequence Q, the key vector sequence K, and the value vector sequence V are obtained: ;in, This is the corresponding linear transformation matrix; (3.3.4) In the self-attention model, the scaled dot product is used as the attention scoring function, and the output self-attention is as follows: ; Output self-attention vector , Each term represents the self-attention magnitude of the i-th character with respect to other characters. This value is input into the color mapping function colorProject to obtain the color mapping score matrix C, which is used to draw the data link graph and matrix graph.

3. The visual analysis method based on Transformer self-attention according to claim 2, characterized in that, The steps for drawing the data link diagram are as follows: Input attention data Z, calculate length Len; Calculate the size of the data link graph, with a width of TextBoxWidth * 2 + AttentionWidth and a height of TextBoxHeight * Len; Using the attention matrix Z, determine the corresponding quantitative relationship between each character and calculate the connection offset; By using the color mapping fraction matrix C and combining it with the corresponding positional relationships, the colors of the corresponding connecting lines are determined. Draw a data link diagram.

4. The visual analysis method based on Transformer self-attention according to claim 2, characterized in that, The steps for drawing the matrix diagram are as follows: Input attention data Z, calculate length Len; Calculate the size of the matrix diagram, with a width of MatrixBox * Len + TextBoxWidth and a height of MatrixBox * Len + TextBoxHeight; Generate a matrix grid; Use the color fraction matrix C to color the matrix grid.

5. The visual analysis method based on Transformer self-attention according to claim 1, characterized in that, The user selects either a global normalization method or a local normalization method according to their needs, specifically: When the user selects the local normalization method, the algorithm will perform normalization operation in each attention map; that is, it will perform min-max normalization in L×num_heads attention maps respectively, so that the image value is 0-255; first, the temporary matrix temp=A - min(A) is obtained by processing the attention map A, and then the normalized image A^' is obtained by A^'=temp / max(temp)×255; When the user selects the global normalization method, the algorithm will visualize the entire range of attention maps corresponding to the clicked image patch in the visual analysis image. This method allows observation of the magnitude of attention values ​​for certain layers and heads. During global normalization, for the L×num_heads attention map matrix A corresponding to the clicked image patch, a temporary matrix temp=A - min(A) is obtained after processing. Then, the normalized L×num_heads images A^' are obtained after processing A^'=temp / max(temp)×255.

6. An apparatus for a visual analysis method based on Transformer self-attention according to any one of claims 1-5, characterized in that, The device includes the following modules: Model training module: Users train their own models; Data acquisition module: Users can use the provided API interface to save the self-attention data and raw input data generated by the model during training; Log writing module: It uses the model training data saved by the user, rewrites it into a log data format that the system can parse, and feeds it into the visual analysis system to obtain visual analysis results; Results Analysis Module: Users can view the visual analysis results obtained by the visual analysis system, analyze the model results step by step through multi-dimensional visual analysis charts and interactive links with the charts, and explore the self-attention mechanism inside the model.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the visual analysis method based on Transformer self-attention as described in any one of claims 1-5.

Images