Method for optimizing a neural network based on a workflow and comprising an attention layer

Abstract

A method for optimizing a workflow-based neural network including an attention layer is provided. The method includes training the workflow-based neural network to assign one or more attention placements and a plurality of weights to one or more input elements based on an original attention function through the attention layer, one or more prediction results from the one or more input elements under a prediction model; obtaining an original attention mask pattern and a suggested attention mask pattern; creating an attention mask update function based on the original attention mask pattern and the suggested attention mask pattern; and combining the attention mask update function with the original attention function to form an updated attention function.

Description

Technical Field

[0001] This invention generally relates to artificial intelligence and deep learning technologies. More specifically, this invention relates to workflow-based neural network optimization techniques. Background Technology

[0002] When training neural networks for machine learning (ML) models, limited knowledge exists regarding which features the ML model should learn. In cases of inaccurate predictions, the ML model output provides engineers with so little information that they cannot intuitively guess what is wrong with the model or the training data. Consequently, optimizing the model is always time-consuming and resource-intensive.

[0003] Attention layers are widely used in transformer models to perform various tasks, such as optical character recognition. By using attention layers, transformer models place more attention (or attention weights) on certain regions of the input in an attempt to improve the accuracy of predictions.

[0004] For example, a deep learning model implemented using a convolutional neural network (CNN) and a vanilla transformer decoder with one or more attention layers can be trained to predict the correct sequence of the three characters "a", "b", and "c" in a text image. Figure 1 As shown, at time t=1, the model correctly predicted the first character "b"; however, the attention weights were placed on both "b" and "a", which seems illogical. At time t=2, even though the attention weights were placed on the incorrect character "b", the model correctly predicted the second character "a". A similar mismatch between the output and the placement of the attention weights can be observed at time t=3. Therefore, since we don't know exactly what the model is doing, it's difficult to troubleshoot the model when the output is incorrect.

[0005] Various methods have been employed in the industry to improve the performance of transformer models that include attention layers. Some methods utilize hyperparameters, such as model scaling, loss functions, regularizations, and optimizer parameters. Applications in ML model training include grid search, random search, and hand-tuning. However, most of these methods require intensive resources to find the optimal combination through trial and error among many possible hyperparameter combinations. Although some automation techniques can be utilized, these methods are generally only usable by resource-rich organizations because the number of combinations grows exponentially with the number of hyperparameters.

[0006] Some approaches are based on data augmentation techniques, such as data collection, data cleaning, and data augmentation. Among these methods, providing higher-quality training data during the training process can significantly improve ML models. However, high-quality training data is often hard to come by. Data for certain knowledge domains may be difficult to obtain. There may be hidden inconsistencies between data points in the training dataset; the dataset may also contain imbalanced data.

[0007] Other approaches based on architectural innovation improve model performance by implementing innovative architectural designs. For example, Chinese Patent Publication No. CN114049408A discloses an auxiliary branch for a transform encoder used to assist its main branch. Some architectural innovations can be effective in improving the performance of ML models. However, these innovative ideas may require extensive experimentation before widespread adoption, which is costly. Due to resource and time constraints, these innovative ideas are not always commercially viable.

[0008] Other feature engineering-based methods involve analyzing, selecting, and processing raw data to transform it into specific features, which are then used as input to a neural network. This can eliminate the negative impact of outliers and filter out unnecessary information. However, most of these methods rely on good domain knowledge and trial-and-error to determine which features work and which don't. Therefore, while these methods are accepted by data scientists, they are not popular with machine learning engineers.

[0009] Nevertheless, a common problem with the above methods is the need for large training datasets to train machine learning models. When the amount of training data is limited, the model tends to learn some incorrect things, and its predictive performance is affected. There is a need in the field for a better approach to provide engineers with clear direction in their work with machine learning models, enabling them to have better insights into the work of machine learning models and allowing human knowledge to be injected into the models to correct for shortcomings in machine learning. Summary of the Invention

[0010] One object of the present invention is to provide a method for optimizing workflow-based neural networks with attention layers by injecting human knowledge during training.

[0011] According to one embodiment of the present invention, a method for optimizing a workflow-based neural network with an attention layer is provided. The method includes: training the workflow-based neural network; assigning attention placement positions and weights to one or more input elements based on an original attention function through the attention layer; predicting results from the input elements under a prediction model until the prediction model converges; obtaining an original attention mask pattern and a proposed attention mask pattern; creating an attention mask update function based on the original attention mask pattern and the proposed attention mask pattern; and combining the attention mask update function with the original attention function to form an updated attention function.

[0012] The generation of attention mask pattern suggestions is an iterative process. It provides insights and useful feedback on the performance of the neural network's predictive model, allowing human knowledge to be infused by progressively eliminating potential problems in the predictive model. In turn, the optimization time and cost of the neural network can be significantly reduced. Attached Figure Description

[0013] Embodiments of the invention will now be described in more detail with reference to the accompanying drawings, wherein:

[0014] Figure 1 An example of character recognition performed by a traditional machine learning model is depicted;

[0015] Figure 2 A flowchart is depicted according to an embodiment of the present invention for optimizing a workflow-based neural network including an attention layer;

[0016] Figure 3 A functional block diagram of a workflow-based neural network is depicted.

[0017] Figure 4 A flowchart illustrating the generation of attention pattern suggestions according to one embodiment is shown;

[0018] Figure 5A flowchart illustrating the creation of an attention mask update function according to one embodiment is shown;

[0019] Figure 6 An exemplary original attention mask pattern on an input image is shown according to one embodiment; and

[0020] Figure 7 This shows how to obtain the distance in the x-direction between the centroid of the attention weights at the previous timestep and the centroid of the attention weights at the current timestep. Detailed Implementation

[0021] In the following description, methods for optimizing neural networks are illustrated as preferred examples. It will be apparent to those skilled in the art that modifications, including additions and / or substitutions, can be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, this disclosure is written to enable those skilled in the art to practice the teachings herein without excessive experimentation.

[0022] refer to Figure 2 and Figure 3 And the following description is provided. For example... Figure 3 As shown, the workflow-based neural network 300 according to various embodiments of the present invention includes an attention layer 301, one or more preceding layers 302 and one or more following layers 303.

[0023] According to an embodiment of the present invention, a method for optimizing a workflow-based neural network 300 is provided. This method includes the following processing steps:

[0024] S202: Train a workflow-based neural network 300, assign one or more attention placements and multiple weights to one or more input elements D201 based on the original attention function D202 through the attention layer 301, and predict the result from the input elements D201 under the prediction model until the prediction model converges.

[0025] S204: Generate attention mask pattern suggestions to obtain the original attention mask pattern D203 and the suggested attention mask pattern D204.

[0026] S206: Create an attention mask update function D206 based on the original attention mask pattern D203 and the proposed attention mask pattern D204; and

[0027] S208: Combine the attention mask update function D206 with the original attention function D202 to form the updated attention function D208 of attention layer 301.

[0028] According to one embodiment, the original attention function D202 is a scaled dot-product attention function with input (input vector), wherein the input includes a key matrix of multiple queries, a key matrix of multiple keywords, and a value matrix of multiple values. Both the key matrix and the keyword matrix have dimension d. k A numerical matrix comprising multiple values ​​has dimension d. v The attention weights for input element D201 are calculated by multiplying the query and keyword by the dot product, and then dividing each dot product by d. k The attention weights are obtained by taking the square root of the value. Then, the obtained attention weights are applied to the value to obtain the attention function. That is, the original attention function D202(Attentoon) can be expressed as:

[0029]

[0030] Where Q represents a character matrix containing multiple queries, K represents a character matrix containing multiple keywords, and V represents a numerical matrix containing multiple values.

[0031] Next, the updated attention function D208(Attentoon′) can be expressed as:

[0032]

[0033] Where f() represents the attention mask update function D206.

[0034] According to one embodiment, the updated attention function D208 of the attention layer 301 is obtained by directly applying the attention mask update function D206 to the original attention function D202 in the attention layer 301.

[0035] According to another embodiment, the updated attention function D208 of the attention layer 301 is obtained by training the workflow-based neural network 300 through backpropagation with an auxiliary loss function, defined by the original attention function D202 and the attention mask update function D206. Thus, the attention layer 301 learns the attention mask update function D206 itself and incorporates it into the workflow-based neural network 300. The auxiliary loss function (AuxLoss) can be expressed as:

[0036]

[0037] In an embodiment of a workflow-based neural network 300 having more than one attention layer, the processing steps S204 to S208 are repeated for each attention layer.

[0038] Figure 4 A flowchart of attention pattern suggestion generation S204 according to a preferred embodiment is shown. As shown, attention pattern suggestion generation S204 includes the following processing steps:

[0039] S2041: Based on multiple features extracted from input element D201, identify the spatial, temporal, and / or contextual characteristics / patterns of input element D201;

[0040] S2042: Visualize attention layer 301 to identify the original attention mask pattern D203 based on multiple attentions placed according to the spatial, temporal and / or contextual features / patterns of the identified input element D201 and one or more original prediction results D209.

[0041] S2044: Determines whether the original attention mask pattern D203 is intuitively meaningful to a human observer, i.e., whether the attention is correctly placed according to the features / patterns of the identified input element D201 and the original prediction result D209 and has appropriate attention weights; and

[0042] S2046: If the original attention mask pattern D203 is not intuitively meaningful, a proposed attention mask pattern D204 is created based on one or more desired attention placement positions and weights on the input element D201. The proposed attention mask pattern D204 can be one of multiple attention mask patterns, each of which has a different set of desired attention placement positions and weights; therefore, there may be multiple attention pattern proposals.

[0043] It should be understood that various types of attention pattern proposals can be created depending on the nature of the prediction model and the input elements. For example, input element D201 may include spatial features of a text image, such as those used for predicting words and phrases; or input element D201 may include temporal features used to determine the time step used in text character prediction; or input element D201 may include contextual features, such as those used for natural language translation prediction.

[0044] The inventive concept of processing step S2044 is obtained by referring to Figure 6 Let's illustrate this with an example. Suppose a workflow-based neural network 300 is trained to predict characters at each time step from multiple input elements D201 with temporal features, and suppose the input elements D201 are multiple text images, denoted as ("I know someone who knows someone who worked for TI"). In the prediction of the first occurrence of the character "w" by the unoptimized workflow-based neural network 300, the original attention mask pattern shows multiple attention placement positions 601. Visualization reveals that attention placement position 601a is incorrect. Therefore, the original attention mask pattern D203 of the attention layer has no intuitive meaning.

[0045] In the aforementioned example, the original attention mask pattern D203 on multiple input elements D201 with temporal characteristics has no intuitive meaning. Therefore, a proposed attention mask pattern could be a pattern with attention placement positions at a single time step. Another proposed attention mask pattern D204 could be a pattern that shifts the attention placement position forward (or to the right) by a small distance at each time step.

[0046] In one embodiment, the proposed attention mask pattern can be a pattern that needs to weaken attention weights far from the centroid of attention weights. In this case, the attention mask update function f can be defined as:

[0047]

[0048] in, This represents multiple attention weights at time t, c t-1 Yes, the center of gravity is W. t-1 Attention weights, e x It is the attention weight on the element at a distance x from this centroid, and the threshold is the pre-defined difference of the attention weight threshold value.

[0049] Figure 5 A flowchart of the creation of an attention mask update function S206 according to a preferred embodiment is shown, which includes the following processing steps:

[0050] S2062: Design the bias function to obtain the bias value D2062, which represents the quantifiable bias between the original attention mask pattern D203 bias and the proposed attention mask pattern D204.

[0051] The concept of the deviation function was invented through Figure 7 Let's illustrate with examples. For instance... Figure 7 As shown, the attention weight W at the previous time step t-1 can be obtained. t-1 The center of gravity c t-1 And the attention weight W at the current time step t t The center of gravity c t The distance d between them in the x-direction (i.e., d = c) t -d t(1 This is used as a deviation value to assess how the original attention mask pattern D203 deviates from the suggested attention mask pattern D204.

[0052] S2064: Determines whether an attention mask update function can be created to satisfy the attention pattern proposal; that is, whether the original attention mask pattern D203 can be manipulated to obtain an updated attention mask pattern that approximates or slightly violates the proposed attention mask pattern D204; in other words, the attention mask update function is created based on the bias function; if the bias function cannot be designed, or cannot be specifically expressed mathematically, then the attention mask update function cannot be created.

[0053] S2066: If an attention mask update function can be created to satisfy the proposed attention mask pattern D204, then create the attention mask update function D206;

[0054] S2067: If the attention mask update function D206 cannot be created to satisfy the suggested attention mask pattern D204, then consider generating a new attention pattern suggestion by performing the processing step S204.

[0055] S2068: Determine whether a new attention pattern suggestion can be obtained. If so, repeat the execution of multiple processing steps starting from processing step S204.

[0056] S2069: If no new attention pattern proposal can be obtained, then a reinforcement learning (RL) model is used as the attention mask update function D206.

[0057] In one embodiment, the bias function is used as part of the reward in the RL model, and one or more loss results from the original prediction model of the workflow-based neural network 300 are used as another part of the reward in the RL model. The actions available to the agent in the RL model are designed to limit the size of the action set. These actions may include changing one or more attention placements and weights.

[0058] experiment

[0059] In an experiment, the performance of an exemplary workflow-based neural network was evaluated, which was optimized using a method for optimizing a workflow-based neural network according to an embodiment of the present invention. The exemplary workflow-based neural network includes a residual network (ResNet50) and a decoder with multiple original transformers. The prediction model of the workflow-based neural network was trained to perform an optical character recognition task, outputting multiple characters from inputs of multiple text images with variable widths. 780,000 English news sentences and synthetic printed English text in over 100 font types were used as the training dataset. The original (unoptimized) prediction model, the prediction model optimized in inference, and the cross-entropy performance of the prediction model optimized through further training were tested using a test dataset of 10,000 non-synthetic text images, and measured using the cross-entropy loss function. Table 1 shows the performance of exemplary workflow-based neural networks in terms of word error rate (WER) and character error rate (CER). As shown in the figure, the model optimized using the method provided in this invention has lower WER and CER than the pre-trained model.

[0060] Table 1: Performance comparison of the original (pre-trained) model, the improved inference model, and the optimized model after further training.

[0061]

[0062] The embodiments disclosed herein may be implemented using computing devices, computer processors, or electronic circuits, including but not limited to application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and other programmable logic devices configured or programmed according to the teachings of this disclosure. Based on the teachings of this disclosure, those skilled in the art of software or electronics can readily prepare computer instructions or software code to run on general-purpose or special-purpose computing devices, computer processors, or programmable logic devices.

[0063] All or some embodiments may be implemented in one or more general-purpose or computing devices, including server computers, personal computers, laptops, mobile computing devices such as smartphones and tablets.

[0064] The embodiments include a computer storage medium storing computer instructions or software code, which can be used to program a computer or microprocessor to perform any of the processes of the present invention. The storage medium may include, but is not limited to, floppy disks, optical disks, Blu-ray discs, DVDs, CD-ROMs and magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of medium or device suitable for storing instructions, code, and / or data.

[0065] Various embodiments of the present invention can also be implemented in distributed computing environments and / or cloud computing environments, wherein one or more processing devices interconnected by a communication network execute all or part of the machine instructions in a distributed manner, the communication network including an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0066] The foregoing description of the invention has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations will be apparent to those skilled in the art.

[0067] The embodiments were chosen and described in order to better explain the principles of the invention and its practical application, thereby enabling other persons skilled in the art to understand the various embodiments and modifications of the invention, suitable for their intended specific use.

Claims

1. A method for optimizing a workflow-based neural network including an attention layer, characterized in that, The workflow-based neural network prediction model is trained to perform an optical character recognition task, the method comprising: The workflow-based neural network is trained by assigning one or more attention placement positions and multiple weights to one or more input elements based on the original attention function through the attention layer, and predicting one or more results from the one or more input elements under the prediction model until the prediction model converges. Generate attention mask pattern suggestions to obtain the original attention mask pattern and the suggested attention mask pattern; An attention mask update function is created based on the original attention mask pattern and the proposed attention mask pattern. and The attention mask update function is combined with the original attention function to form an updated attention function for the attention layer.

2. The method according to claim 1, characterized in that, The generation of the attention pattern suggestion includes: Based on one or more features extracted from the plurality of input elements, identify one or more elements of the plurality of input elements; The attention layer is visualized based on one or more attentions placed according to the multiple identification elements of the multiple input elements and one or more original prediction results to identify the original attention mask pattern. Determining whether the original attention mask pattern has intuitive meaning includes: Determine whether the multiple attentions are correctly placed based on the multiple recognition elements of the multiple input elements and the multiple original prediction results, and have multiple appropriate attention weights; and If the original attention mask pattern does not have intuitive meaning, the proposed attention mask pattern is created based on one or more desired attention placement locations and multiple attention weights on the plurality of input elements.

3. The method according to claim 1, characterized in that, The creation of the attention mask update function includes: Design a bias function to obtain a bias value, wherein the bias value represents a quantifiable bias between the original attention mask pattern bias and the proposed attention mask pattern; Determine whether an attention mask update function can be created to satisfy the attention pattern proposal; If the attention mask update function can be created to satisfy the proposed attention mask pattern, then the attention mask update function is created; If the attention mask update function cannot be created to satisfy the proposed attention mask pattern, then consider generating a new attention pattern proposal; and If the attention mask update function cannot be created to satisfy the proposed attention mask pattern and if a new attention pattern proposal cannot be obtained, then a reinforcement learning (RL) model is used as the attention mask update function.

4. The method according to claim 1, characterized in that, Combining the attention mask update function with the original attention function to form the updated attention function of the attention layer includes: The workflow-based neural network is trained by backpropagation with an auxiliary loss function, which is defined by the original attention function and the attention mask update function.

5. The method according to claim 1, characterized in that, Combining the attention mask update function with the original attention function to form the updated attention function of the attention layer includes: The attention mask update function is directly applied to the original attention mask pattern in the attention layer.

6. The method according to claim 1, characterized in that, The original attention function is a scaled dot product attention function with input, wherein the input includes a character matrix of multiple queries, a character matrix of multiple keywords, and a numerical matrix of multiple values, wherein both the character matrix of the multiple queries and the character matrix of the multiple keywords have dimension d. k ; The multiple attention weights are calculated by computing multiple dot products between multiple queries and the multiple keys, and then dividing each dot product by d. k It is obtained by taking the square root of the ; The multiple attention weights obtained are applied to the multiple values ​​to obtain the attention function.

7. The method according to claim 6, characterized in that, The original attention function is expressed as: ; Where Q represents a character matrix containing multiple queries, K represents a character matrix containing multiple keywords, and d k It is represented by the dimensions of both the character matrix containing multiple queries and the character matrix containing multiple keywords, and V is represented by the numerical matrix containing multiple values; and The updated attention function is expressed as: ; Where f() represents the attention mask update function.

8. The method according to claim 7, characterized in that, The attention mask update function is expressed as: ; in Represented as multiple attention weights at time t, It is the center of gravity Attention weights It is the distance from the center of gravity. The attention weight of the element at the distance, and the threshold is a predetermined difference within the attention weight threshold.

9. A workflow-based neural network including an attention layer, characterized in that... ； The workflow-based neural network is trained by assigning one or more attention placement positions and multiple weights to one or more input elements based on the original attention function through the attention layer, and predicts one or more results from the one or more input elements under the prediction model until the prediction model converges; and The attention layer described herein has an original attention function, which is updated to form an updated attention layer in the following manner: Generate attention mask pattern suggestions to obtain the original attention mask pattern and the suggested attention mask pattern; An attention mask update function is created based on the original attention mask pattern and the proposed attention mask pattern. and The attention mask update function is combined with the original attention function to update the original attention function, thereby forming the updated attention layer.

10. The workflow-based neural network according to claim 9, characterized in that, The generation of the attention pattern suggestion includes: Based on one or more features extracted from the plurality of input elements, identify one or more elements of the plurality of input elements; The attention layer is visualized based on one or more attentions placed by the attention layer to identify the original attention mask pattern, wherein the multiple attentions relate to the multiple input elements of the multiple identification elements and one or more original prediction results; Determining whether the original attention mask pattern has intuitive meaning includes: Determine whether the multiple attentions are correctly placed based on the multiple identification elements of the multiple input elements and the multiple original prediction results, and have multiple appropriate attention weights; and If the original attention mask pattern does not have intuitive meaning, the proposed attention mask pattern is created based on one or more desired attention placement locations and multiple attention weights on the plurality of input elements.

11. The workflow-based neural network according to claim 9, characterized in that, The creation of the attention mask update function includes: Design a bias function to obtain a bias value, wherein the bias value represents a quantifiable bias between the original attention mask pattern bias and the proposed attention mask pattern; Determine whether an attention mask update function can be created to satisfy the attention pattern proposal; If the attention mask update function can be created to satisfy the proposed attention mask pattern, then the attention mask update function is created; If the attention mask update function cannot be created to satisfy the proposed attention mask pattern, then consider generating a new attention pattern proposal; and If the attention mask update function cannot be created to satisfy the proposed attention mask pattern and if a new attention pattern proposal cannot be obtained, then a reinforcement learning (RL) model is used as the attention mask update function.

12. The workflow-based neural network according to claim 9, characterized in that, The updated attention function, which combines the attention mask update function with the original attention function to form the attention layer, includes: The workflow-based neural network is trained by backpropagation with an auxiliary loss function, which is defined by the original attention function and the attention mask update function.

13. The workflow-based neural network according to claim 9, characterized in that, The updated attention function, which combines the attention mask update function with the original attention function to form the attention layer, includes: The attention mask update function is directly applied to the original attention mask pattern in the attention layer.

14. The workflow-based neural network according to claim 9, characterized in that, The original attention function is a scaled dot product attention function with input, wherein the input includes a character matrix of multiple queries, a character matrix of multiple keywords, and a numerical matrix of multiple values, wherein both the character matrix of the multiple queries and the character matrix of the multiple keywords have dimension d. k ; The multiple attention weights are calculated by computing multiple dot products between the multiple queries and the multiple keys, and then dividing each dot product by d. k It is obtained by taking the square root of the ; The multiple attention weights obtained are applied to the multiple values ​​to obtain the attention function.

15. The workflow-based neural network according to claim 14, characterized in that, The original attention function is expressed as: ; Where Q represents a character matrix containing multiple queries, K represents a character matrix containing multiple keywords, and d k It is represented by the dimensions of both the character matrix containing multiple queries and the character matrix containing multiple keywords, and V is represented by the numerical matrix containing multiple values; and The updated attention function is expressed as: ; Where f() represents the attention mask update function.

16. The workflow-based neural network according to claim 15, wherein the attention mask update function is expressed as: ; in Represented as multiple attention weights at time t, It is the center of gravity Attention weights It is the distance from the center of gravity. The attention weight of the element at the distance, and the threshold is a predetermined difference within the attention weight threshold.

Images