Gated cross attention information fusion-based production index evaluation method and device, and medium
By introducing a gated cross-attention mechanism with reset and update gates into the Transformer decoder, the problem of Q-information loss in long-term memory sequences is solved, enabling accurate evaluation of industrial production indicators.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CENT SOUTH UNIV
- Filing Date
- 2026-01-21
- Publication Date
- 2026-05-01
AI Technical Summary
In complex industrial processes, the cross-attention mechanism of existing Transformer decoders is prone to loss of query Q information when dealing with long-term memory sequences, resulting in inaccurate evaluation of production indicators.
We improve the cross-attention of the Transformer decoder by using reset gates and update gates. We design a learnable gated cross-attention mechanism. The reset gate makes the query more sparse and focused, while the update gate controls the information flow fusion and retains the query Q information.
It improves the accuracy of production index estimation tasks for complex industrial processes and enhances the efficiency and accuracy of information fusion.
Smart Images
Figure CN121961337A_ABST
Abstract
Description
A method, apparatus, and medium for evaluating production indicators based on gating cross-attention information fusion. Technical Field
[0001] This invention belongs to the field of industrial control technology, specifically relating to a method, device, and medium for evaluating production indicators based on gating cross-attention information fusion. Background Technology
[0002] When estimating production indicators for complex industrial processes, it is often necessary to integrate historical information and calculate the current sequence to estimate production indicators. When the historical information spans a long period, i.e., is a long-term memory sequence, efficiently, dynamically, and accurately fusing the information flow is a significant challenge. The cross-attention mechanism in the Transformer decoder provides a good solution; however, the attention output is determined by a weighted average of values V, and the information contained in the query Q is not directly retained in the attention output. When the sequence length of values V is large, this can easily lead to the loss of query Q information, resulting in inaccurate production assessments. Summary of the Invention
[0003] This invention provides a method, apparatus, and medium for evaluating production indicators based on gated cross-attention information fusion, which can accurately complete the task of estimating industrial production indicators.
[0004] To achieve the above technical objectives, the present invention adopts the following technical solution:
[0005] A production performance evaluation method based on gated cross-attention information fusion includes:
[0006] Historical industrial process and technology index data are used to construct a long-term memory sequence, and current industrial process and technology index data are used to construct the current sequence.
[0007] The long-term memory sequence and the current sequence are input into the trained production indicator evaluation model to obtain the current industrial production indicator value.
[0008] The production indicator evaluation model includes two Transformer encoders and one Transformer decoder. The Transformer decoder uses a reset gate and an update gate to improve the original cross-attention to obtain gated cross-attention. The inputs of the two Transformer encoders are the current sequence and the long-term memory sequence, respectively, and the outputs are used as Q vectors and K and V vectors, respectively, to the cross-attention of the Transformer decoder.
[0009] Furthermore, it is used to evaluate production indicators in the zinc electrolysis process.
[0010] Furthermore, the industrial process data includes DC current data, DC voltage data, and / or tank temperature data; the process index data includes fresh zinc concentration data, waste zinc concentration data, and / or waste acid concentration data; the production index includes zinc precipitation yield, current efficiency, and / or DC power consumption per ton of zinc.
[0011] Furthermore, the gated cross-attention includes:
[0012] (1) The current sequence is used as the Q vector by the encoder output vector, and the long memory sequence is used as the K vector and V vector by the encoder output vector;
[0013] (2) Calculate the scaled dot product of the Q vector and the K vector to obtain the original attention score, and calculate the reset gate and update gate for the original attention score respectively;
[0014] (3) After calculating the dot product of the reset gate calculation result and the original attention score, softmax calculation is performed to obtain the candidate weights;
[0015] (4) Calculate the product of the V vector and the candidate weights to form the candidate output;
[0016] (5) Calculate the output of gated cross attention based on the candidate output, candidate weights and update gate calculation results.
[0017] Furthermore, the calculation in step (2) is expressed as follows:
[0018]
[0019]
[0020]
[0021] in, For the sigmoid function, The original attention score is calculated using the scaled dot product; This is the Q-vector, which is the output vector of the current sequence after passing through the encoder; Let K be the K-vector, which is the output vector of the long-term memory sequence after passing through the encoder; To reset the calculation results for the door, To update the calculation results of the gate, and These are the learnable and updatable parameters in the production indicator evaluation model. , These represent the dimensions of the K vector, the V vector, and the model dimension, respectively. These represent the total number of heads and their numbers in multi-head attention, respectively.
[0022] Furthermore, the calculation in step (3) is expressed as follows:
[0023]
[0024]
[0025] in, The original attention score. To reset the calculation results for the door, for and The result of the dot product operation, The softmax activation function is used. The candidate weights are...
[0026] Furthermore, step (4) calculates the product of the V vector and the candidate weights as follows:
[0027]
[0028] in, This is a candidate output for gated cross-attention. Candidate weights; V is the output vector of the long-term memory sequence via the encoder.
[0029] Furthermore, the calculation in step (5) is expressed as follows:
[0030]
[0031]
[0032] in, To update the calculation results of the gate, This represents the average calculation. The dimension representing the average is... This represents the result of the update gate calculation after averaging along the last dimension; This is a candidate output for gated cross-attention. This is the final output of gated cross-attention; It is the Q vector, which is the output vector of the current sequence via the encoder.
[0033] A production performance evaluation device based on gated cross-attention information fusion includes:
[0034] The sequence data construction module is used to: acquire historical industrial process and process indicator data to construct long-term memory sequences, and acquire current industrial process and process indicator data to construct the current sequence;
[0035] The production indicator evaluation module is used to: use a trained production indicator evaluation model to evaluate industrial production indicators based on the input long-term memory sequence and the current sequence, and output the current industrial production indicator value.
[0036] The production indicator evaluation model includes two Transformer encoders and one Transformer decoder. The Transformer decoder uses a reset gate and an update gate to improve the original cross-attention to obtain gated cross-attention. The inputs of the two Transformer encoders are the current sequence and the long-term memory sequence, respectively, and the outputs are used as Q vectors and K and V vectors, respectively, to the cross-attention of the Transformer decoder.
[0037] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the production performance evaluation method based on gated cross-attention information fusion as described above.
[0038] Compared with the prior art, the advantages of the present invention are as follows:
[0039] To address the problem that the estimation of production indicators for complex industrial processes requires the integration of historical information over a long time span, leading to the loss of query Q information and resulting in inaccurate production assessments, this invention improves the cross-attention mechanism of the existing Transformer decoder by designing learnable and updatable reset and update gate parameters. The reset gate makes the query more sparse and focused, while the update gate helps the model decide to what extent to adopt the weighted average value V information and to what extent to retain the query Q information, thereby controlling the information flow fusion and enabling accurate estimation of industrial production indicators. Attached Figure Description
[0040] Figure 1 is a flowchart of the production indicator evaluation model according to an embodiment of the present invention evaluating the current industrial production indicators based on the current sequence and the long-term memory sequence.
[0041] Figure 2 shows the results of evaluating the production indicators of the zinc electrolysis process based on gated cross-attention information fusion in the embodiment of the present invention; sub-figures (a), (b), and (c) correspond to the production indicators: current efficiency, DC power consumption per ton of zinc, and zinc precipitation yield, respectively.
[0042] Figure 3 is a visualization of the attention weights of the production index evaluation method based on gating cross-attention information fusion described in the embodiment of the present invention.
[0043] Figure 4 shows the results of evaluating the production indicators of the zinc electrolysis process using the standard Transformer decoder used for comparison in the embodiments of the present invention; sub-figures (a), (b), and (c) correspond to the production indicators: current efficiency, DC power consumption per ton of zinc, and zinc precipitation yield, respectively.
[0044] Figure 5 is a visualization of the attention weights of the standard Transformer decoder used for comparison in the embodiments of the present invention. Detailed Implementation
[0045] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.
[0046] Example 1
[0047] This embodiment uses the production index evaluation method based on gating cross-attention information fusion provided by the present invention to evaluate the production index of the zinc electrolysis process.
[0048] First, construct the long-term memory sequence and the current sequence: obtain historical industrial process and process indicator data to construct the long-term memory sequence, and obtain current industrial process and process indicator data to construct the current sequence.
[0049] This embodiment obtains zinc electrolysis industrial process parameters (DC current, DC voltage, cell temperature), process indicators (concentration of fresh zinc solution, concentration of waste zinc solution, and concentration of waste acid solution), production indicators (zinc yield, current efficiency, and DC power consumption per ton of zinc) and log records from a multi-perspective knowledge base of the zinc electrolysis process, namely, the fused characterization sequence information of a smelting enterprise's data over four consecutive years. Specifically, the current sequence is constructed from industrial process and process indicator data within a certain time period, and a long-term memory sequence is constructed from historical industrial process and process indicator data from the current time to a certain time period prior, which are used as input parameters for subsequent production indicator evaluation.
[0050] More preferably, one data point is added to the end of the current sequence, specifically the average of historical production indicators (the average of historical production indicator data from the current time to a certain time period). In this embodiment, the current sequence consists of 13 data points, with each data point representing 2 hours; that is, the current sequence is composed of industrial process and technological indicator data for the current 24 hours, and the last data point is the average of historical production indicators. The long-term memory sequence consists of 1461 data points, with each day representing one data point; that is, the long-term memory sequence is composed of industrial process and technological indicator data from the previous 4 years. The production indicator has a length of 1, which means that the production indicators for the current day are evaluated. Generally, 365 days or more is considered long-term.
[0051] Specifically, before constructing sequences using industrial process and technology index data, the acquired data is preprocessed using existing technologies, and then the current sequence and long-term memory sequence are constructed.
[0052] Then, the long-term memory sequence and the current sequence are input into the trained production indicator evaluation model to obtain the current industrial production indicator values.
[0053] The production index evaluation model in this embodiment includes two Transformer encoders and one Transformer decoder. The Transformer decoder uses a reset gate and an update gate to improve the original cross attention to obtain gated cross attention. The inputs of the two Transformer encoders are the current sequence and the long-term memory sequence, respectively, and the outputs are used as Q vectors and K and V vectors, respectively, to the cross attention of the Transformer decoder.
[0054] Referring to Figure 1, the process of using a production indicator evaluation model to evaluate current industrial production indicators based on the current sequence and long-term memory sequence includes:
[0055] S1: The current sequence and the long memory sequence are respectively position-encoded, and then processed by two different Transformer encoders. The corresponding output of the current sequence after the encoder is used as the multi-head Q vector, and the corresponding output of the long memory sequence after the encoder is used as the multi-head K vector and multi-head V vector.
[0056] S2: Calculate the scaled dot product of the Q vector and the K vector to obtain the original attention score, and then perform reset and update gate calculations on the original attention score. The calculation is represented as:
[0057]
[0058]
[0059]
[0060] in, For the sigmoid function, The original attention score is calculated using the scaled dot product; This is the Q-vector, which is the output vector of the current sequence after passing through the encoder; Let K be the K-vector, which is the output vector of the long-term memory sequence after passing through the encoder; To reset the calculation results for the door, To update the calculation results of the gate, and These are the learnable and updatable parameters in the production indicator evaluation model. , These represent the dimensions of the K vector, the V vector, and the model dimension, respectively. and These represent the total number of heads and their numbers in multi-head attention, respectively.
[0061] S3: After calculating the dot product of the reset gate result and the original attention score, a softmax calculation is performed to obtain the candidate weights. The calculation is expressed as follows:
[0062]
[0063]
[0064]
[0065] in, The original attention score. To reset the calculation results for the door, for and The result of the dot product operation, The softmax activation function is used. Candidate weights; For vectors The first in One element, For vectors The index of the target element in the data. For vectors Iteration index for summation of elements in a given set of elements. For vectors Dimensions.
[0066] S4: Calculate the product of the V vector and the candidate weights to form the candidate output. The calculation is expressed as:
[0067]
[0068] in, This is a candidate output for gated cross-attention. Candidate weights; V is the output vector of the long-term memory sequence via the encoder.
[0069] S5: Based on the candidate outputs, candidate weights, and the update gate calculation results, calculate the output of the gated cross-attention. The calculation is expressed as follows:
[0070]
[0071]
[0072] in, To update the calculation results of the gate, The dimension representing the average is... The representative calculates the average. This represents the result of the update gate calculation after averaging along the last dimension; This is a candidate output for gated cross-attention. This is the final output of gated cross-attention; It is the Q vector, which is the output vector of the current sequence via the encoder.
[0073] Example 2
[0074] This embodiment provides a production performance evaluation device based on gated cross-attention information fusion, including:
[0075] The sequence data construction module is used to: acquire historical industrial process and process indicator data to construct long-term memory sequences, and acquire current industrial process and process indicator data to construct the current sequence;
[0076] The production indicator evaluation module is used to: use a trained production indicator evaluation model to evaluate industrial production indicators based on the input long-term memory sequence and the current sequence, and output the current industrial production indicator value.
[0077] The production indicator evaluation model includes two Transformer encoders and one Transformer decoder. The Transformer decoder uses a reset gate and an update gate to improve the original cross-attention to obtain gated cross-attention. The inputs of the two Transformer encoders are the current sequence and the long-term memory sequence, respectively, and the outputs are used as Q vectors and K and V vectors, respectively, to the cross-attention of the Transformer decoder.
[0078] The production index evaluation device described in this embodiment is used to evaluate the production index of the zinc electrolysis process as described in Embodiment 1. The specific structure and calculation formula of its production index evaluation model are the same as those in Embodiment 1.
[0079] Example 3
[0080] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the production indicator evaluation method described in Embodiment 1.
[0081] Figure 2 shows the results of estimating the production indicators of the zinc electrolysis process using the method of the present invention. Compared with Figure 4, the estimation results of the present method are generally closer to the actual values. Combining the attention weight visualization results of Figures 3 and 5, it can be seen that the method of the present invention performs gating adjustment on the attention distribution of the original standard. The reset gate dynamically adjusts the information flow, making the attention distribution more sparse and focused.
[0082] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.
Claims
1. A production performance evaluation method based on gated cross-attention information fusion, characterized in that, include: Historical industrial process and technological indicator data are acquired to construct a long-term memory sequence, and current industrial process and technological indicator data are acquired to construct a current sequence. The long-term memory sequence and the current sequence are input into a trained production indicator evaluation model to obtain the current industrial production indicator value. The production indicator evaluation model includes two Transformer encoders and one Transformer decoder, and the Transformer decoder uses a reset gate and an update gate to improve the original cross-attention to obtain a gated cross-attention. The inputs of the two Transformer encoders are the current sequence and the long-term memory sequence, respectively, and the outputs are used as Q vectors and K and V vectors, respectively, to the cross-attention of the Transformer decoder.
2. The production indicator evaluation method based on gated cross-attention information fusion according to claim 1, characterized in that, Used to evaluate production indicators in the zinc electrolysis process.
3. The production indicator evaluation method based on gated cross-attention information fusion according to claim 2, characterized in that, The industrial process data includes DC current data, DC voltage data, and / or tank temperature data; the process index data includes fresh zinc concentration data, waste zinc concentration data, and / or waste acid concentration data; the production index includes zinc precipitation yield, current efficiency, and / or DC power consumption per ton of zinc.
4. The production indicator evaluation method based on gated cross-attention information fusion according to claim 1, characterized in that, The gated cross-attention includes: (1) taking the current sequence as the Q vector through the encoder's output vector and the long-term memory sequence as the K vector and V vector through the encoder's output vector; (2) performing a scaled dot product calculation on the Q vector and the K vector to obtain the original attention score, and performing reset gate and update gate calculations on the original attention score respectively; (3) performing a dot product calculation on the reset gate calculation result and the original attention score, and then performing softmax calculation to obtain the candidate weights; (4) performing a dot product calculation on the V vector and the candidate weights to form the candidate output; (5) calculating the output of the gated cross-attention based on the candidate output, the candidate weights and the update gate calculation result.
5. The production indicator evaluation method based on gated cross-attention information fusion according to claim 4, characterized in that, The calculation in step (2) is expressed as follows: ； ； ;in, For the sigmoid function, The original attention score is calculated using the scaled dot product; This is the Q-vector, which is the output vector of the current sequence after passing through the encoder; Let K be the K-vector, which is the output vector of the long-term memory sequence after passing through the encoder; To reset the calculation results for the door, To update the calculation results of the gate, and These are the learnable and updatable parameters in the production indicator evaluation model. ， These represent the dimensions of the K vector, the V vector, and the model dimension, respectively. These represent the total number of heads and their numbers in multi-head attention, respectively.
6. The production indicator evaluation method based on gated cross-attention information fusion according to claim 4, characterized in that, The calculation in step (3) is expressed as follows: ； ;in, The original attention score. To reset the calculation results for the door, for and The result of the dot product operation, The softmax activation function is used. The candidate weights are...
7. The production indicator evaluation method based on gated cross-attention information fusion according to claim 4, characterized in that, Step (4) calculates the product of the V vector and the candidate weights as follows: ;in, This is a candidate output for gated cross-attention. Candidate weights; V is the output vector of the long-term memory sequence via the encoder.
8. The production indicator evaluation method based on gated cross-attention information fusion according to claim 4, characterized in that, The calculation in step (5) is expressed as follows: ； ;in, To update the calculation results of the gate, This represents the average calculation. The dimension representing the average is... This represents the result of the update gate calculation after averaging along the last dimension; This is a candidate output for gated cross-attention. This is the final output of gated cross-attention; It is the Q vector, which is the output vector of the current sequence via the encoder.
9. A production indicator evaluation device based on gated cross-attention information fusion, characterized in that, include: The sequence data construction module is used to: acquire historical industrial process and process indicator data to construct long-term memory sequences, and acquire current industrial process and process indicator data to construct the current sequence; The production indicator evaluation module is used to: evaluate industrial production indicators based on the input long-term memory sequence and the current sequence using a trained production indicator evaluation model, and output the current industrial production indicator value; wherein, the production indicator evaluation model includes two Transformer encoders and one Transformer decoder, and the Transformer decoder uses a reset gate and an update gate to improve the original cross-attention to obtain gated cross-attention; the inputs of the two Transformer encoders are the current sequence and the long-term memory sequence, respectively, and the outputs are respectively used as Q vector and K, V vectors input to the cross-attention of the Transformer decoder.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.