Prediction methods and related equipment for lightweight edge models of the Industrial Internet
By using nonlinear mapping instead of similarity calculation in the lightweight edge model of the Industrial Internet, the problem of low computational efficiency of the Transformer model is solved, and more efficient state analysis is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-26
AI Technical Summary
The Transformer model has low computational efficiency in industrial equipment condition analysis, resulting in low analysis efficiency.
We adopt a lightweight edge model for the Industrial Internet, which replaces similarity calculation by nonlinear mapping of query tensors and key tensors in the self-attention layer, thereby reducing computational complexity and improving the efficiency of state analysis.
By reducing computational complexity, the efficiency of condition analysis for industrial equipment has been improved.
Smart Images
Figure CN121579997B_ABST
Abstract
Description
Technical Field
[0001] This application relates to artificial intelligence, and more particularly to a prediction method and related equipment for a lightweight edge model of the industrial internet. Background Technology
[0002] In industrial production, the condition of industrial equipment directly affects the production schedule. Therefore, continuous monitoring of the equipment's condition during operation is crucial for predicting potential malfunctions and allowing for proactive intervention. For example, collecting and analyzing equipment status data during operation yields a status analysis report.
[0003] In existing technologies, state data can be analyzed using artificial intelligence models to obtain state analysis results for industrial equipment. Commonly used artificial intelligence models include the Transformer model, which can extract attention from state data and generate prediction results.
[0004] However, the Transformer model suffers from low computational efficiency, resulting in low efficiency in the state analysis of industrial equipment. Summary of the Invention
[0005] This application provides a prediction method and related equipment for a lightweight edge model of the Industrial Internet, which can improve the efficiency of state analysis of industrial equipment.
[0006] In a first aspect, embodiments of this application provide a prediction method for a lightweight edge model of the Industrial Internet, including:
[0007] The process involves acquiring state data from industrial equipment, which consists of data sequences collected by one or more sensors. Tensorization is performed on the state data to obtain a state tensor, the order of which is greater than the order of the state data (2). An encoding model is used to generate an encoded result tensor corresponding to the state tensor. This encoding model includes multiple cascaded encoders, each consisting of a self-attention layer and a position feedforward layer following the self-attention layer. The position feedforward layer includes a first multilayer perceptron. In the self-attention layer, the encoder's input tensor is mapped to query, key, and value tensors using query mapping tensors, key mapping tensors, and value mapping tensors. The output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query and key tensors, and the value tensor. Finally, a second multilayer perceptron predicts the state analysis results of the industrial equipment based on the encoded result tensor.
[0008] In one possible implementation, the output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, respectively, and the value tensor, including:
[0009] By using low-rank tensors with rank less than or equal to a preset threshold, the key tensor and value tensor are transformed into low-order key tensors and low-order value tensors, respectively. Based on the product of the nonlinear mapping tensors corresponding to the query tensor and the low-order key tensor, and the low-order value tensor, the output tensor of the self-attention layer is generated.
[0010] In one possible implementation, the output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the low-order key tensor, respectively, and the low-order value tensor, including:
[0011] The query tensor and low-order key tensor are mapped to nonlinear query tensor and nonlinear key tensor respectively using a kernel function; the first product of the nonlinear query tensor and nonlinear key tensor is calculated, and the second product of the nonlinear query tensor, nonlinear key tensor and low-order value tensor is calculated, and the ratio of the second product to the first product is used as the output tensor of the self-attention layer.
[0012] In one possible implementation, mapping the encoder's input tensor to query tensors, key tensors, and value tensors via query mapping tensors, key mapping tensors, and value mapping tensors includes:
[0013] For each of the query mapping tensor, key mapping tensor, and value mapping tensor, the mapping tensor is decomposed into two chained tensors. The product of the two chained tensors equals the mapping tensor. The two chained tensors corresponding to the mapping tensor include a first chained tensor and a second chained tensor. The encoder's input tensor is converted into an intermediate mapping tensor through each first chained tensor. Each key mapping tensor, query mapping tensor, and value mapping tensor corresponds to an intermediate mapping tensor. The intermediate mapping tensors corresponding to each mapping tensor are mapped into query tensors, key tensors, and value tensors, respectively, through the second chained tensors of each mapping tensor.
[0014] In one possible implementation, the encoder further includes a post-processing layer located after the position feedforward layer, in which the output tensor of the position feedforward layer is normalized and nonlinearly processed to serve as the output tensor of the encoder.
[0015] In one possible implementation, tensor quantization is performed on the state data to obtain a state tensor, including:
[0016] Obtain the number of sensors collecting state data and the length of the data sequence collected by each sensor; convert the number of sensors into the product of J first values and the length of the data sequence into the product of I second values, where I and J are both positive integers greater than or equal to 2; tensor the state data into an I+J order state tensor based on the J first values and I second values, where the J first values and I second values are both the length of the data sequence of the state tensor in the corresponding dimension.
[0017] Secondly, embodiments of this application provide a prediction device for a lightweight edge model of the Industrial Internet, comprising:
[0018] The status data acquisition module is used to acquire status data of industrial equipment. Status data is a data sequence collected by one or more sensors.
[0019] The tensor quantization module is used to perform tensor quantization on the state data to obtain the state tensor. The order of the state tensor is greater than the order of the state data (2).
[0020] The encoding module is used to generate the encoded result tensor corresponding to the state tensor through the encoding model. The encoding model includes multiple encoders cascaded in sequence. Each encoder includes a self-attention layer and a position feedforward layer located after the self-attention layer. The position feedforward layer includes a first multilayer perceptron. In the self-attention layer, the input tensor of the encoder is mapped to the query tensor, key tensor, and value tensor through query mapping tensor, key mapping tensor, and value mapping tensor. The output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and key tensor, respectively, and the value tensor. In the position feedforward layer, the output tensor of the self-attention layer in this encoder is linearly mapped.
[0021] The prediction module is used to predict the state analysis results of industrial equipment based on the encoding result tensor using a second multilayer perceptron.
[0022] In one possible implementation, the encoding module is also used for:
[0023] By using low-rank tensors with rank less than or equal to a preset threshold, the key tensor and value tensor are transformed into low-order key tensors and low-order value tensors, respectively. Based on the product of the nonlinear mapping tensors corresponding to the query tensor and the low-order key tensor, and the low-order value tensor, the output tensor of the self-attention layer is generated.
[0024] In one possible implementation, the encoding module is also used for:
[0025] The query tensor and low-order key tensor are mapped to nonlinear query tensor and nonlinear key tensor respectively using a kernel function; the first product of the nonlinear query tensor and nonlinear key tensor is calculated, and the second product of the nonlinear query tensor, nonlinear key tensor and low-order value tensor is calculated, and the ratio of the second product to the first product is used as the output tensor of the self-attention layer.
[0026] In one possible implementation, the encoding module is also used for:
[0027] For each of the query mapping tensor, key mapping tensor, and value mapping tensor, the mapping tensor is decomposed into two chained tensors. The product of the two chained tensors equals the mapping tensor. The two chained tensors corresponding to the mapping tensor include a first chained tensor and a second chained tensor. The encoder's input tensor is converted into an intermediate mapping tensor through each first chained tensor. Each key mapping tensor, query mapping tensor, and value mapping tensor corresponds to an intermediate mapping tensor. The intermediate mapping tensors corresponding to each mapping tensor are mapped into query tensors, key tensors, and value tensors, respectively, through the second chained tensors of each mapping tensor.
[0028] In one possible implementation, the encoder also includes a post-processing layer located after the position feedforward layer. The encoding module is further configured to normalize and nonlinearly process the output tensor of the position feedforward layer in the post-processing layer, and use it as the output tensor of the encoder.
[0029] In one possible implementation, the tensor quantization module is also used for:
[0030] Obtain the number of sensors collecting state data and the length of the data sequence collected by each sensor; convert the number of sensors into the product of J first values and the length of the data sequence into the product of I second values, where I and J are both positive integers greater than or equal to 2; tensor the state data into an I+J order state tensor based on the J first values and I second values, where the J first values and I second values are both the length of the data sequence of the state tensor in the corresponding dimension.
[0031] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0032] The memory stores the instructions that the computer executes;
[0033] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0034] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0035] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0036] The prediction method and related equipment for the lightweight edge model of the Industrial Internet provided in this application, when extracting features through the self-attention layer, perform nonlinear mapping on the query tensor and the key tensor, and replace similarity with the product of the nonlinear mapping tensor of the query tensor and the nonlinear mapping tensor of the key tensor, thereby reducing computational complexity and improving state analysis efficiency. Attached Figure Description
[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0038] Figure 1 This is a schematic diagram of an application scenario provided in this application;
[0039] Figure 2 This is a schematic diagram of a data processing flow provided in this application;
[0040] Figure 3 This is a flowchart illustrating the steps of a prediction method for a lightweight edge model of the Industrial Internet provided in this application.
[0041] Figure 4 This is a schematic diagram of the structure of an encoding model provided in this application;
[0042] Figure 5 This application provides a data processing flowchart for a single-head self-attention layer;
[0043] Figure 6 This is a schematic diagram of the predictive device for the lightweight edge model of the Industrial Internet provided in this application;
[0044] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application.
[0045] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.
[0047] This application is used for condition analysis of industrial equipment. Figure 1 This is a schematic diagram of an application scenario provided in this application, for reference. Figure 1 As shown, one or more sensors can be placed on or around industrial equipment in an industrial setting. These sensors are used to collect status data from different locations on the industrial equipment. These sensors are communicatively connected to electronic devices and transmit the collected status data to the electronic devices, enabling the electronic devices to analyze the status of the industrial equipment based on the status data.
[0048] It should be noted that, Figure 1 Four sensors are provided as an example, but in real-world scenarios, the number and type of sensors can be flexibly set according to actual needs. These sensors can be of the same type and / or different types, and can be placed at any location on industrial equipment. Sensors can be position sensors, image sensors, laser sensors, etc., and this application does not limit the type of sensor.
[0049] The aforementioned electronic devices can be any device with computing capabilities, such as computers, mobile phones, tablets, servers, etc.
[0050] Understandably, the aforementioned electronic devices can be equipped with artificial intelligence models to perform state analysis. These AI models typically utilize the Transformer model, which includes a self-attention mechanism. A typical self-attention mechanism is implemented using the following formula:
[0051] ; (1)
[0052] in, Let D be the output matrix of the self-attention mechanism, and D be the scaling factor. Let K be the transpose of K. Q, K, and V are the query matrix, key matrix, and value matrix obtained after linear transformation of the input matrix X of the self-attention mechanism, respectively. , , ,in, , and These are the query mapping matrix, key mapping matrix, and value mapping matrix, respectively. The softmax function is a normalization exponential function used to normalize each element in a matrix or vector. It is a well-known function in the field of artificial intelligence and will not be elaborated upon here.
[0053] Based on the above formula (1), the i-th row of the output matrix of the self-attention mechanism as follows:
[0054] ; (2)
[0055] Where N is the number of columns in the input matrix X, that is, the length of each row. Similarity function Used to represent the softmax function. To query the i-th row of matrix Q, Let j be the j-th column of the key matrix K. Let be the j-th column of the value matrix V. For the above formula (1)... The i-th row requires two nested loops to compute each query vector. and each key vector Sum value vector The corresponding new representations are given, therefore, the computational complexity is O(n). It can be seen that the conventional Transformer self-attention mechanism has high computational complexity, which leads to a large amount of computation for state analysis of industrial equipment, resulting in low analysis efficiency.
[0056] To address the aforementioned technical issues, this application proposes replacing the similarity calculation process with the product of two nonlinear mapping tensors, thereby linearizing the similarity and effectively reducing computational complexity, thus improving the efficiency of state analysis for industrial equipment.
[0057] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0058] Figure 2 This is a schematic diagram of a data processing flow provided in this application, for reference. Figure 2 As shown, the data processing flow of this application involves three modules: a preprocessing module, an encoding model, and a second multilayer perceptron. The input to the preprocessing module is state data, and the output of the preprocessing module serves as the input to the encoding model. The output of the encoding model serves as the input to the second multilayer perceptron, and the output of the second multilayer perceptron is the state output result of the industrial equipment. In the data processing flow of this application, firstly, the state data collected by the sensors from the industrial equipment is processed by the preprocessing module to obtain a state tensor; then, the state tensor is input into the encoding model to obtain an encoded result tensor; finally, the second multilayer perceptron analyzes the encoded result tensor to obtain the state analysis result.
[0059] Among some possible implementations, refer to Figure 2 As shown, the preprocessing module described above may specifically include a position encoding unit and a tensor quantization unit. The position encoding unit is used to add position information to the state data, and the tensor quantization unit is used to perform tensor quantization on the state data with added position information to obtain a state tensor.
[0060] Figure 3This is a flowchart illustrating the steps of a prediction method for a lightweight edge model of the Industrial Internet provided in this application, executed by an electronic device. (Refer to...) Figure 3 As shown, the prediction method of the above-mentioned lightweight edge model of the Industrial Internet includes the following steps:
[0061] S201. Acquire status data of industrial equipment. Status data is a data sequence collected by one or more sensors.
[0062] Status data refers to any data used to represent the operating status of industrial equipment, and its form is related to the sensor. When the sensor is an image sensor, the status data can be an image; when the sensor is a position sensor, the status data can be position data.
[0063] Specifically, status data can be collected by sensors and transmitted to electronic devices via wireless or wired communication. It is understood that status data can be collected and transmitted to electronic devices in real time, or it can be transmitted to electronic devices periodically. Alternatively, status data can be stored in a database by the sensors, allowing electronic devices to retrieve the status data from the database. This application does not limit the method of transmitting status data.
[0064] State data can include multiple data sequences, each corresponding to data collected by a sensor at multiple times. It can be understood that each sensor continuously collects data sequences, and after acquiring these sequences, the electronic device can use a sliding window to continuously extract data sequences from the most recent time period to obtain data sequences for state analysis. Each sensor corresponds to one data sequence, and multiple sensors correspond to multiple data sequences. For example, when each data sequence is of length N and the number of sensors is M, M state data of length N can be obtained, that is, state data X1 (X1... ) is an N×M matrix, and the state data X1 is used to execute the subsequent step S202.
[0065] S202. Tensor quantization is performed on the state data to obtain the state tensor. The order of the state tensor is greater than the order of the state data (2).
[0066] Tensor quantization is used to transform data from a lower-order form to a higher-order form. State data is a matrix, which can be understood as second-order data, while state tensors are tensors of order greater than 2.
[0067] The tensor quantization process described above can be any tensor algorithm. In some implementations, to more accurately represent the state data and avoid data loss, the state tensor can be generated through the following process: First, obtain the number of sensors collecting state data and the length of the data sequence collected by each sensor; then, convert the number of sensors into the product of J first values, and convert the data sequence length into the product of I second values, where I and J are both positive integers greater than or equal to 2; finally, based on the J first values and I second values, tensor the state data into an I+J order state tensor, where the J first values and I second values are both the data sequence lengths of the state tensor in the corresponding dimensions.
[0068] The number of sensors can be pre-sent to the electronic device or pre-input into the electronic device, while the data sequence length is set according to actual needs. Number of sensors Data sequence length Therefore, the state tensor , The i-th dimension contains a step size of The total length is Time series data.
[0069] In some implementations, refer to Figure 2 As shown, before performing the tensor quantization process described above using the tensor quantization unit, positional information can be added to the state data using the tensor quantization unit to include the position of each element within the state data. This allows the subsequent coding model to perform state analysis more accurately.
[0070] Unlike matrix operations in conventional human-computer intelligence models, based on the tensor quantization process described above, the operations performed in the coding model and the second multilayer perceptron in this application are all tensor operations.
[0071] S203. Generate the encoded result tensor corresponding to the state tensor through the encoding model. The encoding model includes multiple encoders cascaded in sequence. Each encoder includes a self-attention layer and a position feedforward layer located after the self-attention layer. The position feedforward layer includes a first multilayer perceptron. In the self-attention layer, the input tensor of the encoder is mapped to the query tensor, key tensor, and value tensor through the query mapping tensor, key mapping tensor, and value mapping tensor. The output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and key tensor, respectively, and the value tensor.
[0072] In a cascaded encoder, the output of each encoder serves as the input to the next encoder. That is, the input of the first encoder serves as the input to the encoding model, and the output of the last encoder serves as the output of the encoding model. For intermediate encoders, the input is the output of the previous encoder, and the output serves as the input to the next encoder.
[0073] Figure 4 This is a schematic diagram of the structure of an encoding model provided in this application. (Refer to...) Figure 4 As shown, the coding model includes L encoders C1 to CL cascaded sequentially. The input of the first encoder C1 serves as the input of the coding model, and the output of encoder C1 serves as the input of the next encoder C2. The output of the second encoder C2 serves as the input of the next encoder, and so on, until the last encoder CL, whose output serves as the output of the coding model.
[0074] from Figure 4 As can be seen, each encoder can include at least a self-attention layer and a position feedforward layer. The encoder's input serves as the input to the self-attention layer, the self-attention layer's output serves as the input to the position feedforward layer, and the position feedforward layer's output serves as the encoder's output. In some possible implementations, a post-processing layer can be included after the position feedforward layer to further improve model performance.
[0075] In the aforementioned self-attention layer, unlike conventional algorithms, this application replaces the similarity between the query matrix and the key matrix with the product of the nonlinear mapping tensor of the query tensor and the key tensor, in order to reduce computational complexity and thus improve the efficiency of state analysis of industrial equipment.
[0076] In the self-attention layer of this application, the encoder's input tensor is first mapped three times, including: mapping the encoder's input tensor to a query tensor through a query mapping tensor, mapping the encoder's input tensor to a key tensor through a key mapping tensor, and mapping the encoder's input tensor to a value tensor through a value mapping tensor; then, the output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, and the value tensor.
[0077] For the above three mappings, in one possible implementation, the product of the encoder's input tensor and the query mapping tensor is calculated to obtain the query tensor; the product of the encoder's input tensor and the key mapping tensor is calculated to obtain the key tensor; and the product of the encoder's input tensor and the value mapping tensor is calculated to obtain the value tensor.
[0078] In another possible implementation of the three mappings described above, the query mapping tensor, key mapping tensor, and value mapping tensor can all be referred to as mapping tensors. First, for each of these mapping tensors, the mapping tensor is decomposed into two chained tensors. The product of the two chained tensors equals the mapping tensor. The two chained tensors corresponding to the mapping tensor include a first chained tensor and a second chained tensor. Then, the encoder's input tensor is converted into an intermediate mapping tensor using each of the first chained tensors. Finally, the intermediate mapping tensors corresponding to each mapping tensor are mapped into query tensors, key tensors, and value tensors, respectively, using the second chained tensors of each mapping tensor. In this way, the mapping tensor forms a tensor ring. Tensor ring decomposition (TR) yields two chained tensors, achieving tensor lightweighting, reducing the size of the self-attention layer, and thus improving computational speed.
[0079] The query mapping tensor is decomposed into a first chained tensor and a second chained tensor, denoted as LZ1 and LZ2, respectively. The key mapping tensor is also decomposed into a first chained tensor and a second chained tensor, denoted as LZ3 and LZ4, respectively. The value mapping tensor is also decomposed into a first chained tensor and a second chained tensor, denoted as LZ5 and LZ6, respectively.
[0080] After obtaining multiple first-chain tensors, the encoder's input tensor is converted into an intermediate mapping tensor using each of these first-chain tensors. This includes: converting the encoder's input tensor into a middle-key mapping tensor (denoted as ZYZ1) using the first-chain tensor LZ3 corresponding to the key mapping tensor; converting the encoder's input tensor into a middle-key mapping tensor (denoted as ZYZ2) using the first-chain tensor LZ5 corresponding to the value mapping tensor; and converting the encoder's input tensor into a middle-key mapping tensor (denoted as ZYZ3). In other words, the key mapping tensor, query mapping tensor, and value mapping tensor each correspond to an intermediate mapping tensor.
[0081] After obtaining multiple intermediate mapping tensors as described above, the intermediate mapping tensors corresponding to the mapping tensors are mapped to query tensors, key tensors, and value tensors respectively through the second chain tensors of each mapping tensor. This includes: mapping the intermediate mapping tensor ZYZ1 corresponding to the query mapping tensor to a query tensor through the second chain tensor LZ2 of the query mapping tensor; mapping the middle key mapping tensor ZYZ2 corresponding to the key mapping tensor to a key tensor through the second chain tensor LZ4 of the key mapping tensor; and mapping the middle key mapping tensor ZYZ3 corresponding to the value mapping tensor to a value tensor through the second chain tensor LZ6 of the value mapping tensor.
[0082] The following example illustrates how a mapping tensor can be decomposed into two chained tensors. For a mapping tensor... It can achieve the following decomposition process:
[0083] ; (3)
[0084] Among them, the symbol " " indicates the multiplication of two tensors, , , n=1, 2,...,N, m=1, 2,...,M. It is a tensor ring. and These are respectively considered as the first chained tensor and the second chained tensor mentioned above. , Based on this, when the mapping process is Y=X3 When W, where X3 is the encoder's input tensor, W is the mapping tensor, and Y is the mapped query tensor or the aforementioned key tensor or value tensor, the mapping process implemented through the mapping tensor can be rewritten as the following formula:
[0085] ; (4)
[0086] In the conventional algorithm, the tensor ring mentioned above is calculated first. The tensor ring is a high-order tensor. Multiplying the encoder's input tensor by this tensor ring to obtain the mapping result tensor would lead to a large computational and storage load. However, this application splits the aforementioned tensor ring into two chained tensors. and Thus, during the mapping process, the first chain tensor is used first. The encoder's input tensor is transformed into an intermediate mapping tensor G, that is: the intermediate mapping tensor G is the encoder's input tensor X3 and the first chain tensor. The product of the two tensors is then passed through the second chain tensor. Transform the middle-bond mapping tensor G into the final mapping result tensor Y, that is: the mapping result tensor Y is the intermediate mapping tensor G and the second chain tensor. The product of these two factors. Thus, in devices with limited computing and storage capabilities, computational efficiency can be effectively improved.
[0087] After obtaining the query tensor, key tensor, and value tensor through the above mapping process, the output tensor of the self-attention layer can be generated. In one possible implementation, the product of the nonlinear mapping tensors corresponding to the query tensor and key tensor can be multiplied by the value tensor to obtain the output tensor of the self-attention layer. However, this algorithm has high computational complexity.
[0088] Considering that the softmax function of the mapping matrix in the Transformer model is usually low-rank, the computational complexity of key-value pairs can be reduced by using two low-rank tensors. In some possible implementations, firstly, the key tensor and value tensor are transformed into low-order key tensors and low-order value tensors respectively using low-rank tensors with rank less than or equal to a preset threshold. Then, the output tensor of the self-attention layer is generated based on the product of the query tensor, the nonlinear mapping tensors corresponding to the low-order key tensors, and the low-order value tensor.
[0089] Understandably, the key tensor and value tensor can be chosen to be the same or different low-rank tensors. The low-rank key tensor is the product of the key tensor K and its corresponding low-rank tensor E, denoted as: A low-order value tensor is the product of a value tensor V and its corresponding low-rank tensor F, denoted as: .when hour, .
[0090] After obtaining the aforementioned low-order key tensor and low-order value tensor, both can be substituted into the aforementioned formula (2) to obtain the self-attention calculation process using the query tensor, low-order key tensor, and low-order value tensor. However, in this application, the query tensor and low-order key tensor can be nonlinearly mapped, and self-attention calculation is performed using the product of the nonlinear query tensor and nonlinear key tensor corresponding to the query tensor and low-order key tensor, respectively, and the low-order value tensor.
[0091] In some possible implementations, the output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the low-order key tensor, respectively, and the low-order value tensor. This can include: performing nonlinear mapping on the query tensor and the low-order key tensor through a kernel function to map them into a nonlinear query tensor and a nonlinear key tensor, respectively; calculating the first product of the nonlinear query tensor and the nonlinear key tensor; and calculating the second product of the nonlinear query tensor, the nonlinear key tensor, and the low-order value tensor, and using the ratio of the second product to the first product as the output tensor of the self-attention layer.
[0092] Among them, the output tensor of the aforementioned self-attention layer This can be expressed by the following formula:
[0093] ; (5)
[0094] Where IM is an N×1 matrix, Through the first kernel function The nonlinear query tensor obtained by mapping the query tensor Q. Through the second kernel function low-order bond tensors The nonlinear bond tensor obtained by mapping.
[0095] It is understood that the above describes the calculation process of a single-head local attention layer. In this application, a multi-head attention layer can also be used. The input tensor input to the encoder is divided into h head tensors, where h is the number of heads. For each head tensor, the output tensor of the single-head self-attention layer is calculated according to the aforementioned single-head mapping algorithm. The output tensors corresponding to all head tensors are then concatenated and multiplied by a mapping tensor WO to obtain the output tensor of the multi-head self-attention layer. This high-order tensor-based attention network can achieve multi-scale time-series feature extraction.
[0096] After transforming the key tensor and value tensor using the low-rank tensor described above, the computational complexity can be reduced from... Reduce to After mapping the query tensor and low-order key tensor to nonlinear key tensors and nonlinear query tensors respectively using kernel functions, the computational complexity can be reduced from... Reduce to .
[0097] Figure 5 This is a data processing flowchart for a single-head self-attention layer provided in this application. (Refer to...) Figure 5 As shown, the input tensor of the self-attention layer is X3, which passes through three first chain tensors. , and These are mapped to three intermediate mapping tensors respectively. , and The three intermediate mapping tensors GQ, GK, and GV are respectively passed through three second-chain tensors. , and , respectively mapped to query tensors Key tensors Sum tensor Specifically, the two key tensors K and value tensor V are transformed into low-order key tensors through low-rank tensors E and F, respectively. and low-order value tensors Query tensor Q and lower-order key tensors, respectively, through kernel functions. and Nonlinear mapping to nonlinear query tensor and nonlinear bond tensor Finally, for the nonlinear query tensor... Nonlinear bond tensor and low-order value tensors The output tensor of the self-attention layer is calculated. .
[0098] After the self-attention layer processing described above, the tensor data needs to be processed by the position feedforward layer. This position feedforward layer is implemented using a multilayer perceptron (MLP), referred to as the first multilayer perceptron. In the position feedforward layer, the output tensor of the self-attention layer in this encoder undergoes a nonlinear mapping. This nonlinear mapping process specifically includes: firstly, mapping the tensor through the first mapping tensor... The output tensor of the self-attention layer is linearly mapped to obtain a self-attention linear tensor. Then, a nonlinear function (e.g., the ReLU function) is used to perform a nonlinear mapping on the self-attention linear tensor to obtain a self-attention nonlinear tensor. Finally, a second mapping tensor is used... The output tensor of the position feedforward layer is obtained by linearly mapping the self-attention nonlinear tensor.
[0099] The aforementioned first multilayer perceptron is used for position feedforward from multiple dimensions, mapping data from multiple dimensions to a single output data. An MLP consists of multiple neurons (or nodes), each connected to other neurons via weights. It is commonly used to solve problems such as classification and regression, and is one of the fundamental structures in deep learning. The computation process of an MLP can be represented by the following formula:
[0100] ; (6)
[0101] in, and These are two linear mapping tensors of the MLP. XZ is the input tensor of the MLP, and MLP(XZ) is the output tensor of the MLP. The ReLU function is a simple and effective activation function that achieves nonlinear mapping by truncating negative inputs to 0 and preserving the original values of positive inputs.
[0102] When MLP is the first multilayer perceptron, XZ is the output tensor of the self-attention layer in the encoder where the first multilayer perceptron is located, which is also the input tensor of the first multilayer perceptron. MLP(XZ) is the output tensor of the first multilayer perceptron. MLP(XZ) can also be used as the output tensor of the encoder or the input tensor of the layer after the first multilayer perceptron.
[0103] Additionally, when the MLP is a first-layer perceptron, and .
[0104] In some implementations, refer to Figure 4As shown, each of the above encoders may further include a post-processing layer located after the position feedforward layer. That is, the output tensor of the position feedforward layer serves as the input tensor of the post-processing layer in this encoder. In the post-processing layer, the output tensor of the position feedforward layer is normalized and nonlinearly processed to obtain the output tensor of the post-processing layer, which serves as the output tensor of this encoder.
[0105] Specifically, the processing procedure of the post-processing layer described above can be represented by the following formula:
[0106] ; (7)
[0107] Where X4 is the output tensor of the position feedforward layer, which is also the input tensor of the post-processing layer, and X5 is the output tensor of the post-processing layer. LayerNorm is a non-linear processing function, while LayerNorm is a normalization processing function, which can also be understood as a normalization processing function.
[0108] This application can optimize the performance of the coding network through normalization and residual connections in the post-processing layer described above.
[0109] S204. The state analysis results of industrial equipment are predicted based on the encoding result tensor by the second multilayer perceptron.
[0110] The second multilayer perceptron is used for state prediction. It is also an MLP (Multilayer Perceptron Language Processing), and the MLP formula can be found in the first multilayer perceptron description. When the encoded result tensor is obtained, the data tensor of the last moment in the data sequence is extracted for state prediction.
[0111] The aforementioned second multilayer perceptron is used to perform a nonlinear mapping on the data at the last moment to obtain the state analysis results. This nonlinear mapping process specifically includes: firstly, using the first mapping tensor... The data tensor at the last moment is linearly mapped to obtain a linear data tensor. Then, a nonlinear function (e.g., the ReLU function) is used to nonlinearly map the linear data tensor to obtain a nonlinear data tensor. Finally, a second mapping tensor is used... The tensor corresponding to the state analysis result is obtained by linearly mapping the nonlinear tensor of the data.
[0112] The calculation formula for the second multilayer perceptron is the same as that for the MLP in the first multilayer perceptron, except for the different parameters, which will not be repeated here. The first mapping tensor of the second multilayer perceptron... Second mapping tensor .
[0113] The prediction method for the lightweight edge model of the Industrial Internet provided in this application performs nonlinear mapping on the query tensor and the key tensor when extracting features through the self-attention layer. The similarity is replaced by the product of the nonlinear mapping tensor of the query tensor and the nonlinear mapping tensor of the key tensor, which reduces the computational complexity and improves the efficiency of state analysis.
[0114] Figure 6 This is a schematic diagram of the predictive device for the lightweight edge model of the Industrial Internet provided in this application, as shown below. Figure 6 As shown, the prediction device 400 for the lightweight edge model of the Industrial Internet provided in this embodiment includes:
[0115] The status data acquisition module 401 is used to acquire status data of industrial equipment. The status data is a data sequence collected by one or more sensors.
[0116] Tensor quantization module 402 is used to perform tensor quantization on state data to obtain state tensors, where the order of the state tensor is greater than the order of the state data (2).
[0117] The encoding module 403 is used to generate the encoded result tensor corresponding to the state tensor through the encoding model. The encoding model includes multiple encoders cascaded in sequence. Each encoder includes a self-attention layer and a position feedforward layer located after the self-attention layer. The position feedforward layer includes a first multilayer perceptron. In the self-attention layer, the input tensor of the encoder is mapped to a query tensor, a key tensor, and a value tensor through a query mapping tensor, a key mapping tensor, and a value mapping tensor. The output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, respectively, and the value tensor. In the position feedforward layer, the output tensor of the self-attention layer in this encoder is linearly mapped.
[0118] The prediction module 404 is used to predict the state analysis results of industrial equipment based on the encoding result tensor using a second multilayer perceptron.
[0119] In one possible implementation, the encoding module 403 is also used for:
[0120] By using low-rank tensors with rank less than or equal to a preset threshold, the key tensor and value tensor are transformed into low-order key tensors and low-order value tensors, respectively. Based on the product of the nonlinear mapping tensors corresponding to the query tensor and the low-order key tensor, and the low-order value tensor, the output tensor of the self-attention layer is generated.
[0121] In one possible implementation, the encoding module 403 is also used for:
[0122] The query tensor and low-order key tensor are mapped to nonlinear query tensor and nonlinear key tensor respectively using a kernel function; the first product of the nonlinear query tensor and nonlinear key tensor is calculated, and the second product of the nonlinear query tensor, nonlinear key tensor and low-order value tensor is calculated, and the ratio of the second product to the first product is used as the output tensor of the self-attention layer.
[0123] In one possible implementation, the encoding module 403 is also used for:
[0124] For each of the query mapping tensor, key mapping tensor, and value mapping tensor, the mapping tensor is decomposed into two chained tensors. The product of the two chained tensors equals the mapping tensor. The two chained tensors corresponding to the mapping tensor include a first chained tensor and a second chained tensor. The encoder's input tensor is converted into an intermediate mapping tensor through each first chained tensor. Each key mapping tensor, query mapping tensor, and value mapping tensor corresponds to an intermediate mapping tensor. The intermediate mapping tensors corresponding to each mapping tensor are mapped into query tensors, key tensors, and value tensors, respectively, through the second chained tensors of each mapping tensor.
[0125] In one possible implementation, the encoder also includes a post-processing layer located after the position feedforward layer. The encoding module 403 is further configured to normalize and nonlinearly process the output tensor of the position feedforward layer in the post-processing layer, and use it as the output tensor of the encoder.
[0126] In one possible implementation, the tensor quantization module 402 is also used for:
[0127] Obtain the number of sensors collecting state data and the length of the data sequence collected by each sensor; convert the number of sensors into the product of J first values and the length of the data sequence into the product of I second values, where I and J are both positive integers greater than or equal to 2; tensor the state data into an I+J order state tensor based on the J first values and I second values, where the J first values and I second values are both the length of the data sequence of the state tensor in the corresponding dimension.
[0128] The prediction device for the lightweight edge model of the industrial Internet provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0129] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. For example... Figure 6 As shown, the electronic device 600 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the electronic device 600 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.
[0130] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.
[0131] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0132] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.
[0133] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0134] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0135] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0136] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0137] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0138] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0139] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0141] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0142] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0143] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0144] Finally, it should be noted that other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A prediction method of an industrial internet lightweight edge model, characterized in that, include: Acquire status data of industrial equipment, wherein the status data is a data sequence collected by one or more sensors, including an image sensor, and the status data includes images; The state data is subjected to tensor quantization to obtain a state tensor. The order of the state data is 2, and the order of the state tensor is greater than that of the state data. The encoding result tensor corresponding to the state tensor is generated through an encoding model. The encoding model includes multiple encoders cascaded in sequence. Each encoder includes a self-attention layer and a position feedforward layer located after the self-attention layer. The position feedforward layer includes a first multilayer perceptron. In the self-attention layer, the input tensor of the encoder is mapped to a query tensor, a key tensor, and a value tensor through a query mapping tensor, a key mapping tensor, and a value mapping tensor. The output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, respectively, and the value tensor. A post-processing layer is also included after the position feedforward layer. The state analysis results of the industrial equipment are predicted by a second multilayer perceptron based on the tensor of the encoded results. The step of generating the output tensor of the self-attention layer based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, respectively, and the value tensor includes: The key tensor and the value tensor are transformed into low-order key tensors and low-order value tensors respectively using low-rank tensors with rank less than or equal to a preset threshold. The query tensor and the low-order key tensor are mapped to nonlinear query tensors and nonlinear key tensors respectively using kernel functions; Calculate the first product of the nonlinear query tensor and the nonlinear key tensor, and calculate the second product of the nonlinear query tensor, the nonlinear key tensor and the low-order value tensor, and use the ratio of the second product to the first product as the output tensor of the self-attention layer.
2. The method of claim 1, wherein, The step of mapping the encoder's input tensor to query tensors, key tensors, and value tensors through query mapping tensors, key mapping tensors, and value mapping tensors includes: For each of the query mapping tensor, the key mapping tensor, and the value mapping tensor, the mapping tensor is decomposed into two chain tensors. The product of the two chain tensors is equal to the mapping tensor. The two chain tensors corresponding to the mapping tensor include a first chain tensor and a second chain tensor. The input tensor of the encoder is converted into an intermediate mapping tensor by each of the first chain tensors, and the key mapping tensor, the query mapping tensor and the value mapping tensor each correspond to an intermediate mapping tensor; Through the second chain tensor of each of the mapping tensors, the intermediate mapping tensors corresponding to the mapping tensors are respectively mapped to the query tensor, the key tensor, and the value tensor.
3. The method according to claim 1 or 2, characterized in that, The encoder further includes: normalizing and nonlinearizing the output tensor of the position feedforward layer in the post-processing layer, and using it as the output tensor of the encoder.
4. The method according to claim 1 or 2, characterized in that, The process of tensorizing the state data to obtain the state tensor includes: Obtain the number of sensors collecting the status data and the length of the data sequence collected by each sensor; The number of sensors is converted into the product of J first values, and the length of the data sequence is converted into the product of I second values, where I and J are both positive integers greater than or equal to 2; Based on the J first values and the I second values, the state data is tensored into an I+J order state tensor, where the J first values and the I second values are both the data sequence lengths of the state tensor in the corresponding dimensions. 5.A prediction device of an industrial internet lightweight edge model, characterized in that, include: A status data acquisition module is used to acquire status data of industrial equipment. The status data is a data sequence collected by one or more sensors, including an image sensor, and the status data includes images. The tensor quantization module is used to perform tensor quantization processing on the state data to obtain a state tensor. The order of the state data is 2, and the order of the state tensor is greater than the order of the state data. An encoding module is used to generate an encoded result tensor corresponding to the state tensor through an encoding model. The encoding model includes multiple encoders cascaded in sequence. Each encoder includes a self-attention layer and a position feedforward layer located after the self-attention layer. The position feedforward layer includes a first multilayer perceptron. In the self-attention layer, the input tensor of the encoder is mapped to a query tensor, a key tensor, and a value tensor through a query mapping tensor, a key mapping tensor, and a value mapping tensor. The output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, respectively, and the value tensor. In the position feedforward layer, the output tensor of the self-attention layer in this encoder is linearly mapped. A post-processing layer is also included after the position feedforward layer. The prediction module is used to predict the state analysis results of the industrial equipment based on the encoding result tensor using a second multilayer perceptron. The encoding module is specifically used to convert the key tensor and the value tensor into low-order key tensors and low-order value tensors respectively using low-rank tensors with rank less than or equal to a preset threshold. The query tensor and the low-order key tensor are mapped to nonlinear query tensors and nonlinear key tensors respectively using kernel functions; Calculate the first product of the nonlinear query tensor and the nonlinear key tensor, and calculate the second product of the nonlinear query tensor, the nonlinear key tensor and the low-order value tensor, and use the ratio of the second product to the first product as the output tensor of the self-attention layer.
6. An electronic device, comprising: include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-4.
8. A computer program product, characterised in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-4.