Prediction method of industrial internet lightweight edge model and related equipment

Abstract

The invention provides an industrial internet lightweight edge model prediction method and related equipment, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring state data of industrial equipment; performing tensorization on the state data to obtain a state tensor; generating an encoding result tensor of the state tensor by an encoding model, the encoding model comprising a plurality of encoders, the encoders comprising a self-attention layer in which the input tensor is mapped into a query tensor, a key tensor, and a value tensor by a query mapping tensor, a key mapping tensor, and a value mapping tensor, and a position feed-forward layer in which the input tensor is mapped into the query tensor, the key tensor, and the value tensor; generating an output tensor of the self-attention layer based on the product of the query tensor and the nonlinear mapping tensor of the key tensor and the value tensor; and predicting a state analysis result of the industrial equipment based on the coding result tensor through the second multi-layer perceptron. According to the method, the product of the nonlinear mapping tensor of the query tensor and the nonlinear mapping tensor of the key tensor is used for replacing the similarity, so that the calculation complexity is reduced, and the state analysis efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence, and in particular to a prediction method of an industrial internet lightweight edge model and related equipment. BACKGROUND

[0002] In an industrial production process, the state of an industrial equipment directly affects the progress of the industrial production. Therefore, the state of the industrial equipment can be continuously detected during the operation of the industrial equipment, so as to predict whether the industrial equipment is abnormal in advance, and thus corresponding processing can be performed in advance. For example, state data of the industrial equipment is collected during the operation of the industrial equipment, and the state data is analyzed to obtain a state analysis result of the industrial equipment.

[0003] In the prior art, an artificial intelligence model can be used to analyze the state data to obtain the state analysis result of the industrial equipment. A commonly used artificial intelligence model includes a Transformer model, which can perform attention extraction on the state data and generate a prediction result.

[0004] However, the Transformer model has a low calculation efficiency, which leads to a low state analysis efficiency of the industrial equipment. SUMMARY

[0005] Embodiments of the present application provide a prediction method of an industrial internet lightweight edge model and related equipment to improve the state analysis efficiency of the industrial equipment.

[0006] In a first aspect, embodiments of the present application provide a prediction method of an industrial internet lightweight edge model, comprising:

[0007] obtaining state data of an industrial equipment, the state data being a data sequence collected by one or more sensors; performing tensorization processing on the state data to obtain a state tensor, an order of the state tensor being greater than 2 of an order of the state data; generating an encoding result tensor corresponding to the state tensor through an encoding model, the encoding model including a plurality of encoding layers connected in sequence, each encoding layer including a self-attention layer and a positional feedforward layer located after the self-attention layer, the positional feedforward layer including a first multi-layer perceptron, and in the self-attention layer, an input tensor of the encoding layer is mapped into a query tensor, a key tensor and a value tensor through a query mapping tensor, a key mapping tensor and a value mapping tensor, and an output tensor of the self-attention layer is generated based on a product of a nonlinear mapping tensor corresponding to the query tensor and the key tensor respectively and the value tensor; and predicting a state analysis result of the industrial equipment based on the encoding result tensor through a second multi-layer perceptron.

[0008] In a possible implementation, the output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensor corresponding to the query tensor and the key tensor respectively and the value tensor, comprising:

[0009] The key tensor and the value tensor are respectively converted into a low-order key tensor and a low-order value tensor by a low-rank tensor with a rank less than or equal to a preset threshold; and an output tensor of the self-attention layer is generated based on a product of a nonlinear mapping tensor corresponding to the query tensor and the low-order key tensor, and the low-order value tensor.

[0010] In a possible implementation, the output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensor corresponding to the query tensor and the low-order key tensor, and the low-order value tensor, including:

[0011] The query tensor and the low-order key tensor are respectively mapped into a nonlinear query tensor and a nonlinear key tensor by a kernel function; a first product of the nonlinear query tensor and the nonlinear key tensor is calculated, and a second product of the nonlinear query tensor, the nonlinear key tensor and the low-order value tensor is calculated, and a ratio of the second product to the first product is taken as the output tensor of the self-attention layer.

[0012] In a possible implementation, the input tensor of the encoder is mapped into the query tensor, the key tensor and the value tensor by the query mapping tensor, the key mapping tensor and the value mapping tensor, including:

[0013] For each of the query mapping tensor, the key mapping tensor and the value mapping tensor, the mapping tensor is decomposed into two chain-shaped tensors, and a product of the two chain-shaped tensors is equal to the mapping tensor, the two chain-shaped tensors corresponding to the mapping tensor including a first chain-shaped tensor and a second chain-shaped tensor; the input tensor of the encoder is respectively converted into an intermediate mapping tensor by each first chain-shaped tensor, and the key mapping tensor, the query mapping tensor and the value mapping tensor respectively correspond to one intermediate mapping tensor; the intermediate mapping tensor corresponding to the mapping tensor is respectively mapped into the query tensor, the key tensor and the value tensor by the second chain-shaped tensor of the mapping tensor.

[0014] In a possible implementation, the encoder further includes: an post-processing layer located after the position feedforward layer, in which the output tensor of the position feedforward layer is normalized and nonlinearly processed as an output tensor of the encoder.

[0015] In a possible implementation, the state data is tensorized to obtain a state tensor, including:

[0016] The number of sensors collecting the state data and the length of a data sequence collected by each sensor are obtained; the number of sensors is converted into a product of J first numerical values, and the length of the data sequence is converted into a product of I second numerical values, I and J are both positive integers greater than or equal to 2; and the state data is tensorized into a state tensor of I+J order according to the J first numerical values and the I second numerical values, the J first numerical values and the I second numerical values are both the length of a data sequence in a corresponding dimension of the state tensor.

[0017] In a second aspect, the embodiments of the present application provide a prediction device of an industrial internet lightweight edge model, comprising:

[0018] a state data acquisition module configured to acquire state data of an industrial device, the state data being a data sequence collected by one or more sensors.

[0019] a tensorization module configured to perform tensorization on the state data to obtain a state tensor, an order of the state tensor being greater than 2 of an order of the state data.

[0020] an encoding module configured to generate an encoding result tensor corresponding to the state tensor by an encoding model, the encoding model comprising a plurality of encoding layers connected in sequence, each encoding layer comprising a self-attention layer and a positional feedforward layer located after the self-attention layer, the positional feedforward layer comprising a first multi-layer perceptron, in the self-attention layer, an input tensor of the encoding layer is mapped into a query tensor, a key tensor and a value tensor by a query mapping tensor, a key mapping tensor and a value mapping tensor, and an output tensor of the self-attention layer is generated based on a product of a nonlinear mapping tensor corresponding to the query tensor and the key tensor respectively and the value tensor, and in the positional feedforward layer, the output tensor of the self-attention layer in the current encoding layer is linearly mapped.

[0021] a prediction module configured to predict a state analysis result of the industrial device based on the encoding result tensor by a second multi-layer perceptron.

[0022] In a possible implementation, the encoding module is further configured to:

[0023] transform the key tensor and the value tensor into a low-order key tensor and a low-order value tensor respectively by a low-rank tensor with a rank less than or equal to a preset threshold, and generate the output tensor of the self-attention layer based on a product of nonlinear mapping tensors corresponding to the query tensor and the low-order key tensor respectively and the low-order value tensor.

[0024] In a possible implementation, the encoding module is further configured to:

[0025] map the query tensor and the low-order key tensor into a nonlinear query tensor and a nonlinear key tensor respectively by a kernel function, calculate a first product of the nonlinear query tensor and the nonlinear key tensor, and calculate a second product of the nonlinear query tensor, the nonlinear key tensor and the low-order value tensor, and take a ratio of the second product to the first product as the output tensor of the self-attention layer.

[0026] In a possible implementation, the encoding module is further configured to:

[0027] For each of the query mapping tensor, the key mapping tensor and the value mapping tensor, the mapping tensor is decomposed into two chain-shaped tensors, the product of the two chain-shaped tensors is equal to the mapping tensor, and the two chain-shaped tensors corresponding to the mapping tensor include a first chain-shaped tensor and a second chain-shaped tensor; the input tensor of the encoder is converted into an intermediate mapping tensor through each first chain-shaped tensor, and the query mapping tensor, the key mapping tensor and the value mapping tensor correspond to an intermediate mapping tensor respectively; the intermediate mapping tensor corresponding to the mapping tensor is mapped into the query tensor, the key tensor and the value tensor through the second chain-shaped tensor of the mapping tensor.

[0028] In a possible implementation, the encoder further includes a post-processing layer located after the position feedforward layer, and the encoding module is further configured to: perform normalization processing and nonlinear processing on the output tensor of the position feedforward layer at the post-processing layer, as the output tensor of the encoder.

[0029] In a possible implementation, the tensorization module is further configured to:

[0030] obtain the number of sensors for collecting state data and the length of a data sequence collected by each sensor; convert the number of sensors into a product of J first numerical values, and convert the length of the data sequence into a product of I second numerical values, I and J are both positive integers greater than or equal to 2; and according to the J first numerical values and the I second numerical values, tensorize the state data into an I+J order state tensor, the J first numerical values and the I second numerical values are both the length of a data sequence in a corresponding dimension of the state tensor.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0032] The memory stores computer execution instructions;

[0033] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and the computer program is executed by the processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0036] The prediction method of the industrial internet lightweight edge model and the related device provided by the embodiment of the present application, when extracting features through the self-attention layer, the query tensor and the key tensor are nonlinearly mapped, so as to replace the similarity by the product of the nonlinear mapping tensor of the query tensor and the nonlinear mapping tensor of the key tensor, thereby reducing the calculation complexity and improving the state analysis efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] Figure 1 is an application scenario schematic diagram provided by the present application;

[0039] Figure 2 is a data processing flowchart of an industrial internet lightweight edge model provided by the present application;

[0040] Figure 3 is a step flowchart of the prediction method of the industrial internet lightweight edge model provided by the present application;

[0041] Figure 4 is a structure schematic diagram of an encoding model provided by the present application;

[0042] Figure 5 is a data processing flowchart of a single-head self-attention layer provided by the present application;

[0043] Figure 6 is a structure schematic diagram of a prediction device of an industrial internet lightweight edge model provided by the present application;

[0044] Figure 7 is a structure schematic diagram of an electronic device provided by the present application.

[0045] Through the above-mentioned drawings, the specific embodiments of the present application have been shown, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0046] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. Unless otherwise indicated, the same numbers on different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present application.

[0047] The present application is used for state analysis of industrial equipment, Figure 1 is an application scenario schematic diagram provided by the present application,Figure 1 As shown, one or more sensors can be arranged on or around the industrial equipment in the industrial scene, which are used to collect state data of different positions of the industrial equipment. These sensors are in communication connection with the electronic device, and transmit the collected state data to the electronic device, so that the electronic device analyzes the state of the industrial equipment according to the state data.

[0048] It should be noted that, Figure 1 Four sensors are exemplarily shown, but in actual scenarios, the number and types of sensors can be flexibly set according to actual needs, and these sensors can be sensors of the same type and / or sensors of different types, and can be arranged at any position of the industrial equipment. The sensors can be position sensors, image sensors, laser sensors, etc., and the type of the sensor is not limited in the present application.

[0049] The above-mentioned electronic device can be any device with computing capability, such as a computer, a mobile phone, a tablet, a server, etc.

[0050] It can be understood that an artificial intelligence model can be arranged on the above-mentioned electronic device to realize state analysis. The artificial intelligence model herein can generally use a Transformer model, which includes a self-attention mechanism. The conventional self-attention mechanism is realized by the following formula:

[0051] ; (1)

[0052] Wherein, is the output matrix of the self-attention mechanism, D is the scaling factor, is the transpose matrix of K. Q, K and V are respectively the query matrix, the key matrix and the value matrix obtained by linear transformation of the input matrix X of the self-attention mechanism, , , Wherein, , and are respectively the query mapping matrix, the key mapping matrix and the value mapping matrix. The softmax function is a normalized exponential function, which is used to normalize each element item in a matrix or vector, and is a function known in the artificial intelligence field, which will not be described here.

[0053] Based on the above formula (1), the i-th row of the output matrix of the self-attention mechanism As follows:

[0054] ; (2)

[0055] Wherein, N is the column number of the input matrix X, that is, the length of each row. The similarity function is used to represent a softmax function. is the i-th row of the query matrix Q, is the j-th column of the key matrix K, is the j-th column of the value matrix V. For the above formula (1), two layers of loops are required to calculate each query vector , and each key vector and value vector corresponding to the new representation, and thus the computational complexity is It can be seen that the computational complexity of the conventional Transformer self-attention mechanism is high, which leads to a large amount of state analysis calculation of the industrial equipment, thereby resulting in low analysis efficiency.

[0056] To solve the above technical problems, the present application considers replacing the above similarity calculation process with the product of two nonlinear mapping tensors to linearize the similarity, which helps to effectively reduce the computational complexity and further improve the state analysis efficiency of the industrial equipment.

[0057] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0058] Figure 2 is a data processing flow diagram provided by the present application, referring to Figure 2 , the data processing flow of the present application involves three modules: a preprocessing module, an encoding model, and a second multilayer perceptron. The input of the preprocessing module is state data, the output of the preprocessing module is used as the input of the encoding model, the output of the encoding model is used as the input of the second multilayer perceptron, and the output of the second multilayer perceptron is the state output result of the industrial equipment. In the data processing flow of the present application, first, the state data collected by the sensor for the industrial equipment is processed by the preprocessing module to obtain a state tensor; then, the state tensor is input into the encoding model to obtain an encoding result tensor; finally, the second multilayer perceptron analyzes the encoding result tensor to obtain a state analysis result.

[0059] In some possible implementation manners, referring to Figure 2 , the above preprocessing module can specifically include a position encoding unit and a tensorization unit. The position encoding unit is used to add position information to the state data, and the tensorization unit is used to tensorize the state data with the added position information to obtain a state tensor.

[0060] Figure 3is a step flowchart of an industrial internet lightweight edge model prediction method provided by the present application, executed by an electronic device. Referring to Figure 3 The industrial internet lightweight edge model prediction method includes the following steps:

[0061] S201, obtaining state data of an industrial device, the state data being a data sequence collected by one or more sensors.

[0062] The state data is any data used to represent the running state of the industrial device, and its form is related to the sensor. When the sensor is an image sensor, the state data can be an image, and when the sensor is a position sensor, the state data can be position data.

[0063] Specifically, the state data can be collected by the sensor and sent to the electronic device through wireless or wired communication. It can be understood that the state data can be collected by the sensor in real time and sent to the electronic device in real time, or sent to the electronic device at regular intervals, or the state data is stored in a database by the sensor so that the electronic device reads the state data from the database. The present application does not limit the sending mode of the state data.

[0064] The state data can include multiple data sequences, each data sequence corresponding to data collected by a sensor at multiple times. It can be understood that each sensor continuously collects data sequences, and after the electronic device obtains the data sequences, it can continuously obtain data sequences for state analysis by sliding window from the data sequences in the last period of time. Each sensor corresponds to a data sequence, and multiple sensors correspond to multiple data sequences. For example, when each data sequence has a length of N and the number of sensors is M, M state data with a length of N can be obtained, that is, state data X1(X1 ) is an N x M matrix, and the state data X1 is used to execute the subsequent step S202.

[0065] S202, performing tensorization processing on the state data to obtain a state tensor, the order of the state tensor being greater than 2.

[0066] The tensorization is used to convert data from a low-order form to a high-order form. The state data is a matrix, which can be understood as 2-order data, and the state tensor is a tensor with an order greater than 2.

[0067] The tensorization processing can be an arbitrary tensor algorithm. In some embodiments, in order to more accurately represent state data and avoid data loss, the state tensor can be generated through the following process. First, the number of sensors collecting state data and the data sequence length collected by each sensor are obtained. Then, the number of sensors is converted into the product of J first values, and the data sequence length is converted into the product of I second values, I and J are positive integers greater than or equal to 2. Finally, according to the J first values and the I second values, the state data is tensorized into an I+J order state tensor, and the J first values and the I second values are the data sequence length of the state tensor in the corresponding dimension.

[0068] The number of sensors can be sent to the electronic device in advance or input in advance in the electronic device, and the data sequence length is set according to actual needs. The number of sensors The data sequence length Therefore, the state tensor , The i-th dimension of the state tensor contains the data of the time sequence with a step of and a total length of .

[0069] In some embodiments, as shown in FIG. 2, before the above-mentioned tensorization processing is performed by the tensorization unit, the tensorization unit can also add position information to the state data, so that the position of each element in the state data is included in the state data. In this way, the subsequent encoding model can more accurately analyze the state. Figure 2

[0070] Unlike the matrix operation in the conventional human ability intelligent model, based on the above-mentioned tensorization processing, the operation processing performed by the encoding model and the second multi-layer perception in the present application are all tensor operations.

[0071] S203, generate an encoding result tensor corresponding to the state tensor through an encoding model, the encoding model includes a plurality of encoding units connected in sequence, each encoding unit includes a self-attention layer and a position feedforward layer located after the self-attention layer, the position feedforward layer includes a first multi-layer perception, in the self-attention layer, the input tensor of the encoding unit is mapped into a query tensor, a key tensor and a value tensor through a query mapping tensor, a key mapping tensor and a value mapping tensor, and the output tensor of the self-attention layer is generated based on the product of the corresponding nonlinear mapping tensors of the query tensor and the key tensor, and the value tensor.

[0072] ​In the cascaded encoder, the output of each encoder is taken as the input of the next level encoder, that is, the input of the first encoder is taken as the input of the encoding model, the output of the last encoder is taken as the output of the encoding model, and for the intermediate encoder, the input is the output of the previous level encoder, and the output is the input of the next level encoder.

[0073] Figure 4 is a structural schematic diagram of an encoding model provided by the present application. As shown in Figure 4 , the encoding model includes L encoders C1 to CL which are sequentially cascaded. The input of the first encoder C1 is taken as the input of the encoding model, the output of the encoder C1 is taken as the input of the next level encoder C2, the output of the second encoder C2 is taken as the input of the next level encoder, and so on, until the last encoder CL, whose output is taken as the output of the encoding model.

[0074] As can be seen from Figure 4 , each encoder can at least include a self-attention layer and a position feedforward layer. The input of the encoder is taken as the input of the self-attention layer, the output of the self-attention layer is taken as the input of the position feedforward layer, and the output of the position feedforward layer is taken as the output of the encoder. In some possible implementation manners, a post-processing layer can also be included after the position feedforward layer to further improve the performance of the model.

[0075] In the above self-attention layer, unlike the conventional algorithm, the present application replaces the similarity between the query matrix and the key matrix by the product of the nonlinear mapping tensor of the query tensor and the key tensor to reduce the computational complexity and further improve the state analysis efficiency of the industrial equipment.

[0076] In the self-attention layer of the present application, the input tensor of the encoder is first mapped three times, including: mapping the input tensor of the encoder into a query tensor by a query mapping tensor, mapping the input tensor of the encoder into a key tensor by a key mapping tensor, and mapping the input tensor of the encoder into a value tensor by a value mapping tensor; then, based on the product of the corresponding nonlinear mapping tensors of the query tensor and the key tensor, and the value tensor, an output tensor of the self-attention layer is generated.

[0077] For the above three mappings, in one possible implementation manner, the product of the input tensor of the encoder and the query mapping tensor is calculated to obtain the query tensor, the product of the input tensor of the encoder and the key mapping tensor is calculated to obtain the key tensor, and the product of the input tensor of the encoder and the value mapping tensor is calculated to obtain the value tensor.

[0078] For the three mappings described above, in another possible implementation, the query mapping tensor, the key mapping tensor and the value mapping tensor can all be referred to as a mapping tensor. First, for each of the query mapping tensor, the key mapping tensor and the value mapping tensor, the mapping tensor is decomposed into two chain-shaped tensors, the product of the two chain-shaped tensors is equal to the mapping tensor, and the two chain-shaped tensors corresponding to the mapping tensor include a first chain-shaped tensor and a second chain-shaped tensor; then, the input tensor of the encoder is converted into an intermediate mapping tensor through each first chain-shaped tensor; finally, the intermediate mapping tensor corresponding to the mapping tensor is mapped into a query tensor, a key tensor and a value tensor through the second chain-shaped tensor of each mapping tensor. In this way, the mapping tensor is a tensor ring, and the two chain-shaped tensors are obtained through tensor ring decomposition (TR), which realizes tensor lightweight and reduces the size of the self-attention layer, thereby improving the calculation speed.

[0079] wherein the query mapping tensor is decomposed into a first chain-shaped tensor and a second chain-shaped tensor, denoted as LZ1 and LZ2 respectively. The key mapping tensor is also decomposed into a first chain-shaped tensor and a second chain-shaped tensor, denoted as LZ3 and LZ4 respectively. The value mapping tensor is also decomposed into a first chain-shaped tensor and a second chain-shaped tensor, denoted as LZ5 and LZ6 respectively.

[0080] After obtaining the plurality of first chain-shaped tensors, the input tensor of the encoder is converted into an intermediate mapping tensor through each first chain-shaped tensor, including: the input tensor of the encoder is converted into an intermediate mapping tensor through the first chain-shaped tensor LZ1 corresponding to the query mapping tensor, denoted as ZYZ1, the input tensor of the encoder is converted into an intermediate mapping tensor through the first chain-shaped tensor LZ3 corresponding to the key mapping tensor, denoted as ZYZ2, and the input tensor of the encoder is converted into an intermediate mapping tensor through the first chain-shaped tensor LZ5 corresponding to the value mapping tensor, denoted as ZYZ3. That is, the key mapping tensor, the query mapping tensor and the value mapping tensor correspond to an intermediate mapping tensor respectively.

[0081] After obtaining the plurality of intermediate mapping tensors described above, the intermediate mapping tensor corresponding to the mapping tensor is mapped into a query tensor, a key tensor and a value tensor through the second chain-shaped tensor of each mapping tensor, including: the intermediate mapping tensor ZYZ1 corresponding to the query mapping tensor is mapped into a query tensor through the second chain-shaped tensor LZ2 of the query mapping tensor, the intermediate mapping tensor ZYZ2 corresponding to the key mapping tensor is mapped into a key tensor through the second chain-shaped tensor LZ4 of the key mapping tensor, and the intermediate mapping tensor ZYZ3 corresponding to the value mapping tensor is mapped into a value tensor through the second chain-shaped tensor LZ6 of the value mapping tensor.

[0082] The following will be described taking an example of decomposing a mapping tensor into two chain-shaped tensors. For a mapping tensor which can implement the following decomposition process:

[0083] ; (3)

[0084] wherein the symbol "·" represents the multiplication of two tensors, , n = 1, 2,..., N, m = 1, 2,..., M. is a tensor ring, and are respectively the aforementioned first chain tensor and the second chain tensor, , Based on this, when the mapping process is Y = X3 W, wherein X3 is the input tensor of the encoder, W is the mapping tensor, and Y is the query tensor or the key tensor or the value tensor obtained by mapping, the mapping process implemented by the mapping tensor is rewritten as the following formula:

[0085] ; (4)

[0086] In the conventional algorithm, the aforementioned tensor ring is calculated first, which is a high-order tensor, and then the input tensor of the encoder is multiplied by the tensor ring to obtain the mapping result tensor, which will result in large calculation and storage. However, the present application splits the aforementioned tensor ring into two chain tensors and , so that in the mapping process, the input tensor of the encoder is first converted into an intermediate mapping tensor G by the first chain tensor , that is, the intermediate mapping tensor G is the product of the input tensor X3 of the encoder and the first chain tensor , and then the intermediate mapping tensor G is converted into the final mapping result tensor Y by the second chain tensor , that is, the mapping result tensor Y is the product of the intermediate mapping tensor G and the second chain tensor . In this way, the operation efficiency can be effectively improved in devices with weak calculation and storage capabilities.

[0087] After obtaining the query tensor, the key tensor and the value tensor through the aforementioned mapping process, the output tensor of the self-attention layer can be generated. In one possible implementation, the product of the non-linear mapping tensor corresponding to the query tensor and the key tensor is multiplied by the value tensor as the output tensor of the self-attention layer. However, the calculation complexity of this algorithm is high.

[0088] ​Considering that the softmax function of the mapping matrix in the Transformer model is usually low-rank, the computational complexity of key-value pairs can be reduced by using two low-rank tensors. In some possible implementations, firstly, the key tensor and value tensor are transformed into low-order key tensors and low-order value tensors respectively using low-rank tensors with rank less than or equal to a preset threshold. Then, the output tensor of the self-attention layer is generated based on the product of the query tensor, the nonlinear mapping tensors corresponding to the low-order key tensors, and the low-order value tensor.

[0089] Understandably, the key tensor and value tensor can be chosen to be the same or different low-rank tensors. The low-rank key tensor is the product of the key tensor K and its corresponding low-rank tensor E, denoted as: A low-order value tensor is the product of a value tensor V and its corresponding low-rank tensor F, denoted as: .when hour, .

[0090] After obtaining the aforementioned low-order key tensor and low-order value tensor, both can be substituted into the aforementioned formula (2) to obtain the self-attention calculation process using the query tensor, low-order key tensor, and low-order value tensor. However, in this application, the query tensor and low-order key tensor can be nonlinearly mapped, and self-attention calculation is performed using the product of the nonlinear query tensor and nonlinear key tensor corresponding to the query tensor and low-order key tensor, respectively, and the low-order value tensor.

[0091] In some possible implementations, the output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the low-order key tensor, respectively, and the low-order value tensor. This can include: performing nonlinear mapping on the query tensor and the low-order key tensor through a kernel function to map them into a nonlinear query tensor and a nonlinear key tensor, respectively; calculating the first product of the nonlinear query tensor and the nonlinear key tensor; and calculating the second product of the nonlinear query tensor, the nonlinear key tensor, and the low-order value tensor, and using the ratio of the second product to the first product as the output tensor of the self-attention layer.

[0092] Among them, the output tensor of the aforementioned self-attention layer This can be expressed by the following formula:

[0093] ; (5)

[0094] Where IM is an N×1 matrix, Through the first kernel function The nonlinear query tensor obtained by mapping the query tensor Q. Through the second kernel function low-order bond tensors The nonlinear bond tensor obtained by mapping.

[0095] It is understood that the above describes the calculation process of a single-head local attention layer. In this application, a multi-head attention layer can also be used. The input tensor input to the encoder is divided into h head tensors, where h is the number of heads. For each head tensor, the output tensor of the single-head self-attention layer is calculated according to the aforementioned single-head mapping algorithm. The output tensors corresponding to all head tensors are then concatenated and multiplied by a mapping tensor WO to obtain the output tensor of the multi-head self-attention layer. This high-order tensor-based attention network can achieve multi-scale time-series feature extraction.

[0096] After transforming the key tensor and value tensor using the low-rank tensor described above, the computational complexity can be reduced from... Reduce to After mapping the query tensor and low-order key tensor to nonlinear key tensors and nonlinear query tensors respectively using kernel functions, the computational complexity can be reduced from... Reduce to .

[0097] Figure 5 This is a data processing flowchart for a single-head self-attention layer provided in this application. (Refer to...) Figure 5 As shown, the input tensor of the self-attention layer is X3, which passes through three first chain tensors. , and These are mapped to three intermediate mapping tensors respectively. , and The three intermediate mapping tensors GQ, GK, and GV are respectively passed through three second-chain tensors. , and , respectively mapped to query tensors Key tensors Sum tensor Specifically, the two key tensors K and value tensor V are transformed into low-order key tensors through low-rank tensors E and F, respectively. and low-order value tensors Query tensor Q and lower-order key tensors, respectively, through kernel functions. and Nonlinear mapping to nonlinear query tensor and nonlinear bond tensor Finally, for the nonlinear query tensor... Nonlinear bond tensor and low-order value tensors The output tensor of the self-attention layer is calculated. .

[0098] After the processing of the self-attention layer described above, the tensor data is processed by a positional feed-forward layer. The positional feed-forward layer is implemented by a multi-layer perceptron (MLP), referred to as a first multi-layer perceptron. The output tensor of the self-attention layer in the current encoder is non-linearly mapped in the positional feed-forward layer. The non-linear mapping process can specifically include: first, linearly mapping the output tensor of the self-attention layer to obtain a tensor referred to as a self-attention linear tensor, then non-linearly mapping the self-attention linear tensor by a non-linear function (for example, a RELU function) to obtain a self-attention non-linear tensor; and finally, linearly mapping the self-attention non-linear tensor to obtain an output tensor of the positional feed-forward layer. After the processing of the self-attention layer described above, the tensor data is processed by a positional feed-forward layer. The positional feed-forward layer is implemented by a multi-layer perceptron (MLP), referred to as a first multi-layer perceptron. The output tensor of the self-attention layer in the current encoder is non-linearly mapped in the positional feed-forward layer. The non-linear mapping process can specifically include: first, linearly mapping the output tensor of the self-attention layer to obtain a tensor referred to as a self-attention linear tensor, then non-linearly mapping the self-attention linear tensor by a non-linear function (for example, a RELU function) to obtain a self-attention non-linear tensor; and finally, linearly mapping the self-attention non-linear tensor to obtain an output tensor of the positional feed-forward layer. After the processing of the self-attention layer described above, the tensor data is processed by a positional feed-forward layer. The positional feed-forward layer is implemented by a multi-layer perceptron (MLP), referred to as a first multi-layer perceptron. The output tensor of the self-attention layer in the current encoder is non-linearly mapped in the positional feed-forward layer. The non-linear mapping process can specifically include: first, linearly mapping the output tensor of the self-attention layer to obtain a tensor referred to as a self-attention linear tensor, then non-linearly mapping the self-attention linear tensor by a non-linear function (for example, a RELU function) to obtain a self-attention non-linear tensor; and finally, linearly mapping the self-attention non-linear tensor to obtain an output tensor of the positional feed-forward layer.

[0099] The first multi-layer perceptron is used for positional feed-forward from multiple dimensions, and maps data of multiple dimensions to single output data. The MLP is composed of multiple neurons (or nodes), and each neuron is connected to other neurons through weights. It is commonly used to solve classification, regression and other problems, and is one of the basic structures in deep learning. The calculation process of the MLP can be represented by the following formula:

[0100] (6)

[0101] wherein and are two linear mapping tensors of the MLP. XZ is an input tensor of the MLP, and MLP(XZ) is an output tensor of the MLP. The RELU function is a simple and effective activation function that achieves non-linear mapping by truncating negative inputs to 0 and preserving positive inputs as original values.

[0102] When the MLP is the first multi-layer perceptron, XZ is the output tensor of the self-attention layer in the current encoder where the first multi-layer perceptron is located, that is, the input tensor of the first multi-layer perceptron, and MLP(XZ) is the output tensor of the first multi-layer perceptron. MLP(XZ) can also be used as the output tensor of the current encoder or the input tensor of the layer after the first multi-layer perceptron.

[0103] In addition, when the MLP is the first multi-layer perceptron, and .

[0104] In some embodiments, with reference to Figure 4As shown, each of the above-mentioned encoders can further comprise a post-processing layer located after the position feedforward layer. That is, the output tensor of the position feedforward layer is taken as the input tensor of the post-processing layer in the present encoder. The output tensor of the position feedforward layer is normalized and nonlinearly processed in the post-processing layer to obtain the output tensor of the post-processing layer as the output tensor of the present encoder.

[0105] Specifically, the processing process of the above-mentioned post-processing layer can be represented by the following formula:

[0106] ; (7)

[0107] wherein X4 is the output tensor of the position feedforward layer, i.e., the input tensor of the post-processing layer, and X5 is the output tensor of the post-processing layer, is a nonlinear processing function, and LayerNorm is a normalization processing function, which can also be understood as a normalization processing function.

[0108] The present application can optimize the performance of the encoding network through the normalization and residual connection of the above-mentioned post-processing layer.

[0109] S204, predicting the state analysis result of the industrial equipment based on the encoding result tensor through the second multi-layer perception.

[0110] Wherein, the function of the second multi-layer perception is state prediction, and the second multi-layer perception is also an MLP, which can be specifically referred to the MLP formula in the aforementioned first multi-layer perception. When the encoding result tensor is obtained, the data tensor at the last time of the data sequence is extracted therefrom for state prediction.

[0111] The above-mentioned second multi-layer perception is used to nonlinearly map the data at the last time to obtain the state analysis result. The nonlinear mapping process can specifically comprise: first, linearly mapping the data tensor at the last time through a first mapping tensor to obtain a data linear tensor, then nonlinearly mapping the data linear tensor through a nonlinear function (for example, a RELU function) to obtain a data nonlinear tensor; finally, linearly mapping the data nonlinear tensor through a second mapping tensor to obtain a tensor corresponding to the state analysis result.

[0112] The calculation formula of the second multi-layer perception is the same as that of the MLP in the aforementioned first multi-layer perception, except that the parameters in the formula are different, which will not be described herein. The first mapping tensor of the second multi-layer perception is .

[0113] The prediction method of the industrial internet lightweight edge model provided in the embodiments of the present application reduces the calculation complexity and improves the state analysis efficiency by performing nonlinear mapping on the query tensor and the key tensor when extracting features through the self-attention layer, and replacing the similarity with the product of the nonlinear mapping tensor of the query tensor and the nonlinear mapping tensor of the key tensor.

[0114] Figure 6 is a structural schematic diagram of the prediction device of the industrial internet lightweight edge model provided in the present application, as Figure 6 indicated, the prediction device 400 of the industrial internet lightweight edge model provided in the embodiments includes:

[0115] The state data acquisition module 401 is configured to acquire state data of an industrial device, and the state data is a data sequence collected by one or more sensors.

[0116] The tensorization module 402 is configured to perform tensorization processing on the state data to obtain a state tensor, and the order of the state tensor is greater than 2 of the order of the state data.

[0117] The encoding module 403 is configured to generate an encoding result tensor corresponding to the state tensor through an encoding model, and the encoding model includes a plurality of encoding layers that are sequentially cascaded, each encoding layer includes a self-attention layer and a positional feedforward layer located after the self-attention layer, the positional feedforward layer includes a first multi-layer perceptron, in the self-attention layer, the input tensor of the encoding layer is mapped into a query tensor, a key tensor and a value tensor through a query mapping tensor, a key mapping tensor and a value mapping tensor, and an output tensor of the self-attention layer is generated based on the product of the corresponding nonlinear mapping tensors of the query tensor and the key tensor, and the value tensor, and in the positional feedforward layer, the output tensor of the self-attention layer in the encoding layer is linearly mapped.

[0118] The prediction module 404 is configured to predict a state analysis result of the industrial device based on the encoding result tensor through a second multi-layer perceptron.

[0119] In a possible implementation, the encoding module 403 is further configured to:

[0120] convert the key tensor and the value tensor into a low-order key tensor and a low-order value tensor respectively through a low-rank tensor with a rank less than or equal to a preset threshold; and generate the output tensor of the self-attention layer based on the product of the corresponding nonlinear mapping tensors of the query tensor and the low-order key tensor, and the low-order value tensor.

[0121] In a possible implementation, the encoding module 403 is further configured to:

[0122] The query tensor and the low-order key tensor are respectively mapped into a nonlinear query tensor and a nonlinear key tensor through a kernel function; a first product of the nonlinear query tensor and the nonlinear key tensor is calculated, and a second product of the nonlinear query tensor, the nonlinear key tensor and the low-order value tensor is calculated, and a ratio of the second product to the first product is taken as an output tensor of the self-attention layer.

[0123] In a possible implementation, the encoding module 403 is further configured to:

[0124] For each of the query mapping tensor, the key mapping tensor and the value mapping tensor, the mapping tensor is decomposed into two chain-shaped tensors, the product of the two chain-shaped tensors is equal to the mapping tensor, and the two chain-shaped tensors corresponding to the mapping tensor include a first chain-shaped tensor and a second chain-shaped tensor; the input tensor of the encoder is converted into an intermediate mapping tensor through each first chain-shaped tensor, and the key mapping tensor, the query mapping tensor and the value mapping tensor correspond to one intermediate mapping tensor respectively; the intermediate mapping tensor corresponding to the mapping tensor is mapped into the query tensor, the key tensor and the value tensor through the second chain-shaped tensor of the mapping tensor.

[0125] In a possible implementation, the encoder further includes a post-processing layer located after the position feedforward layer, and the encoding module 403 is further configured to: perform normalization processing and nonlinear processing on the output tensor of the position feedforward layer in the post-processing layer, as the output tensor of the encoder.

[0126] In a possible implementation, the tensorization module 402 is further configured to:

[0127] The number of sensors collecting state data and the length of a data sequence collected by each sensor are obtained; the number of sensors is converted into the product of J first numerical values, and the length of the data sequence is converted into the product of I second numerical values, I and J are both positive integers greater than or equal to 2; and the state data is tensorized into an I+J-order state tensor according to the J first numerical values and the I second numerical values, the J first numerical values and the I second numerical values are both the length of a data sequence in a corresponding dimension of the state tensor.

[0128] The prediction device of the industrial internet lightweight edge model provided in the embodiment can execute the method provided in the method embodiment, and has similar implementation principles and technical effects, which will not be described here in detail.

[0129] Figure 7 is a structural schematic diagram of an electronic device provided in the present application. As shown in Figure 6 The electronic device 600 provided in the embodiment includes at least one processor 601 and a memory 602. Optionally, the electronic device 600 further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected through a bus 604.

[0130] In the implementation process, the at least one processor 601 executes the computer execution instructions stored in the memory 602, so that the at least one processor 601 executes the above-mentioned method.

[0131] The specific implementation process of the processor 601 can refer to the method embodiments described above, which have similar implementation principles and technical effects, and details are not described here.

[0132] In the above embodiments, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, CPU for short), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, DSP for short), application specific integrated circuits (English: Application Specific Integrated Circuit, ASIC for short), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0133] The memory can contain a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0134] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0135] The present application also provides a computer program product, comprising a computer program, which is executed by a processor to implement the above-mentioned method.

[0136] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when the processor executes the computer execution instructions, the above-mentioned method is implemented.

[0137] The above-mentioned readable storage medium can be realized by any type of volatile or nonvolatile storage devices or their combinations, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0138] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0139] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0141] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0142] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0143] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction-related hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: a ROM, a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0144] Finally, it should be noted that: those skilled in the art will easily derive other embodiments of the present application after considering the specification and practicing the disclosed content. The present application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or conventional technical means in the art that are not disclosed in the present application, and is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

Claims

1. A prediction method for a lightweight edge model of the Industrial Internet, characterized in that, include: Acquire status data of industrial equipment, wherein the status data is a data sequence collected by one or more sensors; The state data is subjected to tensor quantization to obtain a state tensor, and the order of the state tensor is greater than the order of the state data (2). The encoding result tensor corresponding to the state tensor is generated by an encoding model, which includes multiple encoders cascaded in sequence. Each encoder includes a self-attention layer and a position feedforward layer located after the self-attention layer. The position feedforward layer includes a first multilayer perceptron. In the self-attention layer, the input tensor of the encoder is mapped to a query tensor, a key tensor, and a value tensor by query mapping tensor, key mapping tensor, and value mapping tensor. The output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, respectively, and the value tensor. The state analysis results of the industrial equipment are predicted by a second multilayer perceptron based on the encoding result tensor.

2. The method according to claim 1, characterized in that, The step of generating the output tensor of the self-attention layer based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, respectively, and the value tensor, includes: The key tensor and the value tensor are transformed into low-order key tensors and low-order value tensors respectively using low-rank tensors with rank less than or equal to a preset threshold. The output tensor of the self-attention layer is generated based on the product of the query tensor, the nonlinear mapping tensors corresponding to the low-order key tensor, and the low-order value tensor.

3. The method according to claim 2, characterized in that, The step of generating the output tensor of the self-attention layer based on the product of the query tensor, the nonlinear mapping tensors corresponding to the low-order key tensor, and the low-order value tensor includes: The query tensor and the low-order key tensor are mapped to nonlinear query tensors and nonlinear key tensors respectively using kernel functions; Calculate the first product of the nonlinear query tensor and the nonlinear key tensor, and calculate the second product of the nonlinear query tensor, the nonlinear key tensor and the low-order value tensor, and use the ratio of the second product to the first product as the output tensor of the self-attention layer.

4. The method according to any one of claims 1 to 3, characterized in that, The step of mapping the encoder's input tensor to query tensors, key tensors, and value tensors through query mapping tensors, key mapping tensors, and value mapping tensors includes: For each of the query mapping tensor, the key mapping tensor, and the value mapping tensor, the mapping tensor is decomposed into two chain tensors. The product of the two chain tensors is equal to the mapping tensor. The two chain tensors corresponding to the mapping tensor include a first chain tensor and a second chain tensor. The input tensor of the encoder is converted into an intermediate mapping tensor by each of the first chain tensors, and the key mapping tensor, the query mapping tensor and the value mapping tensor each correspond to an intermediate mapping tensor; Through the second chain tensor of each of the mapping tensors, the intermediate mapping tensors corresponding to the mapping tensors are respectively mapped to the query tensor, the key tensor, and the value tensor.

5. The method according to any one of claims 1 to 3, characterized in that, The encoder further includes a post-processing layer located after the position feedforward layer, in which the output tensor of the position feedforward layer is normalized and nonlinearly processed to serve as the output tensor of the encoder.

6. The method according to any one of claims 1 to 3, characterized in that, The process of tensorizing the state data to obtain the state tensor includes: Obtain the number of sensors collecting the status data and the length of the data sequence collected by each sensor; The number of sensors is converted into the product of J first values, and the length of the data sequence is converted into the product of I second values, where I and J are both positive integers greater than or equal to 2; Based on the J first values ​​and the I second values, the state data is tensored into an I+J order state tensor, where the J first values ​​and the I second values ​​are both the data sequence lengths of the state tensor in the corresponding dimensions.

7. A prediction device for a lightweight edge model of the Industrial Internet, characterized in that, include: A status data acquisition module is used to acquire status data of industrial equipment, wherein the status data is a data sequence collected by one or more sensors; The tensor quantization module is used to perform tensor quantization processing on the state data to obtain a state tensor, wherein the order of the state tensor is greater than the order of the state data (2). An encoding module is used to generate an encoded result tensor corresponding to the state tensor through an encoding model. The encoding model includes multiple encoders cascaded in sequence. Each encoder includes a self-attention layer and a position feedforward layer located after the self-attention layer. The position feedforward layer includes a first multilayer perceptron. In the self-attention layer, the input tensor of the encoder is mapped to a query tensor, a key tensor, and a value tensor through a query mapping tensor, a key mapping tensor, and a value mapping tensor. The output tensor of the self-attention layer is generated based on the product of the nonlinear mapping tensors corresponding to the query tensor and the key tensor, respectively, and the value tensor. In the position feedforward layer, the output tensor of the self-attention layer in this encoder is linearly mapped. The prediction module is used to predict the state analysis results of the industrial equipment based on the encoding result tensor using a second multilayer perceptron.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images