A layer sequence extraction implementation method, system and storage medium

Abstract

The application provides a layer sequence extraction implementation method, system and storage medium. The method comprises the following steps: judging the existence of each operator in a target feature matrix according to a preset probability threshold range, and outputting an operator existence matrix; obtaining an operator number matrix through calculation based on the operator existence matrix; calculating an operation distribution matrix and a first-stage loss function based on the operator existence matrix and the operator number matrix, and performing first-stage iterative back propagation according to the loss gradient; extracting operation values with a count value greater than 0 in the operation distribution matrix, performing sequence coding and fusion, and then obtaining a target layer sequence and a second-stage loss function through calculation decoding, and performing second-stage iterative back propagation according to the CTC loss gradient. Through the two-stage layer sequence extraction method, the problem that the DNN model layer sequence prediction accuracy is greatly reduced due to operation fusion optimization is solved, the layer sequence extraction accuracy is improved, and the error rate is reduced.

Description

Technical Field

[0001] This application belongs to the field of AI model technology, and in particular relates to a method, system and storage medium for layer sequence extraction. Background Technology

[0002] Deep neural networks (DNNs) are a common type of AI model used in the field of autonomous driving. These models often require massive amounts of data and computing power to train and achieve good performance, a process that is extremely expensive. If a deployed DNN model is considered a black box, a systematic approach is needed to analyze the model's layer sequences using bypass information. Many existing studies rely on assumptions about information that is difficult to obtain in practice, such as complete memory tracking information, interventions in the training process, and the scope of the target DNN architecture. Some studies impose additional constraints, such as restrictions on computational mappings and layer types, to simplify the problem. Recent research has shown that deep learning-based methods exhibit reliable performance with limited prior information. However, these methods still assume that the DNN model has not been optimized using methods such as computational fusion. These optimizations can be achieved using common tools like the Apache Tensor Virtual Machine (TVM). Given the performance requirements of deployed models, this assumption is flawed. Evidence suggests that current deep learning-based inverse methods experience a significant drop in accuracy when applied to TVM-optimized regressive neural networks (RNNs) and convolutional neural networks (CNNs). Therefore, a novel deep learning-based approach is needed to overcome this inverse performance degradation caused by model deployment optimization. Summary of the Invention

[0003] To address the shortcomings of the existing technology, this application provides a layer sequence extraction method, system, and storage medium. The method extracts the target layer sequence through a two-stage scheme, solving the problem of a significant decrease in the prediction accuracy of DNN model layer sequences caused by computational fusion optimization. This improves the accuracy of layer sequence extraction and reduces the error rate. Furthermore, it solves the problem of the CTC loss function failing to converge in the second stage due to the number of kernels being less than the number of computational layers. By flattening the fusion operation, the fusion operation is expanded into a sequence longer than the label sequence, ensuring that more information and features are extracted, thereby improving the accuracy and performance of layer sequence extraction.

[0004] To achieve the above objectives, this application provides a method for extracting layer sequences, the method mainly comprising:

[0005] S1: Determine the existence of each operator in the target feature matrix based on the preset probability threshold range, and output the operator existence matrix.

[0006] S2: Obtain the operator quantity matrix by calculating the operator existence matrix.

[0007] S3: Based on the operator existence matrix and the operator quantity matrix, calculate the operation distribution matrix and the first-stage loss function, and perform one-stage iterative backpropagation based on the gradient calculated from the loss in the first-stage loss function.

[0008] S4: Extract the operation values ​​with count values ​​greater than the preset data from the operation distribution matrix, encode and fuse them into sequences, obtain the target hierarchical sequence and the two-stage loss function by calculation and decoding, and perform two-stage iterative backpropagation based on the gradient calculated by the CTC loss in the two-stage loss function.

[0009] In this application, prior to step S1, the following steps are also included:

[0010] Extract the original feature matrix from the target side channel information.

[0011] The original feature matrix is ​​projected onto the latent space H in the first fully convolutional network layer to obtain the latent vector.

[0012] In the second fully convolutional network layer, the latent vectors are mapped back to the operation space to output the target feature matrix; wherein, a recurrent neural network is connected in each fully convolutional network layer.

[0013] In this application, step S1 specifically includes:

[0014] The target feature matrix is ​​mapped to a preset range using a preset activation function.

[0015] Specifically, for each operator, the output is converted into an existence probability according to the preset probability threshold range, and an operator existence matrix is ​​output.

[0016] In this application, step S2 specifically includes:

[0017] The operator is projected onto the latent space H1 in the third fully convolutional network layer.

[0018] The potential space H1 is sliced ​​along the kernel dimension and then fed into the first multi-head attention layer to output the first attention weight matrix.

[0019] The latent space H2 containing the first attention weight matrix is ​​sliced ​​along the kernel dimension and then input into the second multi-head attention layer to output the second attention feature matrix; wherein, the second attention feature matrix is ​​extracted into the latent space H3.

[0020] The intermediate feature matrix output from the latent space H3 is counted and decoded to obtain the operator count matrix.

[0021] In this application, the calculation of the operation distribution matrix specifically involves:

[0022] The result of multiplying the operator existence matrix and the operator quantity matrix is ​​reconstructed according to a preset formula to obtain the operation distribution matrix.

[0023] In this application, the first-stage loss function includes at least BCELoss loss and BCEwithLogitsLoss loss.

[0024] The BCELoss loss is calculated from the operator quantity matrix, and the BCEwithLogitsLoss loss is calculated from the operator existence matrix.

[0025] The first-stage loss function is calculated by weighting and summing the BCELoss loss and the BCEwithLogitsLoss loss.

[0026] In this application, the step of extracting operation values ​​with counts greater than 0 from the operation distribution matrix for sequence encoding and fusion specifically involves:

[0027] The sequence-encoded operation values ​​are arranged into a two-dimensional array; wherein the first row of the two-dimensional array includes integer indices representing operations, and the second row includes operator counts corresponding to each operation value.

[0028] The sequence in the two-dimensional array is input into the recurrent neural network group, and the result is input into the fourth fully convolutional network layer for fusion, and the fused sequence is output.

[0029] In this application, obtaining the target hierarchical sequence through computational decoding specifically involves:

[0030] The fused sequence is input into a preset activation function to calculate the probability distribution vector of the layer sequence.

[0031] The target hierarchical sequence is obtained by inputting the probability distribution vector into the CTC decoder for decoding.

[0032] To achieve the above objectives, this application also provides a layer sequence extraction system, the system mainly comprising:

[0033] The module includes an existence prediction module, a count prediction module, a first data processing module, a sequence encoding module, and a second data processing module.

[0034] The existence prediction module is used to determine the existence of each operator in the target feature matrix according to a preset probability threshold range, and output the operator existence matrix.

[0035] The counting prediction module is used to calculate the number matrix of operators through the operator existence matrix.

[0036] The first data processing module is used to calculate the operation distribution matrix and the first-stage loss function based on the operator existence matrix and the operator quantity matrix, and to perform first-stage iterative backpropagation based on the gradient calculated from the loss in the first-stage loss function.

[0037] The sequence encoding module is used to extract operation values ​​with count values ​​greater than 0 from the operation distribution matrix for sequence encoding and fusion.

[0038] Furthermore, the second data processing module is used to obtain the target hierarchical sequence and the two-stage loss function by calculating the decoding of the fused sequence after sequence encoding and fusion, and to perform two-stage iterative backpropagation by calculating the gradient based on the CTC loss in the two-stage loss function.

[0039] To achieve the above objectives, this application also provides a vehicle, which includes at least the layer sequence extraction system described above.

[0040] To achieve the above objectives, this application also provides a storage medium, which is a type of computer-readable storage medium, having stored thereon a computer program that, when executed by a processor, implements the layer sequence extraction implementation method as described above.

[0041] Compared with the prior art, the advantages of this application are as follows:

[0042] This application proposes a layer sequence extraction method, system, and storage medium. By inputting the target feature matrix into a two-stage scheme and outputting the final target layer sequence, it solves the problem of a significant decrease in the prediction accuracy of DNN model layer sequences caused by computational fusion optimization, improves the accuracy of layer sequence extraction, and reduces the error rate. At the same time, it also solves the problem of the CTC loss function failing to converge in the second stage due to the number of kernels being less than the number of computational layers. By flattening the fusion operation, the fusion operation is expanded into a sequence longer than the label sequence, ensuring that more information and features are extracted, thereby improving the performance of layer sequence extraction. Attached Figure Description

[0043] Figure 1 This is a flowchart of a layer sequence extraction method in one embodiment of this application.

[0044] Figure 2 This is a schematic diagram of the structure of a layer sequence extraction system according to an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0046] Example 1:

[0047] To address the aforementioned technical issues, this application provides a method for extracting layer sequences. It should be noted that when using the current best deep learning inverse tool, DeepSniffer, and DeepSniffer+ for model deployment optimization, the error rate of the DNN dataset is extremely high. After DNN optimization (Opt-RNN, Opt-CNN), the prediction error rate for layer sequences increases significantly, and the larger the value, the higher the error rate.

[0048] In this application, the method aims to solve the technical problem of a significant decrease in the prediction accuracy of DNN model layer sequences caused by optimization methods involving computational fusion. Simultaneously, this application also proposes a solution to the problem of the CTC loss function failing to converge due to the number of kernels being less than the number of computational layers. The CTC loss function is widely used in current state-of-the-art deep learning-based layer extraction tools. The method mainly includes:

[0049] S1: Determine the existence of each operator in the target feature matrix based on the preset probability threshold range, and output the operator existence matrix.

[0050] It should be noted that in this embodiment, the hierarchical sequence can be extracted in two stages. In the first stage, a neural network is used to predict the presence and count of operators, and the output of this stage is the operation distribution matrix OP. i This neural network typically consists of convolutional layers, long short-term memory (LSTM) layers, and activation layers, and incorporates an attention mechanism. The input to this stage is the collected side-channel information X. i This includes the latency of each operator, DRAM read / write information, and caching patterns for memory transactions, as well as the utilization and hit rate of the L1 / L2 cache.

[0051] In this embodiment, before step S1, the following steps are also included:

[0052] Extract the original feature matrix from the target side channel information.

[0053] The extracted raw feature matrix may include: latency information, which may include the execution time of each operator, the instruction issuance time, and the completion time; DRAM read / write information, recording the read and write operations of each operator on DRAM, including the read / write address, data size, and read / write type; memory transaction caching mode, analyzing the memory access mode of each operator, including cache hits and cache line replacement strategies; L1 / L2 cache utilization and hit rate, statistically analyzing the utilization of L1 / L2 cache for each operator, including cache hit rate and cache line replacement, etc., and is not limited to these.

[0054] For a given operator, this information will form part of the original feature matrix and be used as input to the neural network for the prediction and hierarchical extraction process.

[0055] In this embodiment, the extracted original feature matrix is ​​first sent to the existence prediction module, which includes multiple fully convolutional network layers, recurrent neural networks, and preset activation functions.

[0056] Further, the original feature matrix is ​​projected onto the latent space H in the first fully convolutional network layer to obtain the latent vector. Here, the first fully convolutional network layer refers to the first FCN layer.

[0057] It's important to note that the latent space H refers to a low-dimensional representation space in deep learning, containing a high-level abstract representation of the original data. This concept arises during feature extraction and learning; by mapping the original data to the latent space, the inherent structure and features of the data can be better expressed. In this embodiment, projecting the original feature matrix onto the latent space H in the first fully convolutional network layer means that the original features are transformed into a low-dimensional representation in the latent space through the forward propagation process of the neural network.

[0058] A latent vector is a representation of data in a latent space. In deep learning, the encoder part of a neural network typically maps input data into the latent space to obtain the corresponding latent vector. This latent vector contains a high-level abstract representation of the input data, capturing its important features and structural information. In this embodiment, projecting the original feature matrix onto the latent vector in the first fully convolutional network layer means that the encoder part of the neural network obtains a low-dimensional representation of the original features in the latent space.

[0059] In this embodiment, the latent space H represents the mapping space of the original features after passing through the neural network encoder, while the latent vector is the low-dimensional vectorized form of the original features represented in this space.

[0060] In the second fully convolutional network layer, the latent vectors are mapped back to the operation space, outputting the target feature matrix; wherein, a recurrent neural network is connected in each fully convolutional network layer. The second fully convolutional network layer refers to the last FCN layer in the sequence.

[0061] Recurrent neural networks refer to LSTM layers, which learn the inherent short-term and long-term dependencies between kernel traces.

[0062] In this application, step S1 specifically includes:

[0063] The target feature matrix is ​​mapped to a preset range using a preset activation function.

[0064] Preferably, the preset activation function can be the sigmoid function, or the ReLU function. The advantage of the ReLU function is that it is simple to calculate and performs well in practice, and can alleviate the gradient vanishing problem. The Leaky ReLU function is an improvement on the standard ReLU. When the input is negative, it does not take 0 completely, but multiplies it by a small slope α. Other options include the ELU function, the Tanh function, and the Softmax function, etc., and are not limited to these. The preset range can be [0,1].

[0065] Specifically, for each operator, the output is converted into an existence probability according to the preset probability threshold range, and an operator existence matrix is ​​output.

[0066] Preferably, assuming in an example deep learning model, it can be used to obtain X i Predict OP i ,in Therefore, the k-th row represents the operation distribution of the k-th kernel, and the o-th column represents the corresponding operation count. The extracted feature matrix X i The data is first fed into the Existence Prediction (EP) module, which consists of stacked fully connected neural network (FCN) and long short-term memory (LSTM) layers, along with a sigmoid function. The first FCN layer projects the side-channel features into the latent space, while the last FCN layer maps the latent vectors back into the operation space. The LSTM layers learn the inherent short-term and long-term dependencies between kernel traces. Finally, the sigmoid function collapses the output of the last layer into the range [0,1]. This represents the probability of each operator existing.

[0067] In summary, by first feeding the feature matrix into the existence prediction (EP) module to output the probability of each operator's existence, the robustness, accuracy, and interpretability of the model can be improved, while also enhancing its generalization ability. These technical effects contribute to improving the performance of deep learning models in real-world tasks.

[0068] S2: Obtain the operator quantity matrix by calculating the operator existence matrix.

[0069] It should be noted that the output of the EP module is then fed into the Count Prediction (CP) module, which includes an FCN layer, two multi-head attention layers, and a count decoding component.

[0070] In this embodiment, assuming there is an input feature matrix, after processing by the EP module, the following 2x3 operator existence matrix is ​​obtained:

[0071] [[0.9,0.2,0.8],[0.3,0.7,0.5]], where each number represents the probability of the operator existing at the corresponding position.

[0072] This matrix will then be fed into the Count Prediction (CP) module. This CP module includes the following components:

[0073] FCN layer: Fully connected network layer, used to further extract and transform the existence probability of the input.

[0074] Multi-head attention layer: Through the multi-head attention mechanism, the model can simultaneously focus on different parts of the input and learn the relationships between them, thereby better understanding the structure and features of the input data.

[0075] The counting decoding component is used to convert the features processed by the previous layers into a final operator count matrix. This component may involve operations such as integrating, weighting, and nonlinear transformations of multiple features to obtain the final result.

[0076] In actual implementation, each component may involve specific parameter settings, activation function selection, and connection methods between layers. These components work together to ultimately transform the operator existence matrix processed by the EP module into an operator count matrix, thereby completing further processing and inference of the model output.

[0077] Further, in this embodiment, step S2 specifically includes:

[0078] The operator is projected onto the latent space H1 in the third fully convolutional network layer.

[0079] It should be noted that the operators are first stored in a matrix. Projected into potential space Where es represents the embedding size.

[0080] The potential space H1 is sliced ​​along the kernel dimension and then fed into the first multi-head attention layer to output the first attention weight matrix.

[0081] The latent space H2 containing the first attention weight matrix is ​​sliced ​​along the kernel dimension and then input into the second multi-head attention layer to output the second attention feature matrix; wherein, the second attention feature matrix is ​​extracted into the latent space H3.

[0082] It should be noted that... It is defined as follows, through two consecutive multi-head attention layers:

[0083] f mh (Q,K,V)=cat(h1,…,h h W o

[0084]

[0085]

[0086] in, and All dimensions are And W o The dimension is All of these parameters can be learned, among which The first multi-head attention layer enables the model to simultaneously focus on the importance of all kernels for each operation. This includes... The slices are input into a multi-head attention layer to learn the importance of correlations between different kernels. Similarly, Slicing is performed along the kernel dimension and fed into a second multi-head attention layer to capture the importance of correlations between operations. middle.

[0087] The intermediate feature matrix output from the latent space H3 is counted and decoded to obtain the operator count matrix.

[0088] It should be noted that the final stage of the CP module is the counting decoding (CD) component, which consists of several FCN layers and a softmax function. The output dimension of each FCN is determined by the maximum count (OC) of the corresponding operation in the dataset. max Confirmed. After passing through the softmax function, the decoded matrix is ​​obtained. For kernel k, vector The index of the largest element (starting from 1) represents the number of prediction counts for that operation in that core, assuming that the operation exists.

[0089] Preferably, assume there is an operation sequence containing two operations: operation A and operation B. A multi-head attention mechanism will then be used to learn the correlation between these two operations.

[0090] First, represent the sequence of operations as a matrix, where each column represents an operation. For example:

[0091] [[1,0],[0,1]]; where the first column represents operation A and the second column represents operation B.

[0092] Next, the operation sequence is projected onto the latent space H1, and the following results are obtained:

[0093] [[[0.2,0.8],[0.7,0.6]],[[0.9,0.1],[0.3,0.5]]].

[0094] The 2D vector at each position represents the representation of the corresponding operation in the latent space H1.

[0095] Then, the latent space H1 is sliced ​​along the kernel dimension, and the sliced ​​result is input into the first multi-head attention layer. Assume there are two attention heads, each outputting a weight matrix. Assume the output of the first multi-head attention layer is:

[0096] [[0.8,0.2],[0.4,0.6]].

[0097] This matrix represents the attention weights after the first multi-head attention layer has weighted the information in the latent space H1.

[0098] The first attention weight matrix is ​​sliced ​​as H2 and fed into the second multi-head attention layer. Assume the output of the second multi-head attention layer is:

[0099] [[[0.2,0.1],[0.9,0.8]],[[0.5,0.6],[0.2,0.3]]].

[0100] This matrix represents the features obtained by weighting the information in the latent space H2 after the second multi-head attention layer.

[0101] Finally, the output of the second multi-head attention layer is extracted into the latent space H3, resulting in:

[0102] [[[0.2,0.1],[0.9,0.8]],[[0.5,0.6],[0.2,0.3]]].

[0103] This result is the representation in the latent space H3.

[0104] In summary, the above embodiments demonstrate how the multi-head attention mechanism helps the model learn the correlations between operations. First, weighted processing using the multi-head attention mechanism on the latent space H1 captures the correlations and importance between different kernels. Then, the output of the first multi-head attention layer is used as a slice of H2, and weighted processing is performed again using the multi-head attention mechanism to obtain a higher-level feature representation. Finally, the output of the second multi-head attention layer is extracted into the latent space H3 to obtain the final representation.

[0105] S3: Based on the operator existence matrix and the operator quantity matrix, calculate the operation distribution matrix and the first-stage loss function, and perform one-stage iterative backpropagation based on the gradient calculated from the loss in the first-stage loss function.

[0106] In this application, the calculation of the operation distribution matrix specifically involves:

[0107] The result of multiplying the operator existence matrix and the operator quantity matrix is ​​reconstructed according to a preset formula to obtain the operation distribution matrix.

[0108] In this embodiment, With existence matrix Multiplication is used to reconstruct the operation distribution matrix.

[0109] Preferably, the operation distribution matrix OP is used as the final output of the first stage. i It can be constructed using the following formula:

[0110]

[0111] Where SP represents a subset of operations that may have multiple operation counts in a core, and tr is a threshold for the probability of existence, for example, 0.5.

[0112] Furthermore, the first-stage loss function includes at least BCELoss loss and BCEwithLogitsLoss loss.

[0113] The BCELoss loss is calculated from the operator quantity matrix, and the BCEwithLogitsLoss loss is calculated from the operator existence matrix.

[0114] The first-stage loss function is calculated by weighting and summing the BCELoss loss and the BCEwithLogitsLoss loss.

[0115] In this embodiment, the loss function for the first stage consists of two parts. The first part is contributed by the YO matrix through the following binary cross-entropy loss (BCELoss) function:

[0116]

[0117] in express The actual value. The second part is obtained by YP through the BCEwithLogitsLoss function. i Contributions:

[0118]

[0119] The final loss function will be a weighted sum of the two: L = αl op +βl on , where α and β are hyperparameter weights. Then, the gradient is calculated based on the overall loss for backpropagation.

[0120] In summary, the operation distribution matrix is ​​reconstructed by multiplying the operator existence matrix and the operator count matrix using a predefined formula. This operation distribution matrix represents a subset of operations that may have multiple operation counts within a kernel and can be constructed based on a threshold of existence probability. The one-stage loss function consists of two parts. The first part is contributed by the operator count matrix through the binary cross-entropy loss (BCELoss) function, and the second part is contributed by the operator existence matrix through the BCEwithLogitsLoss function. The final loss function is a weighted sum of the two, and the importance of the two parts of the loss can be adjusted using hyperparameter weights. The gradient is calculated based on the overall loss for backpropagation, thereby updating the model parameters to minimize the loss function.

[0121] This embodiment optimizes model parameters and improves task performance by manipulating the distribution matrix and calculating the one-stage loss function, as well as through backpropagation. Furthermore, by decomposing and weighting the loss function, the influence of different loss components can be better controlled, thereby improving the stability and convergence speed of model training.

[0122] S4: Extract the operation values ​​with count values ​​greater than 0 from the operation distribution matrix, encode and fuse them into sequences, obtain the target hierarchical sequence and the two-stage loss function by calculation and decoding, and perform two-stage iterative backpropagation based on the gradient calculated by the CTC loss in the two-stage loss function.

[0123] It should be noted that operation values ​​with counts greater than 0 are extracted from the operation distribution matrix. These values ​​may represent a subset of operations with a certain probability of existence. Sequence encoding and fusion of the extracted operation values ​​may involve mapping these operation values ​​into sequences and performing some kind of fusion operation to better represent the features and information of the target hierarchical sequence.

[0124] By computationally decoding the sequence and utilizing the encoded and fused information, a target hierarchical sequence is obtained. This target hierarchical sequence may represent or structure that we want the model to learn. Based on the decoded target hierarchical sequence, a two-stage loss function is calculated. This loss is used to measure the difference between the model output and the target sequence, and is then used to adjust the model parameters. Gradients are calculated based on the two-stage loss function, and two-stage iterative backpropagation is performed. This process updates the model parameters, making the model more adapted to the target hierarchical sequence.

[0125] This application processes the operation distribution matrix, including extraction, encoding, fusion, and decoding, to obtain the target hierarchical sequence and calculate the two-stage loss function. Then, it adjusts the model parameters through two-stage iterative backpropagation to improve the model's performance on the target sequence.

[0126] Further, the step of extracting operation values ​​with counts greater than 0 from the operation distribution matrix for sequence encoding and fusion specifically involves:

[0127] The sequence-encoded operation values ​​are arranged into a two-dimensional array; wherein the first row of the two-dimensional array includes integer indices representing operations, and the second row includes operator counts corresponding to each operation value.

[0128] In an example neural network of this embodiment, the decoding operation distribution matrix OP is further performed. i To predict hierarchical sequence y i For directly labeling unsegmented sequences, the CTC loss function was employed, the same as that used in existing techniques in DeepSniffer and DeepSniffer+. However, using the CTC loss function typically requires the input sequence length to exceed the label sequence length. This prerequisite is difficult to meet for models optimized through operation fusion.

[0129] Therefore, this application proposes a special sequence encoding module that concatenates the operation count predictions of each kernel to retain the advantages of the CTC loss function in labeling unsegmented sequences, while addressing the aforementioned limitations. In this sequence encoding stage, the operation count predictions of each kernel are scanned. i Each row extracts operations with predicted values ​​greater than 0, forming a two-dimensional array where the first row contains integer indices representing the operations, and the second row contains the corresponding prediction operator counts. The merged operations are then flattened into a sequence that is always longer than the label sequence.

[0130] This application addresses the problem that the CTC loss function cannot converge due to the number of kernels being less than the number of operational layers. By flattening the fusion operation, the fusion operation is expanded into a sequence longer than the label sequence, ensuring that more information and features are extracted, thereby improving the layer sequence extraction performance.

[0131] The sequence in the two-dimensional array is input into the recurrent neural network group, and the result is input into the fourth fully convolutional network layer for fusion, and the fused sequence is output.

[0132] In this embodiment, the encoded sequence is then fed into a two-layer Long Short-Term Memory (LSTM) network to learn operation-level temporal dependency patterns, such as fusion strategies. The output of the LSTM is then fused through a fully connected neural network (FCN) layer.

[0133] This application enables the model to learn operational-level temporal dependency patterns by feeding the input sequence into an LSTM network. The gating mechanism within the LSTM network allows the model to capture long-term dependencies in the sequence, thereby better understanding and modeling sequence data.

[0134] The operation-level time dependency patterns learned by the LSTM network can be used to learn fusion strategies. By analyzing the temporal relationships of sequences, the model can select different fusion methods as needed to achieve better sequence fusion results.

[0135] After processing by the LSTM network and FCN layer, the model can fuse the input sequences and generate a fused sequence as the output. This fused sequence may contain key features and information from the original sequence to better represent the characteristics of the overall sequence.

[0136] By combining neural network layers such as RNN, LSTM, and FCN, this technique enables the fusion and output of input sequences while learning time-dependent patterns and fusion strategies. This helps improve the model's ability to model and understand sequence data, as well as enhance the effectiveness of sequence fusion.

[0137] In this application, obtaining the target hierarchical sequence through computational decoding specifically involves:

[0138] The fused sequence is input into a preset activation function to calculate the probability distribution vector of the layer sequence.

[0139] The target hierarchical sequence is obtained by inputting the probability distribution vector into the CTC decoder for decoding.

[0140] In this embodiment, after passing through the softmax function, a probability distribution vector of the layer category is generated as the output y of the DNN model. i Output y i and actual value The CTC loss is used to calculate gradient updates and backpropagation. Finally, the CTC decoder processes the data from y. i Output To obtain the final target hierarchical sequence.

[0141] It should be noted that the fused sequence is input into a preset activation function, usually a specified activation function such as ReLU or tanh, and then processed by the softmax function to obtain a probability distribution vector for the layer categories. This vector represents the probability of each category, which can be understood as the model's confidence in each category.

[0142] The obtained probability distribution vector and the actual values ​​are then used to calculate the CTC loss. CTC loss is a loss function used in sequence learning that addresses the alignment problem between the input and output sequences, making it suitable for sequence transcription tasks that do not require alignment information. By calculating the CTC loss, the difference between the model output and the actual target can be evaluated and used for gradient updates and backpropagation, thereby optimizing the model parameters.

[0143] Finally, the output from the DNN model is processed by the CTC decoder to obtain the final target hierarchical sequence. The CTC decoder can identify the most likely target sequence based on the probability distribution vector of the model output, thereby achieving decoding and transcription of the model output.

[0144] This application improves the accuracy of the model in sequence transcription tasks by calculating the CTC loss and processing the CTC decoder. Optimizing the CTC loss enables the model to better learn the alignment relationships between sequences, while the CTC decoder helps the model output more accurately transcribed target sequences. The CTC loss calculation, as a loss function during training, guides the optimization of model parameters, enabling them to better fit the distribution characteristics of the target sequence. Simultaneously, the use of the CTC decoder provides more reliable sequence transcription results, helping the model achieve better training performance.

[0145] In summary, Example 1 provides a detailed analysis and illustrative description of the layer sequence extraction method proposed in this application. This application solves the problem of a significant decrease in the accuracy of DNN model layer sequence prediction caused by computational fusion optimization by inputting the target feature matrix into a two-stage scheme and outputting the final target layer sequence. This improves the accuracy of layer sequence extraction and reduces the error rate. In the two-stage process, by integrating the use of preset activation functions, softmax processing, CTC loss calculation, and CTC decoder, this technical solution can improve the performance of the model in sequence transcription tasks, thereby achieving more accurate identification and transcription of sequence data.

[0146] Example 2:

[0147] As attached Figure 2 As shown, in order to solve the above-mentioned technical problems, this application also proposes a layer sequence extraction system, which mainly includes:

[0148] The module includes an existence prediction module, a count prediction module, a first data processing module, a sequence encoding module, and a second data processing module.

[0149] In this embodiment, it is assumed that there is a complex DNN model, such as a convolutional neural network (CNN) for image classification. The layer sequence extraction system of this application can extract the layer sequence of the CNN model, that is, the name and order of each layer.

[0150] It should be noted that, firstly, each operator in the target feature matrix is ​​processed by the existence prediction module, which determines its existence based on a preset probability threshold range and outputs a corresponding operator existence matrix.

[0151] Next, using the operator existence matrix, the counting prediction module calculates the quantity matrix for each operator, reflecting the frequency of each operator's occurrence.

[0152] Based on the operator existence matrix and the operator quantity matrix, the first data processing module calculates the operation distribution matrix and the one-stage loss function. The one-stage loss function is used to calculate the gradient for one-stage iterative backpropagation, thereby optimizing the model parameters.

[0153] Operation values ​​with counts greater than 0 in the operation distribution matrix are extracted for sequence encoding and fusion. This step helps to extract important operation values ​​and prepares them for subsequent target-level sequence extraction.

[0154] For the fused sequence after sequence encoding and fusion, the second data processing module obtains the target hierarchical sequence and the two-stage loss function by calculating the decoding, and performs two-stage iterative backpropagation based on the gradient of the CTC loss in the two-stage loss function to further optimize the model parameters.

[0155] In this application, the system can effectively extract operators from the target feature matrix and calculate their number, providing basic data for subsequent operation distribution and sequence encoding.

[0156] By extracting and fusing important operation values ​​in the operation distribution matrix, the system can extract key operation values, which helps to form the target hierarchical sequence extraction.

[0157] The system performs two backpropagations, using the calculation results of the first-stage and second-stage loss functions, to update and optimize the model parameters twice, thereby improving the model's accuracy and generalization ability in sequence transcription tasks.

[0158] Preferably, the above system is divided into a two-stage scheme, which solves the performance degradation problem faced by the current best DNN model analysis tools compared with existing technologies. Compared with DeepSniffer and DeepSniffer+, the two current best tools, the accuracy of layer sequence extraction of the above system is greatly improved.

[0159] Hierarchical sequence extraction experiments were conducted by implementing the exemplary neural network described above. It was assumed that a dataset containing 8000 DNNs was collected, including 4000 randomly generated RNNs and 4000 CNNs. The primary performance metric used for evaluation was the label deviation (ler), defined as follows:

[0160]

[0161] Where represents the final layer sequence after decoding, and represents the corresponding ground truth value. represents the edit distance, i.e., the minimum number of operation steps required to transform one sequence into another. Table 1 lists a performance comparison between the two-stage scheme proposed in this patent and the current best layer sequence extraction tool.

[0162]

[0163]

[0164] It can be clearly seen that the two-stage scheme proposed in this system has the lowest error rate for extracting the layer sequence of the optimized DNN model.

[0165] In summary, Example 2 provides a detailed analysis and illustrative description of the layer sequence extraction system proposed in this application. The layer sequence extraction system proposed in this application can effectively extract and optimize target layer sequences, providing a comprehensive and effective solution for sequence transcription tasks. The specific implementation process and principles have been described in detail in Example 1 and will not be repeated in this example.

[0166] Example 3:

[0167] This application also provides a vehicle that includes at least the layer sequence extraction system described above.

[0168] It should be noted that the aforementioned vehicles refer to autonomous vehicles or intelligent driver-assisted vehicles. In such vehicles, layer sequence extraction systems can be used to process data output from visual perception systems (such as cameras, radar, and lidar) to help the vehicle understand its surroundings and make corresponding driving decisions.

[0169] Preferably, for an autonomous vehicle equipped with a layer sequence extraction system:

[0170] Visual perception module: Cameras, radars, and lidar on the car capture information about the surrounding environment, such as roads, vehicles, and pedestrians, and convert it into a feature matrix.

[0171] Existence prediction module: The existence prediction module of the layer sequence extraction system can determine whether different features and identified objects (such as vehicles, pedestrians, traffic signs, etc.) exist based on a preset probability threshold range, and output an operator existence matrix.

[0172] Counting prediction module: By using the operator storage matrix, the counting prediction module can calculate the quantity of different objects, such as how many cars or pedestrians there are, and obtain the operator quantity matrix.

[0173] Data processing and decision-making module: The first data processing module uses the operator existence matrix and operator quantity matrix to calculate the operation distribution matrix and the first-stage loss function, and performs backpropagation based on the gradient calculated from the loss. This information helps the vehicle system understand the complexity of the surrounding environment and make corresponding driving decisions, such as avoiding obstacles, decelerating, or accelerating.

[0174] The sequence encoding and decoding module encodes and fuses operation values ​​with counts greater than 0 in the operation distribution matrix to further optimize the understanding of the surrounding environment. The second data processing module decodes the fused sequence, calculates the target hierarchical sequence and the two-stage loss function, thereby further optimizing driving decisions.

[0175] By equipping vehicles with the aforementioned layer sequence extraction system, cars can perceive their surroundings more accurately and make more reliable driving decisions, thereby improving safety and driving efficiency.

[0176] In other embodiments, besides autonomous vehicles, the layer sequence extraction system can also be used in many other application scenarios. Here are some other application examples:

[0177] Speech Recognition System: In a speech recognition system, a layer sequence extraction system can be used to process audio input, extract acoustic features, and perform sequence encoding, helping the system to more accurately understand and transcribe speech content.

[0178] Financial risk control system: In the financial field, layer sequence extraction systems can be applied to analyze complex financial transaction data, identify potential risk patterns and make predictions, thereby improving risk control and decision-making efficiency.

[0179] Medical image analysis: In the field of medical image analysis, layer sequence extraction systems can help analyze medical image data, extract key features, and assist doctors in disease diagnosis and treatment planning.

[0180] Industrial production optimization: In industrial production, layer sequence extraction systems can be used to monitor equipment status, analyze production data, and optimize production processes and equipment maintenance plans to improve production efficiency and reduce costs.

[0181] Smart Home Systems: In the field of smart homes, layer sequence extraction systems can be applied to process sensor data, understand user behavior patterns, and achieve intelligent environmental control and energy management.

[0182] In summary, the above application scenarios can all be optimized through high-accuracy layer sequence extraction systems to improve data processing and decision-making processes, thereby enhancing the system's intelligence and performance.

[0183] Example 4:

[0184] This application also provides a storage medium, which is a type of computer-readable storage medium, having stored thereon a computer program that, when executed by a processor, implements the layer sequence extraction implementation method as described above.

[0185] The computer-readable storage medium can be configured on any computer device, such as a tablet computer, desktop computer, or cloud server, or it can be an in-vehicle terminal or control system. For example, it may also include input / output devices, network access devices, etc.

[0186] In this embodiment, since the storage medium can be configured on any computing device, the computer program for the layer sequence extraction system can run on various devices, enabling cross-platform applications and providing flexible solutions for different fields and scenarios.

[0187] The layer sequence extraction system can be implemented in a storage medium using computer programs, providing efficient data processing capabilities for various devices. It can quickly and accurately process large-scale data, thereby improving the system's intelligence level and performance.

[0188] By implementing a layer sequence extraction system on various computing devices, computer programs can provide these devices with real-time environmental perception and decision support, playing an important role in fields such as autonomous vehicles, smart home systems, and industrial production optimization.

[0189] This technology can also be combined with input / output devices, network access devices, etc., to extend to various application scenarios, such as intelligent traffic management, intelligent manufacturing, and medical image analysis, bringing more efficient and intelligent solutions to these fields.

[0190] In summary, the computer program in this storage medium implements the method of layer sequence extraction system, providing flexible and efficient data processing and decision support capabilities for various computing devices, and is expected to promote the widespread application and development of intelligent technology in various fields.

[0191] The processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The memory is used to store the operating system, application programs, bootloader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0192] In the several embodiments provided in this application, it will be understood that each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0193] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A layer sequence extraction method for processing audio input, extracting acoustic features, and performing sequence encoding, characterized in that, include: S1: Extract the original feature matrix from the target side channel information, and obtain the target feature matrix based on the original feature matrix; wherein, the target side channel information includes at least the latency information when the processor core executes operators, DRAM read / write information, and cache transaction information; determine the existence of each operator in the target feature matrix according to a preset probability threshold range, and output the operator existence matrix; S2: Based on the operator existence matrix, the operator quantity matrix is ​​calculated using the counting decoding component; S3: Based on the operator existence matrix and the operator quantity matrix, calculate the operation distribution matrix and the first-stage loss function, and perform one-stage iterative backpropagation based on the gradient calculated from the loss in the first-stage loss function. S4: Extract the operation values ​​with count values ​​greater than the preset data from the operation distribution matrix, encode and fuse them into sequences, and then use the CTC decoder to calculate and decode the target hierarchical sequence and the two-stage loss function of the deep neural network model after operation fusion optimization. Then, perform two-stage iterative backpropagation based on the gradient calculated by the CTC loss in the two-stage loss function. The first-stage loss function includes at least BCELoss loss and BCEwithLogitsLoss loss; The BCELoss loss is calculated from the operator quantity matrix, and the BCEwithLogitsLoss loss is calculated from the operator existence matrix. The first-stage loss function is calculated by weighting and summing the BCELoss loss and the BCEwithLogitsLoss loss.

2. The layer sequence extraction method according to claim 1, characterized in that, Also includes: The original feature matrix is ​​projected onto the latent space H of the first fully convolutional network layer to obtain the latent vector; In the second fully convolutional network layer, the latent vectors are mapped back to the operation space to output the target feature matrix; wherein, a recurrent neural network is connected in each fully convolutional network layer.

3. The layer sequence extraction method according to claim 2, characterized in that, Step S1 specifically involves: The target feature matrix is ​​mapped to a preset range using a preset activation function; Specifically, for each operator, the output is converted into an existence probability according to the preset probability threshold range, and an operator existence matrix is ​​output.

4. The layer sequence extraction method according to claim 3, characterized in that, Step S2 specifically involves: The operator-existence matrix is ​​projected onto the latent space H1 of the third fully convolutional network layer; The potential space H1 is sliced ​​along the kernel dimension and then fed into the first multi-head attention layer to output the first attention weight matrix; The latent space H2 containing the first attention weight matrix is ​​sliced ​​along the kernel dimension and then input into the second multi-head attention layer to output the second attention feature matrix; wherein, the second attention feature matrix is ​​extracted into the latent space H3; The intermediate feature matrix output from the latent space H3 is counted and decoded to obtain the operator count matrix.

5. The layer sequence extraction method according to claim 4, characterized in that, The calculation of the operation distribution matrix is ​​specifically as follows: The result of multiplying the operator existence matrix and the operator quantity matrix is ​​reconstructed according to a preset formula to obtain the operation distribution matrix.

6. The layer sequence extraction method according to claim 5, characterized in that, The step of extracting operation values ​​from the operation distribution matrix whose count values ​​are greater than preset data, performing sequence encoding and fusion, specifically involves: The sequence-encoded operation values ​​are arranged into a two-dimensional array; wherein, the first row of the two-dimensional array includes integer indices representing operations, and the second row includes operator counts corresponding to each operation value; The sequence in the two-dimensional array is input into the recurrent neural network group, and the result is input into the fourth fully convolutional network layer for fusion, and the fused sequence is output.

7. The layer sequence extraction method according to claim 6, characterized in that, The process of obtaining the target hierarchical sequence through computational decoding specifically involves: The fused sequence is input into a preset activation function to calculate the probability distribution vector of the layer sequence; The target hierarchical sequence is obtained by inputting the probability distribution vector into the CTC decoder for decoding.

8. A layer sequence extraction system for processing audio input, extracting acoustic features, and performing sequence encoding, characterized in that, include: The system comprises an existence prediction module, a counting prediction module, a first data processing module, a sequence encoding module, and a second data processing module. The existence prediction module is used to determine the existence of each operator in the target feature matrix based on a preset probability threshold range and output the operator existence matrix. The counting prediction module is used to calculate the operator quantity matrix through the operator existence matrix; The first data processing module is used to calculate the operation distribution matrix and the first-stage loss function based on the operator existence matrix and the operator quantity matrix, and to perform a first-stage iterative backpropagation based on the gradient calculated from the loss in the first-stage loss function; the sequence encoding module is used to extract the operation values ​​with a count value greater than 0 in the operation distribution matrix for sequence encoding and fusion. And the second data processing module is used to obtain the target hierarchical sequence and the two-stage loss function by calculating the decoding of the fused sequence after sequence encoding and fusion, and to perform two-stage iterative backpropagation by calculating the gradient according to the CTC loss in the two-stage loss function. The first-stage loss function includes at least BCELoss loss and BCEwithLogitsLoss loss; The BCELoss loss is calculated from the operator quantity matrix, and the BCEwithLogitsLoss loss is calculated from the operator existence matrix. The first-stage loss function is calculated by weighting and summing the BCELoss loss and the BCEwithLogitsLoss loss.

9. A storage medium, characterized in that, It is a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the layer sequence extraction method according to any one of claims 1-7.

Images