Edge-oriented seismic wave large model structure pruning method and system

Abstract

The application provides an edge-oriented seismic wave large model structured pruning method and system, and relates to the technical field of intelligent seismic monitoring. The method constructs an importance evaluation mechanism from three levels of attention heads, FFN channels and encoding layers, effectively removes structural redundancy in a seismic wave large model by combining masking verification and structured pruning, and realizes the collaborative compression of model width and depth. In the model compression process, the pruned model always maintains good seismic phase picking ability.

Description

Technical Field

[0001] This invention relates to the field of intelligent earthquake monitoring technology, specifically to a structured pruning method and system for large seismic wave models oriented towards the edge. Background Technology

[0002] Seismic phase picking is a crucial step in earthquake monitoring and automated seismic data processing, and its accuracy directly impacts the reliability of monitoring results. Traditional phase picking methods primarily rely on manually designed features such as the short-time average / long-time average ratio (STA / LTA), energy mutation detection, and statistical discrimination, offering advantages like computational simplicity and ease of implementation. However, with the rapid development of dense seismic arrays and seismic observation technologies, the scale of seismic waveform data has increased dramatically. In scenarios involving complex waveforms with low signal-to-noise ratios and large-scale continuous data processing, traditional methods are prone to problems such as missed picking, false triggering, and increased time-of-arrival positioning errors, making it difficult to meet the demands of high-precision earthquake monitoring operations.

[0003] To address the limitations of traditional methods in processing complex continuous waveforms, deep learning technology has become an important research direction in the field of seismic phase picking. GPD, DetNet & PpkNet, based on CNNs, have verified the effectiveness of deep learning methods in seismic phase picking; PhaseNet and its improved model CSESnet, based on a symmetric encoder-decoder structure, have achieved high-precision seismic phase picking through waveform segmentation; and EqTransformer, based on an attention mechanism, has achieved joint modeling of earthquake event detection and seismic phase picking. With the increasing scale of earthquake observation data and the development of large-scale modeling technology, large-scale seismic wave models are gradually becoming the main direction of intelligent earthquake monitoring. my country's "DiTing" seismic wave large-scale model is the first seismic wave large-scale model with hundreds of millions of parameters. It adopts the MAE technology architecture, is pre-trained based on the "DiTing" dataset, and has functions such as earthquake identification, magnitude estimation, and earthquake early warning.

[0004] However, while large-scale seismic wave models offer improved performance, they also introduce challenges such as high storage costs and computational complexity. Intelligent earthquake monitoring requires real-time response and long-term continuous operation. Many practical applications necessitate real-time phase acquisition at seismic stations, field monitoring terminals, and domestically produced edge computing devices, making centralized processing via high-performance computing servers difficult. In these resource-constrained scenarios, devices are typically limited by factors such as computing power, storage, and power consumption. Directly deploying a complete large-scale seismic wave model can easily lead to a series of problems, including excessively high inference latency and continuous operational instability.

[0005] To meet the deployment requirements of large models on edge computing devices, researchers have proposed various model compression techniques, including model quantization, knowledge distillation, and model pruning. Pruning is a crucial technique widely used in large model compression; its core idea is to identify and remove unnecessary or redundant structural units from the model. The SparseGPT method achieves high sparsity compression of large-scale pre-trained models through a one-time post-training pruning strategy; the LLM-Pruner framework prunes large language models using structured dependency analysis and task-aware pruning strategies; and the Wanda method performs efficient pruning based on joint importance evaluation of weights and activations. Structured pruning can effectively reduce model complexity while maintaining model structural regularity and is more easily adapted to existing inference frameworks and hardware platforms.

[0006] While structured pruning has made significant progress in general artificial intelligence tasks, research on lightweight deployment of large seismic wave models remains insufficient. Seismic waveform data exhibits distinct temporal characteristics, and different structural units contribute differently to phase arrival time identification. General model compression strategies cannot directly reflect the importance of each structural unit to the phase picking task; deleting structural units closely related to phase identification during pruning may lead to a decrease in picking accuracy. Furthermore, existing research largely focuses on preserving model accuracy and theoretical compression effects, with relatively insufficient attention paid to the model's operational efficiency, resource consumption, and deployment feasibility on actual edge devices. In particular, there is a lack of systematic research on the validation of large seismic wave models from model compression to edge deployment. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a structured pruning method and system for large seismic wave models at the edge, which solves the problems of high resource consumption, excessive inference latency, and unstable continuous operation of large seismic wave models during edge deployment.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] A structured pruning method for large seismic wave models oriented towards edge ends, the method comprising:

[0012] Obtain the large seismic wave model to be compressed, and identify the encoder in the model as the main object of structured pruning;

[0013] At the attention head level, for each attention head in each layer of the encoder, a first-order Taylor importance score, an importance score based on head-by-head ablation, and an importance score based on information entropy are calculated. These three scores are then normalized and weighted to obtain a comprehensive attention head importance score. Attention heads are then sorted from highest to lowest according to their comprehensive importance score, and the top three scores from each layer are retained. Select one high-importance attention head, delete the remaining low-importance attention heads, and simultaneously prune the attention projection matrix based on the set of retained attention heads;

[0014] At the FFN channel level, for each hidden channel in the FFN modules of each layer of the encoder, a first-order Taylor importance score, an importance score based on activation amplitude, and an importance score based on weight norm are calculated. These three scores are then normalized and weighted to obtain the comprehensive importance score of the FFN channel. The channels are then sorted from highest to lowest according to their comprehensive importance score, and the top three scores from each layer are retained. Select the most important channels, delete the remaining less important channels, and then prune the remaining channels accordingly. , and Projection matrix;

[0015] At the coding layer level, for each coding layer, an importance score based on the relative change in layer output and a score based on the average importance of attention heads are calculated. These two scores are then normalized and weighted to obtain a comprehensive importance score for the coding layer. The layers are then sorted from highest to lowest according to their comprehensive importance scores, and the top scores are retained. The most important coding layers are selected, the remaining less important coding layers are removed, and the remaining coding layers are rearranged in their original order to obtain the pruned lightweight model.

[0016] A structured pruning system for large seismic wave models oriented towards edge-ends, the system comprising: an encoder positioning module, an attention head pruning module, an FFN channel pruning module, and a coding layer pruning module;

[0017] The encoder positioning module is used to acquire the large seismic wave model to be compressed and to determine that the encoder in the model is the main object of structured pruning.

[0018] The attention head pruning module is used to calculate the first-order Taylor importance score, the importance score based on head-by-head ablation, and the importance score based on information entropy for each attention head in each layer of the encoder. The above three scores are normalized and weighted to obtain the comprehensive importance score of the attention head. The attention heads are sorted according to the scores and high importance attention heads are retained. The attention projection matrix is ​​pruned simultaneously.

[0019] The FFN channel pruning module is used to calculate the first-order Taylor importance score, the importance score based on activation amplitude, and the importance score based on weight norm for each hidden channel in each layer of the encoder's FFN module. These three scores are then normalized and weighted to obtain the comprehensive importance score of the FFN channel. Channels are then sorted by score, with high-importance channels retained, and pruning is performed simultaneously. , and Projection matrix;

[0020] The coding layer pruning module is used to calculate the importance score based on the relative change of the layer output and the score based on the average importance of the attention head for each coding layer. The two scores are normalized and weighted to obtain the comprehensive importance score of the coding layer. The coding layers with high importance are sorted by score and the layers with low importance are deleted. The layers with low importance are rearranged in the original order and the pruned lightweight model is output.

[0021] (III) Beneficial Effects

[0022] This invention provides a structured pruning method and system for large-scale seismic wave models oriented towards edge regions. Compared with existing technologies, it has the following advantages:

[0023] In this invention, the method constructs an importance assessment mechanism from three levels: attention head, FFN channel, and coding layer. Combined with masking verification and structured pruning, it effectively removes structural redundancy in large seismic wave models, achieving coordinated compression of model width and depth. Furthermore, during the model compression process, the pruned model maintains good phase picking ability. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a structured pruning method for large seismic wave models oriented towards the edge end, as described in an embodiment of the present invention.

[0026] Figure 2 This is a diagram of the DiTing model architecture in an embodiment of the present invention.

[0027] Figure 3 This is a heatmap of the importance score of the encoder attention head in Embodiment 1 of the present invention.

[0028] Figure 4This refers to the different attention-based pruning ratios in Embodiment 2 of the present invention. Waves and Diagram showing the variation in wave phase pickup performance.

[0029] Figure 5 This is a heatmap of FFN channel retention rate in Embodiment 3 of the present invention.

[0030] Figure 6 This is a graph showing the importance ranking of the FFN channel in Embodiment 4 of the present invention.

[0031] Figure 7 This is a graph showing the importance evaluation results of the Encoder layer in Embodiment 5 of the present invention.

[0032] Figure 8 This is a graph showing the change in model performance retention rate under different coding layer pruning numbers in Embodiment 6 of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] This application provides a structured pruning method and system for large seismic wave models at the edge, which solves the problems of high resource consumption, high inference latency, and unstable continuous operation of large seismic wave models during edge deployment.

[0035] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0036] Example:

[0037] like Figure 1 As shown, this invention provides a structured pruning method for large seismic wave models oriented towards the edge, the method comprising:

[0038] S1. Obtain the large seismic wave model to be compressed, and determine the encoder in the model as the main object of structured pruning;

[0039] S2. At the attention head level, for each attention head in each layer of the encoder, calculate the first-order Taylor importance score, the importance score based on head-by-head ablation, and the importance score based on information entropy. Normalize and weight these three scores to obtain the comprehensive importance score of the attention head. Sort the attention heads according to their comprehensive importance scores from highest to lowest, and retain the top three scores from each layer. Select one high-importance attention head, delete the remaining low-importance attention heads, and simultaneously prune the attention projection matrix based on the set of retained attention heads;

[0040] S3. At the FFN channel level, for each hidden channel in the FFN module of each layer of the encoder, calculate the first-order Taylor importance score, the importance score based on activation amplitude, and the importance score based on weight norm. Normalize and weight the above three scores to obtain the comprehensive importance score of the FFN channel. Sort the comprehensive importance scores of the FFN channels from high to low, and retain the top scores from each layer. Select the most important channels, delete the remaining less important channels, and then prune the remaining channels accordingly. , and Projection matrix;

[0041] S4. At the coding layer level, for each coding layer, calculate the importance score based on the relative change in layer output and the score based on the average importance of attention heads. Normalize and weight these two scores to obtain the comprehensive importance score of the coding layer. Sort the layers according to their comprehensive importance scores from highest to lowest, retaining the top scores. The most important coding layers are selected, the remaining less important coding layers are removed, and the remaining coding layers are rearranged in their original order to obtain the pruned lightweight model.

[0042] S1 specifically includes:

[0043] This embodiment uses the DiTing large-scale seismic wave model as the research object; such as Figure 2 As shown, DiTing is a large-scale seismic wave model designed for intelligent earthquake monitoring tasks, capable of simultaneously detecting earthquake events and... Wave, The wave phase picking task involves taking a three-component seismic waveform as input and fusing local feature extraction with global temporal features to output the probability of seismic events and the probability of phase arrival time. Its main structure includes a spatial prior branch (SPM), an embedding module, an injector module, an encoder module, and an extractor module.

[0044] To identify potential structural redundancy within the DiTing model, the parameter and computational cost distribution of its main components was first analyzed. The results show that the resource overhead of the DiTing model is primarily concentrated in the Encoder module; the Encoder module accounts for approximately 89.30% of the main module's parameters and 78.60% of its FLOPs, making it the main bottleneck affecting model inference latency, runtime memory, and edge deployment efficiency. In contrast, the Embedding, SPM, Injector, and Extractor modules have relatively lower parameter counts and computational costs, and are not the main sources of resource consumption affecting model deployment efficiency. Therefore, the Encoder is the primary target for structured pruning.

[0045] From a structural perspective, the main redundancy within the encoder comes from three levels: First, the contributions of different attention heads to the phase picking task are not balanced, and some attention heads may have functional overlap or insufficient contribution; second, a large number of hidden channels in the FFN module may have feature representation redundancy; third, as the network depth increases, the marginal contribution of some coding layers to feature representation updates may gradually weaken. Based on the above analysis, multi-level structured pruning is carried out from the three levels of attention heads, FFN channels, and coding layers.

[0046] In S2, the attention head comprehensive importance score The calculation formulas include:

[0047] ;

[0048] in, Indicates the first Layer A first-order Taylor importance score for each attention head;

[0049] Indicates the first Layer Ablation experiment scoring of individual attention heads;

[0050] Indicates the first Layer Information entropy score for each attention head;

[0051] ~ These represent the weight coefficients of the corresponding items; , , .

[0052] First-order Taylor importance is used to approximate the impact of removing a particular attention head on the task loss. Layer Taylor Importance Score for Each Attention Point The calculation formulas include:

[0053] ;

[0054] in, Indicates the first Layer The output feature map of each attention head;

[0055] This represents element-wise multiplication;

[0056] Represents the loss function The gradient with respect to the attention weights;

[0057] This indicates that the average is taken over all spatial locations and samples;

[0058] Considering that the model simultaneously undertakes wave and For the wave phase picking task, this paper calculates the corresponding Taylor importance score, and then performs weighted fusion after normalization to obtain the final Taylor importance score.

[0059] Masking verification is performed by directly masking a single attention head and observing. wave and The importance of the wave is assessed by the change in its F1-score;

[0060] For the Layer The performance changes of an attention head after ablation are as follows:

[0061] ;

[0062] ;

[0063] in, Indicates blocking the first Layer After a moment of attention The decrease in the F1-score;

[0064] Indicates the original model in Performance on wave pickup tasks;

[0065] express Wave Picking Task Shielding Layer Performance after attention head;

[0066] Indicates blocking the first Layer After a moment of attention The decrease in the F1-score;

[0067] Indicates the original model in Performance on wave pickup tasks;

[0068] express Wave Picking Task Shielding Layer Performance after attention head;

[0069] Regarding the above Waves and The ablation experiment score is obtained by weighting and fusing the changes in the F1-score of the wave.

[0070] ;

[0071] in, This represents the weight coefficient of the corresponding item. .

[0072] Attention distribution information entropy is used to measure the degree of concentration in the distribution of attention weights; the... Layer Information entropy score of each attention head The calculation formulas include:

[0073] ;

[0074] ;

[0075] in, Indicates the first Layer Information entropy of each attention head;

[0076] Indicates the first Layer The first one in the attention. The query position is the first Attention weights assigned to each key position;

[0077] Indicates the first Layer The number of query key positions in each attention head.

[0078] In S2, attention heads are sorted from highest to lowest according to their overall importance score, and the top-ranked heads are retained at each level. A highly important attention head is represented as:

[0079] ;

[0080] in, This indicates that the attention head with the highest importance score is selected from all attention heads in the current layer. One's attention;

[0081] The attention projection matrix is ​​synchronously pruned based on the retained set of attention heads, specifically including:

[0082] The corresponding retained channel index is determined based on the retained attention head set; the first Layer Reserved Channel Index Set Represented as:

[0083] ;

[0084] in, Indicates the first The set of attention heads retained by the layer;

[0085] The dimension representing a single attention head;

[0086] In S2, for the first The attention projection matrix of the layer is synchronously pruned as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] ;

[0091] in, This indicates the output channel in the retention matrix that corresponds to the important attention head;

[0092] This indicates the input channel in the retained output projection matrix that corresponds to the retained attention head.

[0093] In S3, the formula for calculating the overall importance score of the FFN channel includes:

[0094] ;

[0095] in, Indicates the first Layer First-order Taylor importance score for each FFN channel;

[0096] Indicates the first Layer Activation amplitude score of each FFN channel;

[0097] Indicates the first Layer Weight norm score of each FFN channel;

[0098] ~ These represent the weight coefficients of the corresponding items; , , .

[0099] First-order Taylor importance is used to approximate the impact of removing a certain FFN channel on task loss. Layer Taylor Importance Score for Each FFN Channel The calculation formulas include:

[0100] ;

[0101] ;

[0102] in, express The Taylor importance score corresponding to the wave;

[0103] express The Taylor importance score corresponding to the wave;

[0104] This represents the weight coefficient of the corresponding item. ;

[0105] Indicates the first Layer Activation matrix of FFN channels;

[0106] The loss function represents the loss function on the th Layer The gradient activated by each FFN channel;

[0107] This indicates that the average is taken over all samples and time locations.

[0108] Activation amplitude scoring is used to characterize the response intensity of a channel in real samples. Layer FFN channel activation amplitude score The calculation formulas include:

[0109] ;

[0110] in, Indicates the first Layer The activation value of each FFN channel;

[0111] This indicates that the average is taken over all samples and time locations.

[0112] The weighted norm score is used to evaluate channel contribution from the perspective of parameter magnitude. Layer Weight norm score of each FFN channel The calculation formulas include:

[0113] ;

[0114] in, express The Middle Row weight;

[0115] express The Middle Row weight;

[0116] express The Middle Column weights;

[0117] express Norm.

[0118] In S3, the channels are sorted from highest to lowest according to their overall importance score, and the top channels in each layer are retained. The highly important channels are represented as follows:

[0119] ;

[0120] in, Indicates the first The set of channels reserved by the layer;

[0121] This indicates selecting the channel with the highest importance score from all FFN channels in the current layer. One channel;

[0122] because , and They all work together in the same set of hidden FFN channels, so all three must be pruned synchronously during pruning;

[0123] In step S3, the channels are pruned synchronously based on the set of channels that are retained. , and The projection matrix is ​​represented as:

[0124] ;

[0125] ;

[0126] ;

[0127] in, , respectively representing the first In the layer FFN module and Projection matrix;

[0128] ,express Output projection matrix;

[0129] Indicates the first The row selection index for the layer-reserved channel;

[0130] Indicates the first Select the column index for the layer-preserved channel.

[0131] In S4, the formula for calculating the overall importance score of the coding layer includes:

[0132] ;

[0133] in, Indicates the first A comprehensive importance score for the layer coding layer;

[0134] Indicates the first Scoring of the relative change in the layer output of the layer coding layer;

[0135] Indicates the first Attention head importance scoring for layer-encoding layers;

[0136] This represents the weight coefficient of the corresponding item; .

[0137] The relative change in the layer output is used to characterize the magnitude of the current coding layer's update to the input features. The score of the relative change in the output of the layer coding layer Represented as:

[0138] ;

[0139] in, Indicates the first The output features of the layer;

[0140] express Norm;

[0141] This represents the average across all samples and time locations.

[0142] Attention head importance is used to characterize the interactive capabilities of global features. The importance score of the attention head in the layer coding layer is represented as follows:

[0143] ;

[0144] in, Indicates the first Layer A comprehensive importance score for each attention point. Indicates the first Number of attention heads in a layer.

[0145] In step S4, the coding layers are sorted from highest to lowest importance score, and the top scores are retained. Each highly important coding layer is represented as:

[0146] ;

[0147] in, This indicates that the layer with the highest importance score is selected from all layers. layer.

[0148] In step S4, the retained coding layers are rearranged in their original order to obtain the pruned lightweight model, specifically including:

[0149] To preserve the hierarchical order during the forward propagation of the DiTing model, the indexes of the retained layers are rearranged according to their order in the original model:

[0150] ;

[0151] ;

[0152] in, This represents the sorted sequence of retained layer indices;

[0153] Indicates the number of branches after pruning. The layer index of the retained coding layer in the original model;

[0154] The pruned Encoder is reconstructed based on the sorted retained layer sequence, and its forward propagation process is represented as follows:

[0155] ;

[0156] in, In the original model, the first... Mapping functions for each coding layer;

[0157] This indicates that the model has undergone the first stage after pruning. The output features after the encoding layer are preserved.

[0158] This invention provides a structured pruning system for large seismic wave models oriented towards the edge end. The system includes: an encoder positioning module, an attention head pruning module, an FFN channel pruning module, and a coding layer pruning module.

[0159] The encoder positioning module is used to acquire the large seismic wave model to be compressed and to determine that the encoder in the model is the main object of structured pruning.

[0160] The attention head pruning module is used to calculate the first-order Taylor importance score, the importance score based on head-by-head ablation, and the importance score based on information entropy for each attention head in each layer of the encoder. The above three scores are normalized and weighted to obtain the comprehensive importance score of the attention head. The attention heads are sorted according to the scores and high importance attention heads are retained. The attention projection matrix is ​​pruned simultaneously.

[0161] The FFN channel pruning module is used to calculate the first-order Taylor importance score, the importance score based on activation amplitude, and the importance score based on weight norm for each hidden channel in each layer of the encoder's FFN module. These three scores are then normalized and weighted to obtain the comprehensive importance score of the FFN channel. Channels are then sorted by score, with high-importance channels retained, and pruning is performed simultaneously. , and Projection matrix;

[0162] The coding layer pruning module is used to calculate the importance score based on the relative change of the layer output and the score based on the average importance of the attention head for each coding layer. The two scores are normalized and weighted to obtain the comprehensive importance score of the coding layer. The coding layers with high importance are sorted by score and the layers with low importance are deleted. The layers with low importance are rearranged in the original order and the pruned lightweight model is output.

[0163] The system also includes: an edge deployment verification module;

[0164] The edge deployment verification module is used to deploy the pruned lightweight model to the domestic RK3588 edge computing platform to test inference latency, throughput and memory usage.

[0165] Example 1:

[0166] Attention head redundancy analysis

[0167] To explain the basis for setting the number of attention heads to be pruned, this embodiment first analyzes the importance distribution of each layer of attention heads in the encoder and draws a heatmap of attention head importance scores, such as... Figure 3 As shown in the figure, the horizontal axis represents the Encoder layer index, the vertical axis represents the attention head number within each layer, and the color represents the normalized overall importance score of the attention head. The darker the color, the higher the contribution of the attention head to the seismic phase picking task.

[0168] Depend on Figure 3 It can be seen that there are significant differences in the importance scores of attention heads between different coding layers and within the same coding layer. A few attention heads have high scores, indicating that they make a strong contribution to the modeling of seismic phase correlation time series features; while a large number of attention heads have consistently low scores, indicating that these attention heads have little impact on the final... wave and The impact of the wave arrival on the pickup results is relatively limited. This phenomenon indicates that the multi-head attention in the DiTing encoder does not function uniformly; some attention heads may undertake similar information interaction functions, resulting in some functional overlap.

[0169] Therefore, pruning at the attention head level has a clear structural basis: retaining high-importance attention heads can maintain the model's ability to model the timing relationships of key seismic phases, while removing low-importance attention heads helps reduce redundant computations in the attention module.

[0170] Example 2:

[0171] Attention head retention count analysis

[0172] After confirming the uneven contribution of attention heads, this paper further determines the acceptable pruning ratio through a masking experiment. The masking experiment does not directly change the model's weight structure, but rather temporarily masks low-importance attention heads and observes the results. Wave, The F1-score variation can be used to evaluate the model's performance sensitivity under different pruning ratios. Performance variations under different attention head pruning ratios are shown below. Figure 4 As shown in the figure. The horizontal axis represents the attention head pruning ratio, and the vertical axis represents the model's performance at the corresponding pruning ratio. wave and F1 score on the wave phase picking task.

[0173] Depend on Figure 4 It can be seen that as the proportion of attention-based pruning gradually increases from a lower level, Waves and The F1-score changes relatively smoothly overall, indicating that the model is not sensitive to the removal of a small number of low-importance attention heads. When the pruning ratio reaches 50%, the model performance remains at a high level, indicating that retaining 4 high-importance attention heads per layer is sufficient to cover the main seismic phase time series modeling needs. Further increasing the pruning ratio leads to a more pronounced performance decline, indicating that high pruning rates begin to affect key attention heads, impacting the model's ability to capture phase-related features.

[0174] Therefore, this paper ultimately chooses to retain 4 high-importance attention heads in each layer and prune the remaining 4 low-importance attention heads. This setting is within a relatively stable pruning range, which can obtain clear benefits from attention module compression while avoiding excessive pruning that would reduce the ability to model key temporal relationships.

[0175] Example 3:

[0176] FFN channel redundancy analysis

[0177] To explain the rationale behind setting the number of FFN channel pruning parameters, this paper further analyzes the spatial distribution characteristics of the importance of FFN channels after multi-index fusion and plots a heatmap of FFN channel retention rates, such as... Figure 5 As shown in the figure, the horizontal axis represents the FFN channel interval number, and the vertical axis represents the Encoder layer index. The color indicates the retention ratio of high-importance channels within that interval; the darker the color, the higher the proportion of high-importance channels retained within that interval. Since each FFN layer contains 2048 hidden channels, direct channel-by-channel visualization would result in an overly dense image. Therefore, this paper divides the FFN channels into several consecutive intervals and calculates the retention ratio of high-importance channels within each interval.

[0178] Depend on Figure 5It can be seen that there are significant differences in the retention rates of FFN channels in different coding layers, and channels with high retention rates are mainly concentrated in certain channel ranges. This result indicates that not all hidden channels within the FFN module contribute equally to the phase picking task; a small number of channels undertake strong nonlinear feature transformation and expression enhancement, while a large number of low-retention-rate channels have a relatively limited impact on the model output.

[0179] Therefore, the key to FFN channel pruning is not to uniformly reduce the hidden dimension, but to retain the main features of the channel transformation according to the channel importance and delete the channels with weaker contributions or overlapping expressions.

[0180] Example 4:

[0181] FFN channel retention count analysis

[0182] To further determine the number of channels retained in the FFN structured pruning, this paper visualizes and analyzes the importance ranking results of FFN channels at each layer, such as... Figure 6 As shown in the figure, the horizontal axis represents the FFN channel number sorted from highest to lowest importance, and the vertical axis represents the normalized channel importance score. The curve reflects the decay trend of FFN channel importance within different layers as the sorting position decreases. This analysis is used to determine whether there is a significant inflection point when channel importance decays from high to low, thereby avoiding the impairment of key feature representation caused by setting the number of channels retained too small.

[0183] Depend on Figure 6 It can be seen that the importance of FFN channels exhibits a clear long-tail distribution. Channels ranked higher have higher importance and decrease more rapidly, indicating that these channels bear the main responsibility for expressing seismic phase features. As the ranking progresses further down the spectrum, the importance curve flattens out, suggesting that retaining more low-importance channels offers limited performance gains. In particular, when the channel ranking approaches 1024, the average importance curve enters a relatively flat range, indicating that retaining the first 1024 channels can cover the main effective expressive capabilities of the FFN module.

[0184] Therefore, this paper ultimately compresses the hidden dimension of each FFN layer from 2048 to 1024, retaining only the top 50% of the most important channels. This setting is located where the importance curve enters a flattening region, significantly reducing the number of parameters and computational cost of the FFN module while preserving the expressive power of the main nonlinear features as much as possible. At the same time, the regular 1024-dimensional structure is also beneficial for subsequent ONNX / RKNN transformations and edge matrix calculations.

[0185] Example 5:

[0186] Importance analysis of coding layer

[0187] To explain the rationale behind setting the number of pruning branches in the coding layers, this paper further visualizes and analyzes the importance distribution of each coding layer, as shown in the following results. Figure 7 As shown in the figure, the horizontal axis represents the Encoder layer index, the vertical axis represents the normalized overall importance score of the encoding layer, and the height of the bars represents the relative contribution of the corresponding encoding layer to the seismic phase picking task.

[0188] Depend on Figure 7 It can be seen that there are significant differences in the overall importance scores of different encoder layers. Shallow coding layers generally have high importance, indicating their crucial role in basic waveform feature transformation and local-to-global feature modeling. In contrast, the importance scores of some mid-to-late coding layers remain consistently low, suggesting that these layers have relatively limited contribution to the final phase picking results. As network depth increases, the importance of some adjacent coding layers continuously decreases, indicating that there may be feature transformation duplication or diminishing marginal returns in deep layer stacking.

[0189] Therefore, the basis for pruning the coding layer is not simply to reduce the network depth, but to prioritize retaining layers that contribute more to waveform representation and phase discrimination, and to delete redundant layers with lower marginal contributions.

[0190] Example 6:

[0191] Analysis of the number of elements retained in the coding layer

[0192] Since encoder layer pruning directly alters model depth, its risks are higher than those associated with attention head and FFN channel pruning. If too many layers are removed, the model's layer-by-layer feature transformation chain may be disrupted, leading to a significant decrease in phase picking performance. Therefore, before actually deleting the encoder layers, this paper first employs layer occlusion experiments to evaluate the performance preservation under different pruning numbers. The results are as follows: Figure 8 As shown in the figure. The horizontal axis represents the cumulative number of low-importance coding layers obscured, and the vertical axis represents... wave and The performance retention rate of the F1-score relative to the original model.

[0193] Depend on Figure 8 It can be seen that as the number of masked coding layers increases, the model in Waves and The performance retention rate in the wave seismic phase picking task generally shows a downward trend, but the performance degradation is not significant when the number of occlusions is small. When a total of 8 low-importance coding layers are occluded, Waves and The F1 score of the masking layers remained above 98% of the original model's performance, indicating that these layers had a relatively limited impact on the model's overall predictive ability. As the number of masking layers increased further, the performance degradation accelerated, suggesting that the removal of more highly important layers began to compromise the model's multi-level feature extraction capabilities.

[0194] Therefore, this paper ultimately reduces the number of coding layers from 24 to 16, by removing 8 low-importance coding layers. This setting corresponds to the stable performance range in the occlusion experiments, and is able to... Waves and While maintaining the basic performance of wave picking, the model depth, number of parameters and inference latency are significantly reduced; if further compressed to even fewer layers, the performance will degrade more significantly, which is not conducive to the requirements of picking reliability in edge deployment scenarios.

[0195] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0196] 1. In this embodiment of the invention, the multi-level structured pruning method proposed can effectively remove structural redundancy in large seismic wave models, achieving coordinated compression of model width and depth. After multi-level pruning, the number of model parameters is reduced from 124.20M to 56.89M, with a compression rate of 54.12%; FLOPs are reduced from 59.03G to 30.77G, a decrease of 47.87%; and the model size is reduced from 475.50MB to 217.68MB, with a compression rate of 54.22%.

[0197] 2. In this embodiment of the invention, the pruned model maintains good phase picking ability throughout the model compression process; the final pruned model performs well on the DiTingV3 test set. wave and The F1-scores for the waves reached 0.8880 and 0.8171 respectively, showing only a slight decrease compared to the original model; partial pruning stage The wave picking performance was even improved, indicating that properly removing low-contribution structures helps reduce interference from redundant features.

[0198] 3. In this embodiment of the invention, the theoretical compression benefits brought by the structured pruning method can be effectively converted into actual deployment benefits on edge devices; on the RK3588 edge platform, the pruning model inference time is reduced by 25.6%, the average inference latency is reduced by 26.4%, the throughput is increased by 35.86%, and the peak memory usage is reduced by about 44.0%; the experimental results verify the effectiveness of the method of the present invention on domestic edge devices.

[0199] 4. In this embodiment of the invention, the method can maintain model performance without pruning and fine-tuning, which simplifies the model compression process and makes it easier to apply in actual deployment scenarios.

[0200] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0201] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A structured pruning method for large-scale seismic wave models oriented towards the edge, characterized in that, The method includes: Obtain the large seismic wave model to be compressed, and identify the encoder in the model as the main object of structured pruning; At the attention head level, for each attention head in each layer of the encoder, a first-order Taylor importance score, an importance score based on head-by-head ablation, and an importance score based on information entropy are calculated. These three scores are then normalized and weighted to obtain a comprehensive attention head importance score. Attention heads are then sorted from highest to lowest according to their comprehensive importance score, and the top three scores from each layer are retained. Select one high-importance attention head, delete the remaining low-importance attention heads, and simultaneously prune the attention projection matrix based on the set of retained attention heads; At the FFN channel level, for each hidden channel in the FFN modules of each layer of the encoder, a first-order Taylor importance score, an importance score based on activation amplitude, and an importance score based on weight norm are calculated. These three scores are then normalized and weighted to obtain the comprehensive importance score of the FFN channel. The channels are then sorted from highest to lowest according to their comprehensive importance score, and the top three scores from each layer are retained. Select the most important channels, delete the remaining less important channels, and then prune the remaining channels accordingly. , and Projection matrix; At the coding layer level, for each coding layer, an importance score based on the relative change in layer output and a score based on the average importance of attention heads are calculated. These two scores are then normalized and weighted to obtain a comprehensive importance score for the coding layer. The layers are then sorted from highest to lowest according to their comprehensive importance scores, and the top scores are retained. The most important coding layers are selected, the remaining less important coding layers are removed, and the remaining coding layers are rearranged in their original order to obtain the pruned lightweight model.

2. The structured pruning method for large seismic wave models oriented towards the edge end as described in claim 1, characterized in that, The large seismic wave model to be compressed is the DiTing large seismic wave model; the encoder in the DiTing model is used as the main object of structured pruning.

3. The structured pruning method for large seismic wave models oriented towards the edge end as described in claim 1, characterized in that, The attention head comprehensive importance score Calculation formula include: ； in, Indicates the first Layer A first-order Taylor importance score for each attention head; Indicates the first Layer Ablation experiment scoring of individual attention heads; Indicates the first Layer Information entropy score for each attention head; ~ These represent the weight coefficients of the corresponding items.

4. The structured pruning method for large seismic wave models oriented towards the edge end as described in claim 1, characterized in that, Sort by attentional importance score from highest to lowest, retaining the top performers at each level. A highly important attention head is represented as: ； in, This indicates that the attention head with the highest importance score is selected from all attention heads in the current layer. One's attention; The attention projection matrix is ​​synchronously pruned based on the retained set of attention heads, specifically including: The corresponding retained channel index is determined based on the retained attention head set; the first Layer Reserved Channel Index Set Represented as: ； in, Indicates the first The set of attention heads retained by the layer; The dimension representing a single attention head; In S2, for the first The attention projection matrix of the layer is synchronously pruned as follows: ； ； ； ； in, This indicates the output channel in the retention matrix that corresponds to the important attention head; This indicates the input channel corresponding to the retained attention head in the retained output projection matrix.

5. The structured pruning method for large seismic wave models oriented towards the edge end as described in claim 1, characterized in that, The formula for calculating the overall importance score of the FFN channel includes: ； in, Indicates the first Layer First-order Taylor importance score for each FFN channel; Indicates the first Layer Activation amplitude score of each FFN channel; Indicates the first Layer Weight norm score of each FFN channel; ~ These represent the weight coefficients of the corresponding items.

6. The structured pruning method for large seismic wave models oriented towards the edge end as described in claim 1, characterized in that, Based on the overall importance score of the FFN channels, channels are sorted from highest to lowest, and the top channels at each level are retained. The highly important channels are represented as follows: ； in, Indicates the first The set of channels reserved by the layer; This indicates selecting the channel with the highest importance score from all FFN channels in the current layer. One channel; Synchronous trimming based on the retained channel set , and The projection matrix is ​​represented as: ； ； ； in, , respectively representing the first In the layer FFN module and Projection matrix; ,express Output projection matrix; Indicates the first The row selection index for the layer-reserved channel; Indicates the first Select the column index for the layer-preserved channel.

7. The structured pruning method for large seismic wave models oriented towards the edge end as described in claim 1, characterized in that, The formula for calculating the overall importance score of the coding layer includes: ； in, Indicates the first A comprehensive importance score for the layer coding layer; Indicates the first Scoring of the relative change in the layer output of the layer coding layer; Indicates the first Attention head importance scoring for layer-encoding layers; This represents the weight coefficient of the corresponding item.

8. The structured pruning method for large seismic wave models oriented towards the edge end as described in claim 7, characterized in that, No. The score of the relative change in the output of the layer coding layer Represented as: ； in, Indicates the first The output features of the layer; express Norm; This represents the average across all samples and time locations; No. The importance score of the attention head in the layer coding layer is represented as follows: ； in, Indicates the first Layer A comprehensive importance score for each attention point. Indicates the first Number of attention heads in a layer.

9. The structured pruning method for large seismic wave models oriented towards the edge end as described in claim 1, characterized in that, Sort by the overall importance score of the coding layer from highest to lowest, and retain the top [number]. Each highly important coding layer is represented as: ； in, This indicates that the layer with the highest importance score is selected from all layers. layer; The retained coding layers are rearranged in their original order to obtain the pruned lightweight model, which specifically includes: To preserve the hierarchical order during the forward propagation of the DiTing model, the indexes of the retained layers are rearranged according to their order in the original model: ； ； in, This represents the sorted sequence of retained layer indices; Indicates the number of branches after pruning. The layer index of the retained coding layer in the original model; The pruned Encoder is reconstructed based on the sorted retained layer sequence, and its forward propagation process is represented as follows: ； in, In the original model, the first... Mapping functions for each coding layer; This indicates that the model has undergone the first stage after pruning. The output features after the encoding layer are preserved.

10. A structured pruning system for large-scale seismic wave models oriented towards edge ends, characterized in that, The system includes: an encoder positioning module, an attention head pruning module, an FFN channel pruning module, and a coding layer pruning module; The encoder positioning module is used to acquire the large seismic wave model to be compressed and to determine that the encoder in the model is the main object of structured pruning. The attention head pruning module is used to calculate the first-order Taylor importance score, the importance score based on head-by-head ablation, and the importance score based on information entropy for each attention head in each layer of the encoder. The above three scores are normalized and weighted to obtain the comprehensive importance score of the attention head. The attention heads are sorted according to the scores and high importance attention heads are retained. The attention projection matrix is ​​pruned simultaneously. The FFN channel pruning module is used to calculate the first-order Taylor importance score, the importance score based on activation amplitude, and the importance score based on weight norm for each hidden channel in each layer of the encoder's FFN module. These three scores are then normalized and weighted to obtain the comprehensive importance score of the FFN channel. Channels are then sorted by score, with high-importance channels retained, and pruning is performed simultaneously. , and Projection matrix; The coding layer pruning module is used to calculate the importance score based on the relative change of the layer output and the score based on the average importance of the attention heads for each coding layer. The two scores are normalized and weighted to obtain the comprehensive importance score of the coding layer. The coding layers with high importance are sorted by score and the layers with low importance are deleted. The layers with low importance are rearranged in the original order and the pruned lightweight model is output.

Images